docs: add document about import dataset from cloud storage platform #756
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pull Request Test Coverage Report for Build 983312820
💛 - Coveralls |
linjiX
requested changes
Jun 25, 2021
Hoteryoung
reviewed
Jun 27, 2021
|
|
||
| The directory ``path/to/dataset`` should be empty when create an authorized storage Fusion Dataset. | ||
|
|
||
| Inport data into tensorby |
There was a problem hiding this comment.
Import Cloud Files into Authorized Storage Dataset
| Inport data into tensorby | ||
| ========================= | ||
|
|
||
| If your use cloud platform to storage data, you will first need to import your data into TensorBay. |
There was a problem hiding this comment.
Line 64 is a little bit redundant ...
Hoteryoung
requested changes
Jun 27, 2021
|
|
||
| If your use cloud platform to storage data, you will first need to import your data into TensorBay. | ||
|
|
||
| There are two options to import your data from raw cloud platform path to authorized storage dataset. |
There was a problem hiding this comment.
There are two methods to import cloud files into an authorized storage dataset.
|
|
||
| There are two options to import your data from raw cloud platform path to authorized storage dataset. | ||
|
|
||
| 1. Import all files with a directory path into datset -- out-of-the-box |
There was a problem hiding this comment.
- OUT-OF-THE-BOX: Import all files under a directory into a dataset.
- CUSTOMIZED: Use AuthData to organize cloud files into a dataset.
| 1. Import all files with a directory path into datset -- out-of-the-box | ||
| 2. Use AuthData to organized a dataset -- customization | ||
|
|
||
| Suppose the cloud storage platform structure like :: |
There was a problem hiding this comment.
Take the following cloud directory as an example::
| └── ... | ||
|
|
||
|
|
||
| Import all files from one directory into a target segment. |
| dataset_client.import_all_files("datas/images", "train") | ||
| Import files and upload label to several segments |
|
|
||
| .. code:: python | ||
| datas = cloud_client.open("datas/labels/0001.json").read() |
| datas = cloud_client.open("datas/labels/0001.json").read() | ||
| Use the file path in cloud platform as the AuthData path. |
There was a problem hiding this comment.
Use the cloud file path as ...
| AuthData("train/data/0001.png") | ||
| load label file from cloud platform to tensorby label format. |
There was a problem hiding this comment.
load -> Load
tensorbay -> TensorBay
| :start-after: """Load label file from cloud platform into the AuthData""" | ||
| :end-before: """""" | ||
|
|
||
| Import dataset from cloud platform to authorized storage dataset. |
There was a problem hiding this comment.
Import cloud files to an authorized storage dataset.
| .. important:: | ||
|
|
||
| The file will copy from raw directory to the authorized cloud storage dataset path, | ||
| set `delete_source=True` to delete raw files when the import is finished. |
There was a problem hiding this comment.
- Please use double apostrophes like
delete_source=Trueto describe inline code.
There are several cases in this PR, please correct them all. - Please describe explicitly the place to set delete_source=True.
Add a line of code, for example.
AChenQ
force-pushed
the
T16354_cloud_doc
branch
2 times, most recently
from
03d7cb8 to
8be2803
Compare
Hoteryoung
requested changes
Jun 29, 2021
| Out-of-the-box Method | ||
| ********************* | ||
|
|
||
| Import all files in `datas/images` directory into `train` segment. |
There was a problem hiding this comment.
Please use double apostrophes when writing inline code.
| cloud_client = dataset_client.get_cloud_client() | ||
| List all files in `datas/train`. |
Hoteryoung
requested changes
Jun 29, 2021
linjiX
requested changes
Jun 29, 2021
|
|
||
| .. code:: python | ||
| contents = cloud_client.open("datas/labels/0001.json").read() |
AChenQ
force-pushed
the
T16354_cloud_doc
branch
4 times, most recently
from
f2e107d to
784643b
Compare
Hoteryoung
approved these changes
Jun 29, 2021
Hoteryoung
requested changes
Jun 29, 2021
|
|
||
| .. important:: | ||
|
|
||
| The file will copy from raw directory to the authorized cloud storage dataset path, |
There was a problem hiding this comment.
Files will be copied from ...
, -> .
| .. important:: | ||
|
|
||
| The file will copy from raw directory to the authorized cloud storage dataset path, | ||
| So the storage space of file will double |
There was a problem hiding this comment.
Thus the storage space will be doubled on the cloud platform.
AChenQ
force-pushed
the
T16354_cloud_doc
branch
2 times, most recently
from
6e9cc32 to
dbff152
Compare
linjiX
requested changes
Jun 29, 2021
docs/code/cloud_storage.py
Outdated
|
|
||
| from tensorbay.label import LabeledBox2D | ||
|
|
||
| images = cloud_client.list_auth_data("datas/images") |
docs/code/cloud_storage.py
Outdated
| labels = cloud_client.list_auth_data("datas/labels") | ||
|
|
||
| auth_data = images[0] | ||
| auth_data.label.box2d = [LabeledBox2D.loads(json.load(labels[0].open()))] |
There was a problem hiding this comment.
classification is better for this example
docs/code/cloud_storage.py
Outdated
| segment = dataset.create_segment() | ||
|
|
||
| segment.append(AuthData("train/data/0001.png")) | ||
| segment.append(AuthData("train/data/0002.png")) |
There was a problem hiding this comment.
Using AuthData.__init__ to get AuthData instance is not recommend now.
linjiX
requested changes
Jun 29, 2021
docs/code/cloud_storage.py
Outdated
| images = cloud_client.list_auth_data("data/images") | ||
| labels = cloud_client.list_auth_data("data/labels") | ||
|
|
||
| for index, auth_data in enumerate(images): |
linjiX
approved these changes
Jun 29, 2021
Hoteryoung
approved these changes
Jun 29, 2021
linjiX
pushed a commit
that referenced
this pull request
Jun 29, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.