Skip to content

Latest commit

 

History

History
112 lines (77 loc) · 3.02 KB

cloud_storage.rst

File metadata and controls

112 lines (77 loc) · 3.02 KB
Raw

Cloud Storage

All data on TensorBay are hosted on cloud.
TensorBay supports two cloud storage modes:
  • DEFAULT CLOUD STORAGE: data are stored on TensorBay cloud
  • AUTHORIZED CLOUD STORAGE: data are stored on other providers' cloud

Default Cloud Storage

In default cloud storage mode, data are stored on TensorBay cloud.
Create a dataset with default storage:

../../../docs/code/getting_started_with_tensorbay.py

Authorized Cloud Storage

You can also upload data to your public cloud storage space.
Now TensorBay support following cloud providers:
  • Aliyun OSS
  • Amazon S3
  • Azure Blob

Config

See cloud storage instruction for details about how to configure cloud storage on TensorBay.

TensorBay SDK supports a method to list a user's all previous configurations.

from tensorbay import GAS

gas = GAS("<YOUR_ACCESSKEY>")
gas.list_auth_storage_configs()

Create Authorized Storage Dataset

Create a dataset with authorized cloud storage:

dataset_client = gas.create_auth_dataset("dataset_name", "config_name", "path/to/dataset")

Import Cloud Files into Authorized Storage Dataset

Take the following cloud directory as an example:

data/
├── images/
│   ├── 00001.png
│   ├── 00002.png
│   └── ...
├── labels/
│   ├── 00001.json
│   ├── 00002.json
│   └── ...
└── ...

Get a cloud client.

../../../docs/code/cloud_storage.py

Import the AuthData from cloud platform and load label file to an authorized storage dataset.

../../../docs/code/cloud_storage.py

Important

Files will be copied from raw directory to the authorized cloud storage dataset path, thus the storage space will be doubled on the cloud platform.