Skip to content
Permalink
main
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

backend.ai-client-py/docs/cli/storage.rst

Go to file
 
 
Cannot retrieve contributors at this time
Storage Management Creating vfolders and managing them File transfers and management Running sessions with storages Creating default files for kernels
116 lines (70 sloc) 3.15 KB
Raw Blame
Open in GitHub Desktop

Storage Management

Note

Please consult the detailed usage in the help of each command (use -h or --help argument to display the manual).

Backend.AI abstracts shared network storages into per-user slices called "virtual folders" (aka "vfolders"), which can be shared between users and user group members.

Creating vfolders and managing them

The command-line interface provides a set of subcommands under backend.ai vfolder to manage vfolders and files inside them.

To list accessible vfolders including your own ones and those shared by other users:

$ backend.ai vfolder list

To create a virtual folder named "mydata1":

$ backend.ai vfolder create mydata1 mynas

The second argument mynas corresponds to the name of a storage host. To list up storage hosts that you are allowed to use:

$ backend.ai vfolder list-hosts

To delete the vfolder completey:

$ backend.ai vfolder delete mydata1

File transfers and management

To upload a file from the current working directory into the vfolder:

$ backend.ai vfolder upload mydata1 ./bigdata.csv

To download a file from the vfolder into the current working directory:

$ backend.ai vfolder download mydata1 ./bigresult.txt

To list files in the vfolder's specific path:

$ backend.ai vfolder ls mydata1 .

To delete files in the vfolder:

$ backend.ai vfolder rm mydata1 ./bigdata.csv

Warning

All file uploads and downloads overwrite existing files and all file operations are irreversible.

Running sessions with storages

The following command spawns a Python session where the virtual folder "mydata1" is mounted. The execution options are omitted in this example. Then, it downloads ./bigresult.txt file (generated by your code) from the "mydata1" virtual folder.

$ backend.ai vfolder upload mydata1 ./bigdata.csv
$ backend.ai run --rm -m mydata1 python:3.6-ubuntu18.04 ...
$ backend.ai vfolder download mydata1 ./bigresult.txt

In your code, you may access the virtual folder via /home/work/mydata1 (where the default current working directory is /home/work) just like a normal directory. If you want to mount vfolders in other path, add '/' as prefix at the forefont of the vfolder path.

By reusing the same vfolder in subsequent sessions, you do not have to donwload the result and upload it as the input for next sessions, just keeping them in the storage.

Creating default files for kernels

Backend.AI has a feature called 'dotfile', created to all the kernels user spawns. As you can guess, dotfile's path should start with .. The following command creates dotfile named .aws/config with permission 755. This file will be created under /home/work every time user spawns Backend.AI kernel.

$ backend.ai dotfile create .aws/config < ~/.aws/config