docs: Merge client-py docs and reorgzize (#435)

* Merge existing `ko` translations
* Merge the client SDK docs
* Reorganize the ToC structure so that it seamelessly integrates with
  any-level of hierarchies
* All directories now have `index.rst` to define its own ToC structure

.gitignore

@@ -91,7 +91,8 @@ ENV/
 
 # IDE/vim
 .idea
-.*.swp
+*.swp
+*.swo
 .vscode
 
 # Pants

changes/435.doc.md

@@ -0,0 +1 @@
+Merge the documentation of the Client SDK for Python into the unified docs

docs/README.md

@@ -5,96 +5,99 @@ Developer guide for Backend.AI documentation
 
 ## Setting up the build environment for docs
 
-> 📌 NOTE: You may need a sudo access for a certain command.
+### Installing pyenv and pyenv-virtualenv
 
-### clone docs in `lablup/backend.ai` project
-Download `lablup/backend.ai` project using `git clone` and rename the clone directory as `meta`.
+* Please refer the official docs:
+  - https://github.com/pyenv/pyenv#installation
+  - https://github.com/pyenv/pyenv-virtualenv#installation
 
-```console
-$ cd ~
-$ git clone https://github.com/lablup/backend.ai meta
-```
-Change the current working directory to `meta/docs`.
-
-```console
-$ cd ~/meta/docs
-```
-
-### pyenv and pyenv-virtualenv installation
-
-#### pyenv installation
-Please refer to the [README](https://github.com/pyenv/pyenv#installation) of the official [pyenv](https://github.com/pyenv/pyenv) repository and install it.
-
-#### pyenv-virtualenv installation (optional)
-`pyenv-virtualenv` is a `pyenv` plugin.  It is not mandatory to build the Backend.AI docs but highly recommended.   
-To install it, please refer to the [README](https://github.com/pyenv/pyenv-virtualenv#installation) of the official [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) repository.
+### Setting up the documenting environment
 
-### Sphinx installation
-
-Install [sphinx](https://www.sphinx-doc.org/en/master/) using [pip](https://pypi.org/project/pip/)
+Then, follow the instructions below:
 
 ```console
-$ pip install sphinx
+$ pyenv virtualenv 3.10.4 venv-bai-docs
+$ git clone https://github.com/lablup/backend.ai backend.ai
+$ cd ~/backend.ai/docs
+$ pyenv local venv-bai-docs
+$ pip install -U pip setuptools wheel
+$ pip install -U -r requirements.txt   # docs/requirements.txt
 ```
 
-### sphinx-intl installation
-
-Install [sphinx-intl](https://github.com/sphinx-doc/sphinx-intl) using pip.
-
-```console
-$ pip install sphinx-intl
-```
-
-
 ## Building HTML document
 
-> 📌 NOTE: Please make sure pyenv installation completed and the activation has been enabled before building documents, and current working directory need to be root directory of `docs`.
+> 📌 NOTE: Please ensure that you are inside the `docs` directory and the virtualenv is activated.
 
 ### Make the html version
+
 ```console
 $ make html
 ```
-### Translation
+
+The compiled documentation is under `_build/html/index.html`.
+You may serve it for local inspection using `python -m http.server --directory _build/html`.
+
+## Translation
 
 #### Generate/update pot (Portable Object Template) files
+
 ```console
 $ make gettext
 ```
 
 #### Build po (Portable Object) files using sphinx-intl
 
-> In this guide, we use Korean as the target translation language.
-
 ```console
-$ sphinx-intl update -p _build/locale/ -l ko_KR
+$ sphinx-intl update -p _build/locale/ -l ko
 ```
 
+The `.po` message files are under `locales/ko/LC_MESSAGES/`.
+Edit them by filling missing translations.
+
 #### Build HTML files with translated version
 
 ```console
+$ sphinx-intl build
 $ make -e SPHINXOPTS="-D language='ko'" html
 ```
 
+
 ## 🚧 Building PDF document (WIP) 🚧
 
-> Help wanted!   
+> Help wanted!
 
 We are looking for people to help with a short guide for building PDF document based on html files derived from sphinx.
 
 
 ## Advanced Settings
 
-### Managing the hierachy of toctree(Table of Contents) of documentation
+### Managing the hierachy of toctree (Table of Contents) of documentation
 
-When documentation of each file gets too big to contain all things in one topic,   
-It should be branched with proper sub-topics.   
-The hierarchy of toctree has been managed through `index.rst`.   
-Please note that contents in `index.rst` must contain the actual directory tree, unless it will not contain documentation you expected.   
+When documentation of each file gets too big to contain all things in one topic,
+It should be branched with proper sub-topics.
+The hierarchy of toctree has been managed through `index.rst`.
+Please note that contents in `index.rst` must contain the actual directory tree, unless it will not contain documentation you expected.
 
 For More Information, Please check out [`index.rst`](https://github.com/lablup/backend.ai/blob/main/docs/index.rst) file.
 
+### Adding a new language translation
 
-## References for newcomers
+Add a new project in readthedocs.org with the "-xx" suffix
+where "xx" is an ISO 639-1 language code, which targets
+the same GitHub address to the original project.
+
+Then configure the main project in readthedocs.org to have
+the new project as a specific language translation.
+
+Example:
+
+* https://readthedocs.org/projects/sorna
+* https://readthedocs.org/projects/sorna-ko
 
-http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
+Please ask the docs maintainer for help.
+
+
+## References for newcomers
 
+- http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
+- https://poedit.net/

docs/agent/accelerators.rst

@@ -0,0 +1,2 @@
+Accelerators (aka Compute Plugins)
+==================================

docs/agent/docker.rst

@@ -0,0 +1,2 @@
+Docker Backend
+==============

docs/agent/index.rst

@@ -0,0 +1,10 @@
+Backend.AI Agent Reference
+==========================
+
+.. toctree::
+   :maxdepth: 2
+
+   rpc
+   docker
+   k8s
+   accelerators

docs/agent/k8s.rst

@@ -0,0 +1,2 @@
+Kubernetes Backend
+==================

docs/agent/rpc.rst

@@ -0,0 +1,2 @@
+RPC Interface for Kernel Management
+===================================

docs/client/cli/apps.rst

@@ -0,0 +1,124 @@
+Container Applications
+======================
+
+.. note::
+
+   Please consult the detailed usage in the help of each command
+   (use ``-h`` or ``--help`` argument to display the manual).
+
+
+Starting a session and connecting to its Jupyter Notebook
+---------------------------------------------------------
+
+The following command first spawns a Python session named "mysession"
+without running any code immediately, and then executes a local proxy which
+connects to the "jupyter" service running inside the session via the local
+TCP port 9900.
+The ``start`` command shows application services provided by the created
+compute session so that you can choose one in the subsequent ``app``
+command.
+In the ``start`` command, you can specify detailed resource options using
+``-r`` and storage mounts using ``-m`` parameter.
+
+.. code-block:: shell
+
+  backend.ai start -t mysession python
+  backend.ai app -b 9900 mysession jupyter
+
+Once executed, the ``app`` command waits for the user to open the displayed
+address using appropriate application.
+For the jupyter service, use your favorite web browser just like the
+way you use Jupyter Notebooks.
+To stop the ``app`` command, press ``Ctrl+C`` or send the ``SIGINT`` signal.
+
+Accessing sessions via a web terminal
+-------------------------------------
+
+All Backend.AI sessions expose an intrinsic application named ``"ttyd"``.
+It is an web application that embeds xterm.js-based full-screen terminal
+that runs on web browsers.
+
+.. code-block:: shell
+
+   backend.ai start -t mysession ...
+   backend.ai app -b 9900 mysession ttyd
+
+Then open ``http://localhost:9900`` to access the shell in a fully
+functional web terminal using browsers.
+The default shell is ``/bin/bash`` for Ubuntu/CentOS-based images and
+``/bin/ash`` for Alpine-based images with a fallback to ``/bin/sh``.
+
+.. note::
+
+   This shell access does *NOT* grant your root access.
+   All compute session processes are executed as the user privilege.
+
+
+Accessing sessions via native SSH/SFTP
+--------------------------------------
+
+Backend.AI offers direct access to compute sessions (containers) via SSH
+and SFTP, by auto-generating host identity and user keypairs for all
+sessions.
+All Baceknd.AI sessions expose an intrinsic application named ``"sshd"``
+like ``"ttyd"``.
+
+To connect your sessions with SSH, first prepare your session and download
+an auto-generated SSH keypair named ``id_container``.
+Then start the service port proxy ("app" command) to open a local TCP port
+that proxies the SSH/SFTP traffic to the compute sessions:
+
+.. code-block:: console
+
+  $ backend.ai start -t mysess ...
+  $ backend.ai download mysess id_container
+  $ mv id_container ~/.ssh
+  $ backend.ai app mysess sshd -b 9922
+
+In another terminal on the same PC, run your ssh client like:
+
+.. code-block:: console
+
+  $ ssh -o StrictHostKeyChecking=no \
+  >     -o UserKnownHostsFile=/dev/null \
+  >     -i ~/.ssh/id_container \
+  >     work@localhost -p 9922
+  Warning: Permanently added '[127.0.0.1]:9922' (RSA) to the list of known hosts.
+  f310e8dbce83:~$
+
+This SSH port is also compatible with SFTP to browse the container's
+filesystem and to upload/download large-sized files.
+
+You could add the following to your ``~/.ssh/config`` to avoid type
+extra options every time.
+
+.. code-block:: text
+
+  Host localhost
+    User work
+    IdentityFile ~/.ssh/id_container
+    StrictHostKeyChecking no
+    UserKnownHostsFile /dev/null
+
+.. code-block:: console
+
+  $ ssh localhost -p 9922
+
+.. warning::
+
+   Since the SSH keypair is auto-generated every time when your launch a
+   new compute session, you need to download and keep it separately for
+   each session.
+
+To use your own SSH private key across all your sessions without
+downloading the auto-generated one every time, create a vfolder named
+``.ssh`` and put the ``authorized_keys`` file that includes the public key.
+The keypair and ``.ssh`` directory permissions will be automatically
+updated by Backend.AI when the session launches.
+
+.. code-block:: console
+
+  $ ssh-keygen -t rsa -b 2048 -f id_container
+  $ cat id_container.pub > authorized_keys
+  $ backend.ai vfolder create .ssh
+  $ backend.ai vfolder upload .ssh authorized_keys

docs/client/cli/code-execution.rst

@@ -0,0 +1,72 @@
+Advanced Code Execution
+=======================
+
+.. note::
+
+   Please consult the detailed usage in the help of each command
+   (use ``-h`` or ``--help`` argument to display the manual).
+
+
+Running concurrent experiment sessions
+--------------------------------------
+
+In addition to single-shot code execution as described in
+:ref:`simple-execution`, the ``run`` command offers concurrent execution of
+multiple sessions with different parameters interpolated in the execution
+command specified in ``--exec`` option and environment variables specified
+as ``-e`` / ``--env`` options.
+
+To define variables interpolated in the ``--exec`` option, use ``--exec-range``.
+To define variables interpolated in the ``--env`` options, use ``--env-range``.
+
+Here is an example with environment variable ranges that expands into 4
+concurrent sessions.
+
+.. code-block:: shell
+
+  backend.ai run -c 'import os; print("Hello world, {}".format(os.environ["CASENO"]))' \
+      -r cpu=1 -r mem=256m \
+      -e 'CASENO=$X' \
+      --env-range=X=case:1,2,3,4 \
+      lablup/python:3.6-ubuntu18.04
+
+Both range options accept a special form of argument: "range expressions".
+The front part of range option value consists of the variable name used for
+interpolation and an equivalence sign (``=``).
+The rest of range expressions have the following three types:
+
+.. list-table::
+   :widths: 24 76
+   :header-rows: 1
+
+   * - Expression
+     - Interpretation
+
+   * - ``case:CASE1,CASE2,...,CASEN``
+     - A list of discrete values. The values may be either string or numbers.
+
+   * - ``linspace:START,STOP,POINTS``
+     - An inclusive numerical range with discrete points, in the same way
+       of ``numpy.linspace()``.  For example, ``linspace:1,2,3`` generates
+       a list of three values: 1, 1.5, and 2.
+
+   * - ``range:START,STOP,STEP``
+     - A numerical range with the same semantics of Python's :func:`range`.
+       For example, ``range:1,6,2`` generates a list of values:
+       1, 3, and 5.
+
+If you specify multiple occurrences of range options in the ``run``
+command, the client spawns sessions for *all possible combinations* of all
+values specified by each range.
+
+.. note::
+
+  When your resource limit and cluster's resource capacity cannot run all
+  spawned sessions at the same time, some of sessions may be queued and the
+  command may take a long time to finish.
+
+.. warning::
+
+  Until all cases finish, the client must keep its network connections to
+  the server alive because this feature is implemented in the client-side.
+  Server-side batch job scheduling is under development!