Skip to content
This repository has been archived by the owner on Jun 14, 2023. It is now read-only.

Commit

Permalink
final(?) version of toy-example description
Browse files Browse the repository at this point in the history
  • Loading branch information
valentin.kozlov committed Jan 15, 2019
1 parent b8bcc7f commit e114f4f
Showing 1 changed file with 34 additions and 22 deletions.
56 changes: 34 additions & 22 deletions source/user/models/toy_model-dog_breeds.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,13 @@ Local Workflow
The described workflow supposes usage of downloaded from DEEP Open Catalogue Docker images, i.e. you need either
`docker <https://docs.docker.com/install/#supported-platforms>`_ or `udocker <https://github.com/indigo-dc/udocker/releases>`_ tool.

**1. Workflow intro**
1. Workflow intro
""""""""""""""""""

1) `DEEPaaS API <https://deepaas.readthedocs.io/en/stable/>`_ uses port 5000 for access, one therefore has to map the container and host ports,
see Examples below.
a. `DEEPaaS API <https://deepaas.readthedocs.io/en/stable/>`_ uses port 5000 for access, one therefore has to map the container and host ports,
see :ref:`sec-examples`.

2) Following two directories inside the docker container are used for input and output:
b. Following two directories inside the docker container are used for input and output:

+--------------------------------+----------------------------------------+
| **Directory inside container** | **Description** |
Expand All @@ -64,9 +65,10 @@ see Examples below.
If you want to perform full training, then you need to mount your data into ``/srv/dogs_breed_det/data``.

If you want to keep trained weights in the persistant place, then you have to mount ``/srv/dogs_breed_det/models``
to a persistent volume. You can either use your local directories or connect your remote storage, see Examples below.
to a persistent volume. You can either use your local directories or connect your remote storage, see :ref:`sec-examples`.

**2. Data input**
2. Data input
""""""""""""""

Original dataset consists of dog's images for 133 breeds. The images are compressed in
one `dogImages.zip <https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/dogImages.zip>`_ file.
Expand All @@ -89,8 +91,8 @@ These directories are automatically de-archived in ``/srv/dogs_breed_det/data/do

Trainig labels are also created automatically based on the directory names, truncating leading numbers, e.g. '002.'.

**The minimum requirement for training is to place this** `dogImages.zip <https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/dogImages.zip>`_
**in** ``/srv/dogs_breed_det/data/raw/`` **directory.**
**The minimum requirement for training is to make** `dogImages.zip <https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/dogImages.zip>`_
**available in** ``/srv/dogs_breed_det/data/raw/`` **directory.**

If you want to use your own dataset then it has to follow similar structure.

Expand Down Expand Up @@ -123,19 +125,21 @@ In this case, you place `dogImages.zip <https://s3-us-west-1.amazonaws.com/udaci
which makes it available in ``/srv/dogs_breed_det/data/raw``.


**3. Accessing application**
3. Accessing application
""""""""""""""""""""""""

* In a minimum case to classify images with already trained Resnet50 model, start the container as::

docker run -ti -p 5000:5000 deephdc/deep-oc-dogs_breed_det:cpu deepaas-run --listen-ip=0.0.0.0
* In more advanced cases (see Examples below) you may need to mount various directories or pass environment settings.
* In more advanced cases (see :ref:`sec-examples`) you may need to mount various directories or pass environment settings.

* Direct your web browser to http://127.0.0.1:5000


**4. Test the classifier**
4. Test the classifier
"""""""""""""""""""""""

* Go to **/models/{model_name}/predict** , click "**Try it out**" button

Expand All @@ -150,9 +154,10 @@ which makes it available in ``/srv/dogs_breed_det/data/raw``.
.. note:: By default only weigths for Dogs_Resnet50 are available (automatically downloaded from the shared link, see above "Pre-trained weights" URL), all other models have to be trained first!


**5. Train the classifier**
5. Train the classifier
"""""""""""""""""""""""

* Connect your data storage with the corresponding directory inside the container (see "Data input" above and Examples below)
* Connect your data storage with the corresponding directory inside the container (see "Data input" above and :ref:`sec-examples` below)
* Go to **/models/{model_name}/train** , click "**Try it out**" button
* Type **model_name**, one of the ``Dogs_Resnet50``, ``Dogs_InceptionV3``, ``Dogs_VGG16``, ``Dogs_VGG19``
* Execute training
Expand All @@ -164,28 +169,34 @@ which makes it available in ``/srv/dogs_breed_det/data/raw``.
DEEP Pilot infrastructure submission
------------------------------------

Please, refere to :doc:`Quickstart Guide <../quickstart>`, section "Run model on DEEP Pilot infrastructure",
on what is required to start the application on DEEP Pilot infrastructure.

.. _sec-examples:

Examples
--------

**1. Mount local host directories**
Mount local host directories
"""""""""""""""""""""""""""""

Example 1 (GPU, default):
**Example 1 (GPU, default):**
::

docker run -ti -p 5000:5000 -v ~/data:/srv/dogs_breed_det/data \
-v ~/models:/srv/dogs_breed_det/models \
deephdc/deep-oc-dogs_breed_det deepaas-run --listen-ip=0.0.0.0

Example 2 (CPU):
**Example 2 (CPU):**
::

docker run -ti -p 5000:5000 -v ~/data:/srv/dogs_breed_det/data \
-v ~/models:/srv/dogs_breed_det/models \
deephdc/deep-oc-dogs_breed_det:cpu deepaas-run --listen-ip=0.0.0.0


**2. Connect to a remote storage by using** ``rclone.conf`` **from your host**
Connect to a remote storage by using ``rclone.conf`` from your host
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

`rclone <https://rclone.org/>`_ tool allows to connect to a plenty of remote storages.
The tool is already installed in the Docker image and expects your ``data/`` and ``models/`` sub-directories to be under ``deepnc:/Datasets/dogs_breed/``.
Expand All @@ -194,7 +205,7 @@ If no data found in your container, rclone attempts to connect to ``deepnc:/`` a
If you are familiar with the `rclone <https://rclone.org/>`_ tool, you probably have ``rclone.conf`` file on your host.
You can rename one of the pre-configured remote storages to ``deepnc``, then mount host directory with your ``rclone.conf`` file into the container:

Example 3: using in the container ``rclone.conf`` from your host
**Example 3:** using in the container ``rclone.conf`` from your host
::

docker run -ti -p 5000:5000 -v $HOSTDIR_WITH_RCLONE_CONF:/srv/rclone \
Expand All @@ -203,7 +214,7 @@ Example 3: using in the container ``rclone.conf`` from your host

`dogImages.zip <https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/dogImages.zip>`_ file is expected to be in ``/Datasets/dogs_breed/data/raw``

Example 4: ``rclone.conf`` with `DEEP-Nextcloud <https://nc.deep-hybrid-datacloud.eu/>`_ configured as ``deepnc`` remote storage:
**Example 4:** ``rclone.conf`` with `DEEP-Nextcloud <https://nc.deep-hybrid-datacloud.eu/>`_ configured as ``deepnc`` remote storage:
::
[deepnc]
type = webdav
Expand All @@ -213,7 +224,7 @@ Example 4: ``rclone.conf`` with `DEEP-Nextcloud <https://nc.deep-hybrid-dataclou
pass = YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY


Example 5: ``rclone.conf`` with Google Drive configured as ``deepnc`` remote storage:
**Example 5:** ``rclone.conf`` with Google Drive configured as ``deepnc`` remote storage:
::
[deepnc]
type = drive
Expand All @@ -223,12 +234,13 @@ Example 5: ``rclone.conf`` with Google Drive configured as ``deepnc`` remote sto

.. note:: Check `rclone <https://rclone.org/>`_ documentation on how to configure different types of remote storage.

**3. Connect to a remote storage by passing rclone configuration parameters as environment settings**
Connect to a remote storage by passing rclone configuration parameters as environment settings
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

It is also possible to pass necessary configuration as environment settings during instantiation of the container,
best is to create a runnable bash script:

Example 6: connecting `DEEP-Nextcloud <https://nc.deep-hybrid-datacloud.eu/>`_ remote storage
**Example 6:** connecting `DEEP-Nextcloud <https://nc.deep-hybrid-datacloud.eu/>`_ remote storage

.. code-block:: bash
Expand Down

0 comments on commit e114f4f

Please sign in to comment.