Merge pull request #13 from deephdc/ignacio-br0

Fix urls + warnings

source/user/howto/develop-model.rst

@@ -111,11 +111,11 @@ To create a python installable package the initial directory structure should lo
                 README
 
 * The top level directory will be the root of your repo, e.g. your_model_package.git. The subdir, also called your_model_package, is the actual python module. 
-* ``setup.py`` is the build script for setuptools. It tells setuptools about your package (such as the name and version) as well as which code files to include. You can find an example of a setup.py file `here <https://github.com/deephdc/image-classification-tf/blob/master/setup.py>`_. For the official documentation on how to write your setup script, you can go `here <https://docs.python.org/2/distutils/setupscript.html>`_.
-* ``setup.cfg`` can be used to get some information from the user, or from the user's system in order to proceed. Configuration files also let you providedefault values for any command option. An example of a setup.cfg file can be found `here <https://github.com/deephdc/image-classification-tf/blob/master/setup.cfg>`_. The official python documentation on the setup configuration file can be found `here <https://docs.python.org/3/distutils/configfile.html>`_.
-* ``requirements.txt`` contains any external requirement needed to run the package. You can see an example of a requirements file `here <https://github.com/deephdc/image-classification-tf/blob/master/requirements.txt>`_. An example of a requirements file can be found `here <https://github.com/deephdc/image-classification-tf/blob/master/requirements.txt>`_.
+* ``setup.py`` is the build script for setuptools. It tells setuptools about your package (such as the name and version) as well as which code files to include. You can find an example of a setup.py file `here <https://github.com/deephdc/image-classification-tf/blob/master/setup.py>`__. For the official documentation on how to write your setup script, you can go `here <https://docs.python.org/2/distutils/setupscript.html>`__.
+* ``setup.cfg`` can be used to get some information from the user, or from the user's system in order to proceed. Configuration files also let you providedefault values for any command option. An example of a setup.cfg file can be found `here <https://github.com/deephdc/image-classification-tf/blob/master/setup.cfg>`__. The official python documentation on the setup configuration file can be found `here <https://docs.python.org/3/distutils/configfile.html>`__.
+* ``requirements.txt`` contains any external requirement needed to run the package. You can see an example of a requirements file `here <https://github.com/deephdc/image-classification-tf/blob/master/requirements.txt>`__. An example of a requirements file can be found `here <https://github.com/deephdc/image-classification-tf/blob/master/requirements.txt>`_.
 * The ``README`` file will contain information on how to run the package or anything else that you may find useful for someone running your package.
-* ``LICENSE`` It’s important for every package uploaded to the Python Package Index to include a license. This tells users who install your package the terms under which they can use your package. For help choosing a license, go `here <https://choosealicense.com/>`_.
+* ``LICENSE`` It’s important for every package uploaded to the Python Package Index to include a license. This tells users who install your package the terms under which they can use your package. For help choosing a license, go `here <https://choosealicense.com/>`__.
 
 To see how to install your model package, check the Dockerfile in the next section.
 
@@ -155,14 +155,14 @@ The simplest Dockerfile could look like this::
     	CMD deepaas-run --listen-ip 0.0.0.0
 
 
-For more details on rclone or on DEEPaas API you can check :doc:`here <rclone>` and `here <https://github.com/indigo-dc/DEEPaaS>`_ respectively.
+For more details on rclone or on DEEPaas API you can check :doc:`here <rclone>` and `here <https://github.com/indigo-dc/DEEPaaS>`__ respectively.
 
-If you want to see an example of a more complex Dockerfile, you can check it `here <https://github.com/indigo-dc/DEEP-OC-image-classification-tf/blob/master/Dockerfile>`_.
+If you want to see an example of a more complex Dockerfile, you can check it `here <https://github.com/indigo-dc/DEEP-OC-image-classification-tf/blob/master/Dockerfile>`__.
 
 In order to compile the Dockerfile, you should choose a name for the container and use the docker build command::
 	
 	docker build -t your_container_name -f Dockerfile
 
 
-You can then upload it to Docker hub so that you can download the already compiled image directly. To do so, follow the instructions `here <https://docs.docker.com/docker-hub/repos/>`_.
+You can then upload it to Docker hub so that you can download the already compiled image directly. To do so, follow the instructions `here <https://docs.docker.com/docker-hub/repos/>`__.
 

source/user/howto/train-model-locally.rst

@@ -2,44 +2,45 @@
 
 .. highlight:: console
 
-
+*********************
 Train a model locally
----------------------
+*********************
 
 This is a step by step guide on how to train a model from the Marketplace with your own dataset.
 
 
 1. Get Docker
-=============
+-------------
 
 The first step is having `Docker <https://www.docker.com>`_ installed. To have an up-to-date installation please follow
 the `official Docker installation guide <https://docs.docker.com/install>`_.
 
 
-1. Search for a model in the marketplace
-========================================
-
-The first step is to choose a model from the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_.  For educational purposes we are going to use a `general model to identify images <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-image-classification-tensorflow.html>`_. This will allow us to see the general workflow.
+2. Search for a model in the marketplace
+----------------------------------------
 
+The first step is to choose a model from the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_.  For educational purposes we are going to use a `general model to identify images <https://marketplace.deep-hybrid-datacloud.eu/modules/deep-oc-image-classification-tensorflow.html>`_. This will allow us to see the general workflow.
 
 
 3. Get the model
-================
-
-.. todo:: Check that names of the docker containers are correct for the image classifier example.
+----------------
 
-Once we have chosen the model at the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_ we will find that it has an associated docker container in `DockerHub <https://hub.docker.com/u/deephdc/>`_. For example, in the example we are running here, the container would be deephdc/deep-oc-image-classification-tf. This means that to pull the docker image and run it you should ::
+Once we have chosen the model at the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_ we will
+find that it has an associated docker container in `DockerHub <https://hub.docker.com/u/deephdc/>`_. For example, in the
+example we are running here, the container would be deephdc/deep-oc-image-classification-tf. This means that to pull the
+docker image and run it you should:
 
 .. code-block:: console
 
     $ docker pull deephdc/deep-oc-image-classification-tf
 
 
-
 4. Export rclone.conf file
+--------------------------
 
-When running the container you should export the rclone.conf file so that it can be reached from within the docker. You can see an example on how to do this here::
+When running the container you should export the rclone.conf file so that it can be reached from within the docker. You can see an example on how to do this here
 
+.. code-block:: console
 
 	$ docker run -ti -v  -p 5000:5000 -p 6006:6006 -v  host_path_to_rclone.conf:/root/.config/rclone/rclone.conf <your-docker-image>
 
@@ -48,20 +49,26 @@ You can see this last step explained more in detail :doc:`here <rclone>`.
 We are using the port ``5000`` to deploy the API and the port ``6006`` to monitor the training (for example using
 `Tensorboard <https://www.tensorflow.org/guide/summaries_and_tensorboard>`_).
 
-4. Upload your data to storage resources
-========================================
+
+
+5. Upload your data to storage resources
+----------------------------------------
 
 To run locally you have two options:
 
 1. Have your data stored locally
+================================
+
+You should make sure that you export inside of the container all the folders you need for the training
 
-You should make sure that you export inside of the container all the folders you need for the training::
+.. code-block:: console
 
 	$ docker run -ti -v  -p 5000:5000 -p 6006:6006 -v path_to_local_folder:path_to_docker_folder -v host_path_to_rclone.conf:/root/.config/rclone/rclone.conf <your-docker-image>
 
 2. Have your data at a remote storage resource
+==============================================
 
-* Nextcloud
+* **Nextcloud**
 
 If you have the files you need for the training stored in Nextcloud you need first to login into `DEEP-Nextcloud  <https://nc.deep-hybrid-datacloud.eu/login>`_ with your DEEP-IAM credentials. Then you have to go to: **(1) Settings (top right corner)** |rarr|  **(2) Security**  |rarr|  **(3) Devices & sessions**
 
@@ -70,7 +77,9 @@ If you have the files you need for the training stored in Nextcloud you need fir
 
 Set a name for your application (for this example it will be **deepnc**) and clik on **Create new app password**. This will generate <your_nextcloud_username> and <your_nextcloud_password> that you should to include in your rclone.conf file (see  :doc:`more details <rclone>`.).
 
-Now you can create the folders that you need in order to store the inputs needed for the training and to retrieve the output. In order to be able to see these folders locally you should run either on your local host or on the docker container::
+Now you can create the folders that you need in order to store the inputs needed for the training and to retrieve the output. In order to be able to see these folders locally you should run either on your local host or on the docker container
+
+.. code-block:: python
 
 	import subprocess
 
@@ -80,23 +89,24 @@ Now you can create the folders that you need in order to store the inputs needed
 	results= subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 	output, error = result.communicate()
 
-* Google Drive
+* **Google Drive**
 
 TBC
 
-* Dropbox
+* **Dropbox**
 
 TBC
 
-5. Train the model
-==================
+
+6. Train the model
+------------------
 
 Now comes the fun! Go to `<http://0.0.0.0:5000>`_ and look for the ``train`` method. Modify the training parameters you wish to
 change and execute. If some kind of monitorization tool is available for this model you will be able to follow the training
 progress from `<http://0.0.0.0:6006>`_.
 
 
-6. Testing the training
-=======================
+7. Testing the training
+-----------------------
 
 Once the training has finished, you can directly test it by clicking on the ``predict`` method. There you can either upload the image your want to classify or give a URL to it. 

source/user/howto/train-model-remotely.rst

@@ -6,13 +6,13 @@
 Train a model remotely 
 **********************
 
-This is a step by step guide on how to train a general model from the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_ with your own dataset. 
+This is a step by step guide on how to train a general model from the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`__ with your own dataset.
 
 
 1. Choose a model
 -----------------
 
-The first step is to choose a model from the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_.  For educational purposes We are going to use a `general model to identify images <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-image-classification-tensorflow.html>`_. Some of the model dependent details can change if using another model, but this tutorial will provide a general overview of the workflow to follow when using any of the models in the Marketplace. A demo showing the different steps in this HowTo has also be recorded and you can find it here :ref:`here <video-demo_train-remotely>`.
+The first step is to choose a model from the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`__.  For educational purposes We are going to use a `general model to identify images <https://marketplace.deep-hybrid-datacloud.eu/modules/deep-oc-image-classification-tensorflow.html>`_. Some of the model dependent details can change if using another model, but this tutorial will provide a general overview of the workflow to follow when using any of the models in the Marketplace. A demo showing the different steps in this HowTo has also be recorded and you can find it here :ref:`here <video-demo_train-remotely>`.
 
 
 
@@ -27,16 +27,16 @@ Before being able to run your model at the Pilot Infraestructure you should firs
     * Install oidc-agent and configure it for DEEP-IAM. Instructions for this can be found  :doc:`here <oidc-agent>`. Make sure you follow the instructions in the *Usage with orchent* section. 
     * Install `orchent <https://github.com/indigo-dc/orchent/releases>`_ tool
 
-For this example we are going to use `DEEP-Nextcloud <https://nc.deep-hybrid-datacloud.eu>`_ for storing you data. This means you also have to:
+For this example we are going to use `DEEP-Nextcloud <https://nc.deep-hybrid-datacloud.eu>`__ for storing you data. This means you also have to:
 
-    * Register at `DEEP-Nextcloud <https://nc.deep-hybrid-datacloud.eu>`_
+    * Register at `DEEP-Nextcloud <https://nc.deep-hybrid-datacloud.eu>`__
     * Follow the **Nextcloud configuration for rclone** :doc:`here <rclone>`. This will give you <your_nextcloud_username> and <your_nextcloud_password>.
 
 
 3. Upload your files to Nextcloud
 ---------------------------------
 
-Upload the files you need for the training to DEEP-Nextcloud. For this, after login into `DEEP-Nextcloud  <https://nc.deep-hybrid-datacloud.eu/login>`_ with your DEEP-IAM credentials, go to: **(1) Settings (top right corner)** |rarr|  **(2) Security**  |rarr|  **(3) Devices & sessions**
+Upload the files you need for the training to DEEP-Nextcloud. For this, after login into `DEEP-Nextcloud  <https://nc.deep-hybrid-datacloud.eu/login>`__ with your DEEP-IAM credentials, go to: **(1) Settings (top right corner)** |rarr|  **(2) Security**  |rarr|  **(3) Devices & sessions**
 
 
 .. image:: ../../_static/nc-access.png
@@ -88,7 +88,7 @@ Remember that the first line should include the name of the remote Nextcloud app
 6. Prepare your TOSCA file
 --------------------------
 
-In the orchent submission script there is a call to a TOSCA file (TOSCA.yml). A generic template can be found `here <https://github.com/indigo-dc/tosca-templates/blob/master/deep-oc/deep-oc-mesos-gpu-webdav.yml>`_. The sections that should be modified are the following (TOSCA experts may modify the rest of the template to their will.)
+In the orchent submission script there is a call to a TOSCA file (TOSCA.yml). A generic template can be found `here <https://github.com/indigo-dc/tosca-templates/blob/master/deep-oc/deep-oc-mesos-gpu-webdav.yml>`__. The sections that should be modified are the following (TOSCA experts may modify the rest of the template to their will.)
 
 * Docker image to deploy. In this case we will be using deephdc/deep-oc-image-classification-tf::
 
@@ -106,7 +106,7 @@ In the orchent submission script there is a call to a TOSCA file (TOSCA.yml). A
       required: yes
       default: "/srv/image-classification-tf/rclone.conf"
 
-For further TOSCA templates examples you can go `here <https://github.com/indigo-dc/tosca-templates/tree/master/deep-oc>`_. 
+For further TOSCA templates examples you can go `here <https://github.com/indigo-dc/tosca-templates/tree/master/deep-oc>`__.
 
 
 

source/user/howto/try-service-locally.rst

@@ -23,9 +23,9 @@ The next step is to look for a service in the `DEEP Open Catalog marketplace <ht
 ----------------
 
 You will find that each model has an associate Docker container in DockerHub. For eaxample
-`DEEP OC Image Classification <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-image-classification-tensorflow.html>`_
+`DEEP OC Image Classification <https://marketplace.deep-hybrid-datacloud.eu/modules/deep-oc-image-classification-tensorflow.html>`_
 is associated with `deephdc/deep-oc-image-classification-tf <https://hub.docker.com/r/deephdc/deep-oc-image-classification-tf>`_,
-`DEEP OC Massive Online Data Streams <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-massive-online-data-streams.html>`_
+`DEEP OC Massive Online Data Streams <https://marketplace.deep-hybrid-datacloud.eu/modules/deep-oc-massive-online-data-streams.html>`_
 is associated with `deephdc/deep-oc-mods <https://hub.docker.com/r/deephdc/deep-oc-mods>`_, etc.
 
 Let call the service you selected ``deep-oc-service_of_interest``.

source/user/index.rst

@@ -1,8 +1,6 @@
 User documentation
 ==================
 
-.. todo:: Documentation is being written at this moment.
-
 
 .. raw:: html
 

source/user/modules/toy_model-dog_breeds.rst

@@ -73,7 +73,9 @@ to a persistent volume. You can either use your local directories or connect you
 Original dataset consists of dog's images for 133 breeds. The images are compressed in 
 one `dogImages.zip <https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/dogImages.zip>`_  file. 
 The archive contains three directories for ``train``, ``test``, and ``valid`` datasets:
-::
+
+.. code-block:: console
+
     test/
         001.Affenpinscher/
         002.Afghan_hound/
@@ -98,7 +100,9 @@ If you want to use your own dataset then it has to follow similar structure.
 
 
 **If local directories are mounted into the container, the following directory structure is suggested:**
-::
+
+.. code-block:: console
+
     data/
         dogImages/
         raw/
@@ -116,7 +120,9 @@ In the 'local' case, you place `dogImages.zip <https://s3-us-west-1.amazonaws.co
 which makes it available in ``/srv/dogs_breed_det/data/raw``.
 
 **If you connect a remote storage, the following directories have to be created there:**
-::
+
+.. code-block:: console
+
     /Datasets/dogs_breed/data
     /Datasets/dogs_breed/data/dogImages
     /Datasets/dogs_breed/data/raw
@@ -182,14 +188,16 @@ Mount local host directories
 """""""""""""""""""""""""""""
 
 **Example 1 (GPU, default):**
-::
+
+.. code-block:: console
 
     docker run -ti -p 5000:5000 -v ~/data:/srv/dogs_breed_det/data \
     -v ~/models:/srv/dogs_breed_det/models \
     deephdc/deep-oc-dogs_breed_det deepaas-run --listen-ip=0.0.0.0
 
 **Example 2 (CPU):**
-::
+
+.. code-block:: console
 
     docker run -ti -p 5000:5000 -v ~/data:/srv/dogs_breed_det/data \
     -v ~/models:/srv/dogs_breed_det/models \
@@ -207,7 +215,9 @@ If you are familiar with the `rclone <https://rclone.org/>`_ tool, you probably
 You can rename one of the pre-configured remote storages to ``deepnc``, then mount host directory with your ``rclone.conf`` file into the container:
 
 **Example 3:** using in the container ``rclone.conf`` from your host
-::
+
+.. code-block:: console
+
 
     docker run -ti -p 5000:5000 -v $HOSTDIR_WITH_RCLONE_CONF:/srv/rclone \
     -e RCLONE_CONFIG=/srv/rclone/rclone.conf \
@@ -216,7 +226,9 @@ You can rename one of the pre-configured remote storages to ``deepnc``, then mou
 `dogImages.zip <https://s3-us-west-1.amazonaws.com/udacity-aind/dog-project/dogImages.zip>`_  file is expected to be in ``/Datasets/dogs_breed/data/raw``
 
 **Example 4:** ``rclone.conf`` with `DEEP-Nextcloud <https://nc.deep-hybrid-datacloud.eu/>`_ configured as ``deepnc`` remote storage:
-::
+
+.. code-block:: console
+
     [deepnc]
     type = webdav
     url = https://nc.deep-hybrid-datacloud.eu/remote.php/webdav/
@@ -226,7 +238,9 @@ You can rename one of the pre-configured remote storages to ``deepnc``, then mou
 
 
 **Example 5:** ``rclone.conf`` with Google Drive configured as ``deepnc`` remote storage:
-::
+
+.. code-block:: console
+
     [deepnc]
     type = drive
     scope = drive