update docs to v2 - ignacio

source/index.rst

@@ -30,6 +30,16 @@ following section.
 
    technical/index
 
+.. admonition:: Useful project links
+   :class: important
+
+    * `DEEP IAM <https://iam.deep-hybrid-datacloud.eu/>`_
+    * `DEEP Open Catalog - Marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_
+    * DEEP Training dashboard (`Standard <https://train.deep-hybrid-datacloud.eu/>`_, `Advanced <https://deep-paas.cloud.ba.infn.it/>`_)
+    * `DEEP Nextcloud <https://nc.deep-hybrid-datacloud.eu>`_
+    * `Official GitHub <https://github.com/deephdc>`_
+    * `Official DockerHub <https://hub.docker.com/u/deephdc/>`_
+
 Indices and tables
 ==================
 

source/user/howto/add-to-DEEP-marketplace.rst

@@ -6,63 +6,91 @@ This document describes how to add your trained service or model to the DEEP mar
 Creating the Github repositories
 --------------------------------
 
-You model must have a repository to host the code and a repository to host the Dockerfiles. Both these repositories can he hosted under your personal Github account. Naming conventions are that the Docker repo name is the same as the code repo name with the prefix ``DEEP-OC-``.
+You model must have a repository to host the code and a repository to host the Dockerfiles.
+Both these repositories can he hosted under your personal Github account.
+Naming conventions enforce that the Docker repo name is the same as the code repo name with the prefix ``DEEP-OC-``.
 
 A typical example of this can be:
 
-* `deephdc/image-classification-tf <https://github.com/deephdc/image-classification-tf>`_ - The Github repo hosting the code of an image classfication model.
-* `deephdc/DEEP-OC-image-classification-tf <https://github.com/deephdc/DEEP-OC-image-classification-tf>`_ - The Github repo hosting the Dockerfiles of the image classification model.
+* `deephdc/image-classification-tf <https://github.com/deephdc/image-classification-tf>`_ -
+  The Github repo hosting the code of an image classification model.
+* `deephdc/DEEP-OC-image-classification-tf <https://github.com/deephdc/DEEP-OC-image-classification-tf>`_ -
+  The Github repo hosting the Dockerfiles of the image classification model.
 
-In case you are only developing a service based on an already existing model (like for example developing an animal classifier based on the image-classification-tf module) you only need to create the Docker repo.
+In case you are only developing a service based on an already existing module (like for example developing an animal
+classifier based on the image-classification-tf module) you only need to create the Docker repo.
 
 The code repo
 ^^^^^^^^^^^^^
 
-This is the repo containing the code of your model. If you are adding a service (ie. a trained model) the weights of the trained model must be stored in a location accessible over a network connection, so that your container can download them upon creation. A few MUSTs your code has to comply with in order to ensure compatibility and ease of use:
+This is the repo containing the code of your model. If you are adding a service (ie. a trained model) the weights of
+the trained model must be stored in a location accessible over a network connection, so that your container can download
+them upon creation.
 
-* your code must be packaged in order to be ``pip`` installable. This should be the default behaviour if you used the :doc:`DEEP cookiecutter template <../overview/cookiecutter-template>` to develop your code.
-* your code must be integrated with the DEEPaaS API. Check :ref:`this guide <user/overview/api:Integrate your model with the API>` on how to do this.
+A few MUSTs your code has to comply with in order to ensure compatibility and ease of use:
+
+* your code must be packaged in order to be ``pip`` installable. This should be the default behaviour
+  if you used the :doc:`DEEP cookiecutter template <../overview/cookiecutter-template>` to develop your code.
+* your code must be integrated with the DEEPaaS API.
+  Check :ref:`this guide <user/overview/api:Integrate your model with the API>` on how to do this.
 
 The Docker repo
 ^^^^^^^^^^^^^^^
 
-This repo has to contain at least the following files (see the `Generic container <https://github.com/deephdc/DEEP-OC-generic-container>`_ for a template):
+If you used the :doc:`DEEP cookiecutter template <../overview/cookiecutter-template>` to develop your code, a
+template of this repo should have been created alongside the template of your code.
+
+This repo has to contain at least the following files (see the `Generic container <https://github.com/deephdc/DEEP-OC-generic-container>`_
+for a template):
 
 * ``Dockerfile``
 
-   This is the file to build a container from your application. If you developed your application from the :doc:`DEEP cookiecutter template <../overview/cookiecutter-template>` you should have a draft of this file under the ``./docker`` folder (although you might need to add additional code depending on the requirements of your model).
+    This is the file used to build a container from your application. If you developed your application from the
+    :doc:`DEEP cookiecutter template <../overview/cookiecutter-template>` you should have a draft of this file
+    (although you might need to add additional code depending on the requirements of your model).
 
-   If you are adding a service instead of a model, it is good practice to draw inspiration from the Dockerfiles of other services derived from the same model (for example the `plant classification Dockerfile <https://github.com/deephdc/DEEP-OC-plants-classification-tf/blob/master/Dockerfile>`_ derived from the `image classification model <https://github.com/deephdc/DEEP-OC-image-classification-tf>`_).
+    If you are adding a service derived from an existing module, it is good practice to draw inspiration from the
+    Dockerfiles of the module or the services derived from that module (see for example the
+    `plant classification Dockerfile <https://github.com/deephdc/DEEP-OC-plants-classification-tf/blob/master/Dockerfile>`_
+    derived from the `image classification model <https://github.com/deephdc/DEEP-OC-image-classification-tf>`_).
 
-   Some steps common to all Dockerfiles include cloning the model code repo, pip installing the DEEPaaS API, installing rclone and downloading the trained weights if you are adding a service. For the details of all these steps please refer to this `Dockerfile example <https://github.com/deephdc/DEEP-OC-plants-classification-tf/blob/master/Dockerfile>`_.
+    Some steps common to all Dockerfiles include cloning the model code repo, pip installing the DEEPaaS API,
+    installing rclone and downloading the trained weights if you are adding a service.
+    For the details of all these steps please refer to this `Dockerfile example <https://github.com/deephdc/DEEP-OC-image-classification-tf/blob/master/Dockerfile>`_.
 
 * ``Jenkinsfile``
 
-   This is the file that runs the Jenkins pipeline. You can copy this `Jenkinsfile example <https://github.com/deephdc/DEEP-OC-plants-classification-tf/blob/master/Jenkinsfile>`_ replacing the repo names with your own Docker repo name.
+    This is the file that runs the Jenkins pipeline. You can copy this `Jenkinsfile example <https://github.com/deephdc/DEEP-OC-image-classification-tf/blob/master/Jenkinsfile>`_
+    replacing the repo names with your own Docker repo name.
 
 * ``metadata.json``
 
-   This file contains the information that is going to be displayed in the Marketplace. You can build your own starting from this `metadata.json example <https://github.com/deephdc/DEEP-OC-plants-classification-tf/blob/master/metadata.json>`_
-   This metadata will be validated during integration tests when the PR is accepted but you can manually `validate the metadata <https://github.com/deephdc/schema4deep>`_  beforehand by running:
+    This file contains the information that is going to be displayed in the Marketplace. You can build your own starting
+    from this `metadata.json example <https://github.com/deephdc/DEEP-OC-image-classification-tf/blob/master/metadata.json>`_.
+    This metadata will be validated during integration tests when the PR is accepted but you can manually
+    `validate the metadata <https://github.com/deephdc/schema4deep>`_  beforehand by running:
 
-   .. code-block:: console
+    .. code-block:: console
 
-    pip install git+https://github.com/deephdc/schema4apps
-    deep-app-schema-validator metadata.json
+        pip install git+https://github.com/deephdc/schema4apps
+        deep-app-schema-validator metadata.json
 
 Making the Pull Request
 -----------------------
 
 Once your repos are set it's time to make a PR to add your model to the marketplace!
-For this you have to fork the code of the DEEP marketplace (`deephdc/deephdc.github.io <https://github.com/deephdc/deephdc.github.io>`_) and add your Docker repo name at the end of the ``project_apps.yml`` file in the **pelican** branch.
+For this you have to fork the code of the DEEP catalog repo (`deephdc/deep-oc <https://github.com/deephdc/deep-oc>`_)
+and add your Docker repo name at the end of the ``project_apps.yml``.
 
 .. code-block:: console
 
-    git clone -b pelican https://github.com/[my-github-fork]
+    git clone https://github.com/[my-github-fork]
     cd [my-github-fork]
     echo '- module: https://github.com/[my-account-name]/DEEP-OC-[my-app-name]' >> project_apps.yml
     git commit -a -m "adding new module to the catalogue"
     git push
 
-Once the changes are done, make a PR of your fork to the original repo (again to the pelican branch) and wait for approval.
+You can also make it `online on GitHub <https://github.com/deephdc/deep-oc/edit/master/MODULES.yml>`_.
+
+Once the changes are done, make a PR of your fork to the original repo and wait for approval.
 Check the `GitHub Standard Fork & Pull Request Workflow <https://gist.github.com/Chaser324/ce0505fbed06b947d962>`_ in case of doubt.

source/user/howto/index.rst

@@ -9,8 +9,8 @@ HowTo's
    Develop a model <develop-model>
    Train a model locally <train-model-locally>
    Train a model remotely <train-model-remotely>
-   Test a service locally <try-service-locally>
-   Add model/service to the DEEP marketplace <add-to-DEEP-marketplace>
+   Perform inference locally <inference-locally>
+   Add module to the DEEP marketplace <add-to-DEEP-marketplace>
    Use rclone <rclone>
    Install and configure oidc-agent <oidc-agent>
    Video demos <video-demos>

source/user/howto/inference-locally.rst

@@ -0,0 +1,49 @@
+.. highlight:: console
+
+*********************
+Try a service locally
+*********************
+
+
+.. admonition:: Requirements
+
+    * having `Docker <https://www.docker.com>`_ installed. For an up-to-date installation please follow
+      the `official Docker installation guide <https://docs.docker.com/install>`_.
+
+
+1. Choose your module
+---------------------
+
+The first step is to choose a module from the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_.
+For educational purposes we are going to use a `general model to identify images <https://marketplace.deep-hybrid-datacloud.eu/modules/train-an-image-classifier.html>`_. This will allow us to see the general workflow.
+
+Once we have chosen the model at the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_ we will
+find that it has an associated docker container in `DockerHub <https://hub.docker.com/u/deephdc/>`_. For example, in the
+example we are running here, the container would be ``deephdc/deep-oc-image-classification-tf``. This means that to pull the
+docker image and run it you should:
+
+.. code-block:: console
+
+    $ docker pull deephdc/deep-oc-image-classification-tf
+
+
+2. Launch the API and predict
+-----------------------------
+
+Run the container with::
+
+	$ docker run -ti -p 5000:5000 deephdc/deep-oc-image-classification-tf
+
+Once running, point your browser to http://127.0.0.1:5000/ui and you will see the API documentation, where you can
+test the module's functionality, as well as perform other actions.
+
+.. image:: ../../_static/deepaas.png
+  :width: 500
+
+Go to the  ``predict()`` function and upload the file/data you want to predict (in the case of the image classifier
+this should be an image file). The appropriate data formats of the files you have to upload are often discussed
+in the module's Marketplace page.
+
+The response from the ``predict()`` function will vary from module to module but usually consists on a JSON dict
+with the predictions. For example the image classifier return a list of predicted classes along with predicted accuracy.
+Other modules might return files instead of a JSON.

source/user/howto/oidc-agent.rst

@@ -1,46 +1,39 @@
 .. highlight:: console
 
-********************************
-Install and configure oidc-agent
-********************************
+********************
+Configure oidc-agent
+********************
 
-1. Installing oidc-agent
-------------------------
-oidc-agent is a tool to manage OpenID Connect tokens and make them easily usable from the command line. Installation instructions and full documentation can be found `here <https://indigo-dc.gitbooks.io/oidc-agent/>`_.
+`oidc-agent <https://indigo-dc.gitbook.io/oidc-agent/>`_ is a tool to manage OpenID Connect tokens
+and make them easily usable from the command line.
 
-2. Configuring oidc-agent with DEEP-IAM
----------------------------------------------------
+.. admonition:: Requirements
 
-.. admonition:: Prerequisites
+    * having a `DEEP IAM <https://iam.deep-hybrid-datacloud.eu/>`_ account
+    * having the oidc-agent installed (follow the `official installation guide <https://indigo-dc.gitbook.io/oidc-agent/install>`_).
 
-    * `DEEP-IAM <https://iam.deep-hybrid-datacloud.eu/>`_ registration
-
-
-* Start oidc-agent::
+Start oidc-agent::
 
 	$ eval $(oidc-agent)
-
-* Run::
-
 	$ oidc-gen
 
 You will be asked for the name of the account to configure. Let's call it **deep-iam**.
-After that you will be asked for the additional client-name-identifier, you should choose the option::
+After that you will be asked for the additional *client-name-identifier*, you should choose the option::
 
 		[2] https://iam.deep-hybrid-datacloud.eu/
 
-Then just click Enter to accept the default values for Space delimited list of scopes [openid profile offline_access].
-
-* After that, if everything has worked properly, you should see the following messages::
+Then just click ``Enter`` to accept the default values for ``Space delimited list of scopes [openid profile offline_access]``.
+After that, if everything has worked properly, you should see the following messages::
 
 	Registering Client ...
 	Generating account configuration ...
 	accepted
 
-* At this point you will be given a URL. You should visit it in the browser of your choice  in order to continue and approve the registered client.
-* For this you will have to login into your DEEP-IAM account and accept the permissions you are asked for.
+At this point you will be given a URL. You should visit it in the browser of your choice in order to continue and
+approve the registered client. For this you will have to login into your DEEP-IAM account and accept the permissions
+you are asked for.
 
-* Once you have done this you will see the following message::
+Once you have done this you will see the following message::
 
 	The generated account config was successfully added to oidc-agent. You don't have to run oidc-add
 
@@ -51,27 +44,19 @@ Next time you want to start oidc-agent from scratch, you will only have to do::
 	Enter encryption password for account config deep-iam: ********
 	success
 
-* You can print the token::
+You can print the token::
 
 	$ oidc-token deep-iam
 
 
-*2.1 Usage with orchent*
+**Usage with orchent**
 
-* You should set OIDC_SOCK (this is not needed, if you did it before)::
+You should set OIDC_SOCK (this is not needed, if you did it before)::
 
 	$ eval $(oidc-agent)
         $ oidc-add deep-iam
 
-* Set the agent account to be used with orchent::
+Set the agent account to be used with orchent and the ORCHENT_URL::
 
 	$ export ORCHENT_AGENT_ACCOUNT=deep-iam
-
-* You also need to set ORCHENT_URL, e.g::
-
 	$ export ORCHENT_URL="https://paas.cloud.cnaf.infn.it/orchestrator"
-
-
-
-
-