Skip to content
This repository has been archived by the owner on Jun 14, 2023. It is now read-only.

Commit

Permalink
Merge pull request #34 from deephdc/ignacio-br1
Browse files Browse the repository at this point in the history
Ignacio br1
  • Loading branch information
laramaktub committed Oct 21, 2022
2 parents 93c134f + 182e481 commit f92b7dd
Show file tree
Hide file tree
Showing 19 changed files with 90 additions and 93 deletions.
6 changes: 3 additions & 3 deletions source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ Welcome to DEEP-Hybrid-DataCloud's documentation!
* `Github <https://github.com/deephdc>`__
* `DockerHub <https://hub.docker.com/u/deephdc/>`__
* `Docs <https://docs.deep-hybrid-datacloud.eu/en/latest/>`__
* `NextCloud <https://nc.deep-hybrid-datacloud.eu/>`__
* `DEEP IAM <https://iam.deep-hybrid-datacloud.eu/>`_
* `NextCloud <https://data-deep.a.incd.pt/>`__
* `DEEP IAM <https://iam.deep-hybrid-datacloud.eu/>`__

User documentation
------------------
Expand All @@ -28,7 +28,7 @@ Component documentation

Individual components' documentation can be found here:

* `DEEPaaS documentation <https://docs.deep-hybrid-datacloud.eu/projects/deepaas/>`_
* `DEEPaaS documentation <https://docs.deep-hybrid-datacloud.eu/projects/deepaas/>`__


Technical documentation
Expand Down
6 changes: 3 additions & 3 deletions source/technical/howto-developers/deploy-orchent.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ Dashboard).

.. admonition:: Requirements

* `oidc-agent <https://github.com/indigo-dc/oidc-agent/releases>`_ installed and configured for `DEEP-IAM <https://iam.deep-hybrid-datacloud.eu/>`_ (see :doc:`Configure oidc-agent <oidc-agent>`).
* `orchent <https://github.com/indigo-dc/orchent/releases>`_ tool
* `oidc-agent <https://github.com/indigo-dc/oidc-agent/releases>`__ installed and configured for `DEEP-IAM <https://iam.deep-hybrid-datacloud.eu/>`_ (see :doc:`Configure oidc-agent <oidc-agent>`).
* `orchent <https://github.com/indigo-dc/orchent/releases>`__ tool


Prepare your TOSCA file (optional)
Expand Down Expand Up @@ -53,7 +53,7 @@ You can use the general template, `deep-oc-mesos-webdav.yml <https://github.com/
#!/bin/bash
orchent depcreate ./deep-oc-marathon-webdav.yml '{ "docker_image": "deephdc/deep-oc-image-classification-tf"
"rclone_url": "https://nc.deep-hybrid-datacloud.eu/remote.php/webdav/",
"rclone_url": "https://data-deep.a.incd.pt/remote.php/webdav/",
"rclone_vendor": "nextcloud",
"rclone_conf": "/srv/image-classification-tf/rclone.conf"
"rclone_user": <your_nextcloud_username>
Expand Down
2 changes: 1 addition & 1 deletion source/technical/howto-developers/develop-dashboard.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Develop Dashboard
=================

The `General purpose dashboard <https://paas.cloud.cnaf.infn.it/>`_ allows you to interact with the underling `TOSCA templates <https://github.com/indigo-dc/tosca-templates/tree/master/deep-oc>`_
The `General purpose dashboard <https://paas.cloud.cnaf.infn.it/>`__ allows you to interact with the underling `TOSCA templates <https://github.com/indigo-dc/tosca-templates/tree/master/deep-oc>`_
(which configure the job requirements) instead of modules and deploy more complex topologies (e.g. a kubernetes cluster).
Modules can either use a `general template <https://github.com/indigo-dc/tosca-templates/blob/master/deep-oc/deep-oc-marathon-webdav.yml>`_
or create a dedicated one based on the `existing templates <https://github.com/indigo-dc/tosca-templates/tree/master/deep-oc>`__.
4 changes: 2 additions & 2 deletions source/technical/howto-developers/oidc-agent.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
Configure oidc-agent
====================

`oidc-agent <https://indigo-dc.gitbook.io/oidc-agent/>`_ is a tool to manage OpenID Connect tokens
`oidc-agent <https://indigo-dc.gitbook.io/oidc-agent/>`__ is a tool to manage OpenID Connect tokens
and make them easily usable from the command line.

.. admonition:: Requirements

* having a `DEEP IAM <https://iam.deep-hybrid-datacloud.eu/>`_ account
* having a `DEEP IAM <https://iam.deep-hybrid-datacloud.eu/>`__ account
* having the oidc-agent installed (follow the `official installation guide <https://indigo-dc.gitbook.io/oidc-agent/install>`_).

Start oidc-agent::
Expand Down
6 changes: 3 additions & 3 deletions source/user/howto/develop-model.rst
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,9 @@ Integrating the module in the Marketplace

Once your repos are set, it's time to make a PR to add your model to the marketplace!

For this you have to fork the code of the DEEP catalog repo (`deephdc/deep-oc <https://github.com/deephdc/deep-oc>`_)
For this you have to fork the code of the DEEP catalog repo (`deephdc/deep-oc <https://github.com/deephdc/deep-oc>`__)
and add your Docker repo name at the end of the ``MODULES.yml``.
You can do this directly `online on GitHub <https://github.com/deephdc/deep-oc/edit/master/MODULES.yml>`_ or via the command line:
You can do this directly `online on GitHub <https://github.com/deephdc/deep-oc/edit/master/MODULES.yml>`__ or via the command line:

.. code-block:: console
Expand All @@ -128,7 +128,7 @@ You can do this directly `online on GitHub <https://github.com/deephdc/deep-oc/e
git push
Once the changes are done, make a PR of your fork to the original repo and wait for approval.
Check the `GitHub Standard Fork & Pull Request Workflow <https://gist.github.com/Chaser324/ce0505fbed06b947d962>`_ in case of doubt.
Check the `GitHub Standard Fork & Pull Request Workflow <https://gist.github.com/Chaser324/ce0505fbed06b947d962>`__ in case of doubt.

When your module gets approved, you may need to commit and push a change to ``metadata.json``
in ``DEEP-OC-your_project`` (`ref <https://github.com/deephdc/DEEP-OC-demo_app/blob/726e068d54a05839abe8aef741b3ace8a078ae6f/Jenkinsfile#L104>`__)
Expand Down
14 changes: 7 additions & 7 deletions source/user/howto/inference-locally.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,23 +10,23 @@ Try a service locally

.. admonition:: Requirements

This section requires having `docker <https://docs.docker.com/install/#supported-platforms>`_ installed.
This section requires having `docker <https://docs.docker.com/install/#supported-platforms>`__ installed.

Starting from version 19.03 docker supports NVIDIA GPUs (see `here <https://docs.docker.com/engine/release-notes/>`__ and `here <https://github.com/moby/moby/pull/38828>`__).
If you happen to be using an older version you can give a try to `nvidia-docker <https://github.com/nvidia/nvidia-docker/wiki/Installation-(version-2.0)>`_
If you happen to be using an older version you can give a try to `nvidia-docker <https://github.com/nvidia/nvidia-docker/wiki/Installation-(version-2.0)>`__

If you need to use docker in an environment without root privileges (eg. an HPC cluster)
check `udocker <https://github.com/indigo-dc/udocker/releases>`_ instead of docker.
check `udocker <https://github.com/indigo-dc/udocker/releases>`__ instead of docker.


1. Choose your module
---------------------

The first step is to choose a module from the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_.
For educational purposes we are going to use a `general model to identify images <https://marketplace.deep-hybrid-datacloud.eu/modules/train-an-image-classifier.html>`_. This will allow us to see the general workflow.
The first step is to choose a module from the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`__.
For educational purposes we are going to use a `general model to identify images <https://marketplace.deep-hybrid-datacloud.eu/modules/train-an-image-classifier.html>`__. This will allow us to see the general workflow.

Once we have chosen the model at the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_ we will
find that it has an associated docker container in `DockerHub <https://hub.docker.com/u/deephdc/>`_. For example, in the
Once we have chosen the model at the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`__ we will
find that it has an associated docker container in `DockerHub <https://hub.docker.com/u/deephdc/>`__. For example, in the
example we are running here, the container would be ``deephdc/deep-oc-image-classification-tf``. This means that to pull the
docker image and run it you should:

Expand Down
4 changes: 2 additions & 2 deletions source/user/howto/inference-remotely.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ Try a service remotely

- `Performing inference in the cloud with the DEEP training dashboard <https://www.youtube.com/watch?v=FyELMIr5Wbo&list=PLJ9x9Zk1O-J_UZfNO2uWp2pFMmbwLvzXa&index=4>`__

The first step is to go to the `DEEP as a Service (or DEEPaaS) <https://deepaas.deep-hybrid-datacloud.eu/>`_.
For educational purposes we are going to use a `general model to identify images <https://marketplace.deep-hybrid-datacloud.eu/modules/train-an-image-classifier.html>`_. This will allow us to see the general workflow.
The first step is to go to the `DEEP as a Service (or DEEPaaS) <https://deepaas.deep-hybrid-datacloud.eu/>`__.
For educational purposes we are going to use a `general model to identify images <https://marketplace.deep-hybrid-datacloud.eu/modules/train-an-image-classifier.html>`__. This will allow us to see the general workflow.

Go to ``Swagger Interfaces`` and click ``Swagger UI for "image-classification-tf"``.
You will see the API documentation, where you can test the module's functionality, as well as perform other actions.
Expand Down
12 changes: 6 additions & 6 deletions source/user/howto/rclone.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ How to use rclone
Installation of rclone in Docker image
--------------------------------------

All applications in the `DEEP Open Catalog <https://deephdc.github.io/>`_ are packed in a Docker image and have
`rclone <https://rclone.org/>`_ tool installed by default. If you want to create a Docker containing your own application, you should install rclone
All applications in the `DEEP Open Catalog <https://deephdc.github.io/>`__ are packed in a Docker image and have
`rclone <https://rclone.org/>`__ tool installed by default. If you want to create a Docker containing your own application, you should install rclone
in the container to be able to access the data stored remotely. The following lines are an example of what has to be
added in the Dockerfile when installation is based on Ubuntu. For other Linux flavors, please, refer to
the `rclone official site <https://rclone.org/downloads/>`_ ::
the `rclone official site <https://rclone.org/downloads/>`__ ::

# Install rclone
RUN wget https://downloads.rclone.org/rclone-current-linux-amd64.deb && \
Expand All @@ -33,7 +33,7 @@ Nextcloud configuration for rclone
----------------------------------
.. image:: ../../_static/nc-access.png

After login into `DEEP-Nextcloud <https://nc.deep-hybrid-datacloud.eu/login>`_ with your DEEP-IAM credentials, go to
After login into `DEEP-Nextcloud <https://data-deep.a.incd.pt/>`__ with your DEEP-IAM credentials, go to
(1) **Settings (top right corner)** |rarr| (2) **Security** |rarr| (3) **Devices & sessions**. Set a name for your
application and click on **Create new app password**. You get <user> and <password> credentials. Next, you need to obscure the <password> for use in the rclone config file (``rclone.conf``). For this you do:

Expand All @@ -52,7 +52,7 @@ In order to create the configuration file (``rclone.conf``) for rclone::
choose "n" for "New remote"
choose name for DEEP-Nextcloud, e.g. deep-nextcloud
choose "Type of Storage" \u2192 "Webdav" (24)
provide DEEP-Nextcloud URL for webdav access: https://nc.deep-hybrid-datacloud.eu/remote.php/webdav/
provide DEEP-Nextcloud URL for webdav access: https://data-deep.a.incd.pt/remote.php/webdav/
choose Vendor, Nextcloud (1)
specify "user" (see "Nextcloud configuration for rclone" above). Your username starts with "DEEP-IAM-..."
specify password (see "Nextcloud configuration for rclone" above).
Expand Down Expand Up @@ -105,7 +105,7 @@ A simple call of rclone from python is via ``subprocess.Popen()``
.. important::
When deploying a module on the DEEP Pilot testbed, you pass rclone parameters e.g. ``rclone_user`` and ``rclone_password`` during the deployment.
If you use our `general template <https://github.com/indigo-dc/tosca-templates/blob/master/deep-oc/deep-oc-marathon-webdav.yml>`_ , the name of the remote storage has to be ``rshare`` as in the example above (``rshare:/Datasets/dogs_breed/data``). If you create your own TOSCA template, you need to pay attention on matching these names in your code and in the template (for example, see environment parameters in the `general template <https://github.com/indigo-dc/tosca-templates/blob/master/deep-oc/deep-oc-marathon-webdav.yml>`_ like RCLONE_CONFIG_RSHARE_USER etc).
If you use our `general template <https://github.com/indigo-dc/tosca-templates/blob/master/deep-oc/deep-oc-marathon-webdav.yml>`__ , the name of the remote storage has to be ``rshare`` as in the example above (``rshare:/Datasets/dogs_breed/data``). If you create your own TOSCA template, you need to pay attention on matching these names in your code and in the template (for example, see environment parameters in the `general template <https://github.com/indigo-dc/tosca-templates/blob/master/deep-oc/deep-oc-marathon-webdav.yml>`_ like RCLONE_CONFIG_RSHARE_USER etc).

Advanced examples
^^^^^^^^^^^^^^^^^
Expand Down

0 comments on commit f92b7dd

Please sign in to comment.