Merge pull request #33 from deephdc/ignacio-br1

docs update

.readthedocs.yaml

@@ -0,0 +1,22 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-20.04
+  tools:
+    python: "3.8"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+   configuration: source/conf.py
+
+# Optionally declare the Python requirements required to build your docs
+python:
+   install:
+   - requirements: ./requirements.txt
+

requirements.txt

@@ -1,4 +1,4 @@
-sphinx>=1.6.2 # BSD
-sphinx-markdown-tables
-recommonmark
-sphinx_rtd_theme
+sphinx>=4.5.0
+sphinx-markdown-tables>=0.0.15
+recommonmark>=0.7.1
+sphinx_rtd_theme>=1.0.0

source/technical/howto-developers/pull_request_workflow.rst

@@ -0,0 +1,162 @@
+Module integration workflow for external (non-deephdc) users
+============================================================
+
+In the case the PR to :ref:`integrate a module <user/howto/develop-model:Integrating the module in the Marketplace>` comes from a **non-deephdc** member, remember you have
+to perform the following additional steps for integration:
+
+
+1. Name check
+~~~~~~~~~~~~~
+
+Check the PR change follows the following convention (edit the PR is
+needed):
+
+   ``- module: https://github.com/deephdc/UC-<original_account>-DEEP-OC-<original_repo_name>``
+
+Note: ``UC`` stands for *User Contributed*. So for example a user with
+Docker repo:
+
+- https://github.com/adnaneds/DEEP-OC-unet
+
+should have the following repo in ``deephdc``:
+
+- https://github.com/deephdc/UC-adnaneds-DEEP-OC-unet ⬅ This is the repo that should be included in the PR change to ``MODULES.yml``
+
+2. Fork creation
+~~~~~~~~~~~~~~~~
+
+Create forks of both ``Code repo`` and ``Docker repo`` in the
+``deephdc`` organization. This is done so that our Jenkins plugin can
+monitor the changes of the repo (it can only do so for repos inside
+``deephdc``).
+
+When performming the forks, remember to un-select
+``Copy the master branch only`` as we want to fork all branches.
+
+The forks should use the following naming conventions:
+
+- For code repos: ``UC-<original_account>-<original_repo_name>``
+- For Docker repos:``UC-<original_account>-DEEP-OC-<original_repo_name>``
+
+For example forking:
+
+- https://github.com/adnaneds/unet
+- https://github.com/adnaneds/DEEP-OC-unet
+
+would give the following forks:
+
+- https://github.com/deephdc/UC-adnaneds-unet
+- https://github.com/deephdc/UC-adnaneds-DEEP-OC-unet
+
+In case of doubt check the Jenkins badges in the respective
+``README``\ s of the repos (code and docker). They should display the
+expected repo names for ``deephdc`` forks.
+
+3. Keep the forks updated
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In order to have the forks updated, so that they continously reflect the
+changes in upstream, we are going to use `Github
+Actions <https://github.com/features/actions>`__. So **for each of the
+two forks**, go to ``Actions`` → ``set a workflow yourself``.
+
+Then copy in the editor the following workflow, **remembering to add the
+corresponding UPSTREAM_REPO**. Following the previous example:
+
+- https://github.com/deephdc/UC-adnaneds-unet
+
+  ➡ **UPSTREAM_REPO**: https://github.com/adnaneds/unet.git
+
+- https://github.com/deephdc/UC-adnaneds-DEEP-OC-unet
+
+  ➡ **UPSTREAM_REPO**: https://github.com/adnaneds/DEEP-OC-unet.git
+
+.. code:: yaml
+
+   # Script is (loosely) based on two references:
+   # * https://stackoverflow.com/questions/23793062/can-forks-be-synced-automatically-in-github
+   # * https://stackoverflow.com/questions/69918635/how-to-keep-all-branches-and-tags-in-sync-in-a-fork-or-mirror-repo
+   # [!] Note: Do not do a force push to not overwrite the .github/workflow/main.yml file.
+
+   # We are not using a predefined action (eg. [1]) to not go through the hassle of having to
+   # manage Github Personal Access Tokens for deephdc.
+   # [1]: https://github.com/repo-sync/github-sync
+
+
+   name: Sync fork with upstream
+   on:
+     schedule:
+        # run at max frequency
+       - cron:  '* * * * *'
+
+     # Allows you to run this workflow manually from the Actions tab
+     workflow_dispatch:
+
+   jobs:
+
+     build:
+
+       runs-on: ubuntu-latest
+
+       steps:
+         # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
+         - uses: actions/checkout@v3
+           with:
+             fetch-depth: 0
+
+         - name: Sync fork with upstream
+           run: |
+
+             #########################################################
+             # YOU SHOULD UPDATE THIS LINE
+             #########################################################
+
+             UPSTREAM_REPO=https://github.com/<username>/<reponame>.git
+
+             #########################################################
+
+             # Bot config
+             git config user.name 'github-actions[bot]'
+             git config user.email 'github-actions[bot]@users.noreply.github.com'
+
+             # Add remote
+             git remote add upstream $UPSTREAM_REPO
+             git fetch upstream
+
+             # Keep track of branch names
+             origin_branches=$(git branch -r | grep -v 'HEAD' | grep 'origin/' | cut -f 2 -d '/')
+             upstream_branches=$(git branch -r | grep 'upstream/' | cut -f 2 -d '/')
+
+             old_branches=$(comm -13 <(printf '%s\n' "${upstream_branches[@]}" | LC_ALL=C sort) <(printf '%s\n' "${origin_branches[@]}" | LC_ALL=C sort))
+             new_branches=$(comm -13 <(printf '%s\n' "${origin_branches[@]}" | LC_ALL=C sort) <(printf '%s\n' "${upstream_branches[@]}" | LC_ALL=C sort))
+             existing_branches=$(comm -13 <(printf '%s\n' "${new_branches[@]}" | LC_ALL=C sort) <(printf '%s\n' "${upstream_branches[@]}" | LC_ALL=C sort))
+
+              # Delete old branches from origin
+             echo "# Deleting old branches ..."
+             for tmp_branch in $old_branches; do
+               echo "## Processing $tmp_branch ..."
+               git push origin --delete $tmp_branch
+             done
+
+             # Create origin branches for new upstream branches
+             echo "# Creating new branches ..."
+             for tmp_branch in $new_branches; do
+               echo "## Processing $tmp_branch ..."
+               git checkout -b $tmp_branch upstream/$tmp_branch
+               git push origin
+             done
+
+             # Merge changes from upstream to origin for existing branches
+             echo "# Merging existing branches ..."
+             git config --add checkout.defaultRemote origin
+             for tmp_branch in $existing_branches; do
+               echo "## Processing $tmp_branch ..."
+               git checkout $tmp_branch
+               git merge --no-edit upstream/$tmp_branch
+               git push origin
+             done
+
+             # Sync tags
+             git tag -d $(git tag -l)
+             git fetch upstream --tags --quiet
+             git push origin --tags --force

source/technical/index.rst

@@ -10,6 +10,7 @@ HowTo's (developers)
 .. toctree::
    :maxdepth: 1
 
+   howto-developers/pull_request_workflow.rst
    howto-developers/develop-dashboard.rst
    howto-developers/oidc-agent.rst
    howto-developers/deploy-orchent.rst

source/user/howto/develop-model.rst

@@ -3,6 +3,13 @@
 Develop a model
 ===============
 
+.. admonition:: Useful video demos
+   :class: important
+
+    - `Deploying the DEEP development module (JupyterLab) <https://www.youtube.com/watch?v=J_l_xWiBGNA&list=PLJ9x9Zk1O-J_UZfNO2uWp2pFMmbwLvzXa&index=3>`__
+    - `Training your model in the cloud with the DEEP training dashboard <https://www.youtube.com/watch?v=uqFXrEwtNNs&list=PLJ9x9Zk1O-J_UZfNO2uWp2pFMmbwLvzXa&index=6>`__
+
+
 Setting the framework
 ---------------------
 
@@ -41,6 +48,37 @@ Try running it with python so you get a more detailed debug message.
 
     python api.py
 
+Remember to leave the ``get_metadata()`` function that comes predefined with your module,
+as all modules should have proper metadata.
+
+In order to improve the readability of the code and the overall maintainability of the project,
+we enforce proper Python coding styles (``pep8``) to all modules added to the Marketplace.
+Modules that fail to pass style tests won't be able to build docker images.
+If you want to check if your module pass the tests, go to your project folder and type::
+
+    flake8
+
+There you should see a detailed report of the offending lines (if any).
+If your project has many offending lines, it's recommended using a code formatter tool like
+`Black <https://black.readthedocs.io>`__. It also helps for having a consistent code style
+and minimizing git diffs. Once `installed <https://black.readthedocs.io/en/stable/getting_started.html#installation>`__,
+you can check how Black would have reformatted your code:
+::
+
+    black <code-folder> --diff
+
+Remember you can `turn off Black formatting <https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html?highlight=fmt#code-style>`__
+if you want to keep some sections of your code untouched.
+And, similarly, you can `turn off flake8 testing <https://stackoverflow.com/a/64431741>`__
+in some parts of the code if long lines are really needed.
+
+If you are happy with the changes, you can make them permanent using:
+::
+
+    black <code-folder>
+
+Have a backup before reformatting, just in case!
+
 Once you are fine with the state of your module, push the changes to Github.
 
 
@@ -85,7 +123,7 @@ You can do this directly `online on GitHub <https://github.com/deephdc/deep-oc/e
 
     git clone https://github.com/[my-github-fork]
     cd [my-github-fork]
-    echo '- module: https://github.com/[my-account-name]/DEEP-OC-[my-app-name]' >> MODULES.yml
+    echo '- module: https://github.com/deephdc/UC-[my-account-name]-DEEP-OC-[my-app-name]' >> MODULES.yml
     git commit -a -m "adding new module to the catalogue"
     git push
 

source/user/howto/inference-locally.rst

@@ -3,6 +3,11 @@
 Try a service locally
 =====================
 
+.. admonition:: Useful video demos
+   :class: important
+
+    - `Running a module locally with docker <https://www.youtube.com/watch?v=3ORuymzO7V8&list=PLJ9x9Zk1O-J_UZfNO2uWp2pFMmbwLvzXa&index=13>`__
+
 .. admonition:: Requirements
 
     This section requires having `docker <https://docs.docker.com/install/#supported-platforms>`_ installed.

source/user/howto/inference-remotely.rst

@@ -3,6 +3,10 @@
 Try a service remotely
 ======================
 
+.. admonition:: Useful video demos
+   :class: important
+
+    - `Performing inference in the cloud with the DEEP training dashboard <https://www.youtube.com/watch?v=FyELMIr5Wbo&list=PLJ9x9Zk1O-J_UZfNO2uWp2pFMmbwLvzXa&index=4>`__
 
 The first step is to go to the `DEEP as a Service (or DEEPaaS) <https://deepaas.deep-hybrid-datacloud.eu/>`_.
 For educational purposes we are going to use a `general model to identify images <https://marketplace.deep-hybrid-datacloud.eu/modules/train-an-image-classifier.html>`_. This will allow us to see the general workflow.

source/user/howto/train-model-locally.rst

@@ -4,6 +4,11 @@
 Train a model locally
 =====================
 
+.. admonition:: Useful video demos
+   :class: important
+
+    - `Running a module locally with docker <https://www.youtube.com/watch?v=3ORuymzO7V8&list=PLJ9x9Zk1O-J_UZfNO2uWp2pFMmbwLvzXa&index=13>`__
+
 This is a step by step guide on how to train a module from the Marketplace with your own dataset.
 
 .. admonition:: Requirements

source/user/howto/train-model-remotely.rst

@@ -4,6 +4,11 @@
 Train a model remotely
 ======================
 
+.. admonition:: Useful video demos
+   :class: important
+
+    - `Training your model in the cloud with the DEEP training dashboard <https://www.youtube.com/watch?v=uqFXrEwtNNs&list=PLJ9x9Zk1O-J_UZfNO2uWp2pFMmbwLvzXa&index=6>`__
+
 This is a step by step guide on how to train a general model from the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`__ with your own dataset.
 
 .. admonition:: Requirements
@@ -122,7 +127,7 @@ and add your Docker repo name at the end of the ``MODULES.yml``.
 
     git clone https://github.com/[my-github-fork]
     cd [my-github-fork]
-    echo '- module: https://github.com/[my-account-name]/DEEP-OC-[my-app-name]' >> MODULES.yml
+    echo '- module: https://github.com/deephdc/UC-[my-account-name]-DEEP-OC-[my-app-name]' >> MODULES.yml
     git commit -a -m "adding new module to the catalogue"
     git push
 

source/user/index.rst

@@ -11,12 +11,14 @@ User documentation
     * `DockerHub <https://hub.docker.com/u/deephdc/>`__
     * `Docs <https://docs.deep-hybrid-datacloud.eu/en/latest/>`__
     * `NextCloud <https://nc.deep-hybrid-datacloud.eu/>`__
-    * `DEEP IAM <https://iam.deep-hybrid-datacloud.eu/>`_
+    * `DEEP IAM <https://iam.deep-hybrid-datacloud.eu/>`__
+    * `Status of services <https://status.deep-hybrid-datacloud.eu/>`__
 
 
     You can also check these `slides <https://cdn.jsdelivr.net/gh/deephdc/deep-docs@master/source/_static/overview.pdf>`_
     along with this `video <https://www.youtube.com/watch?v=cRMIviobF_c>`_  for a quick overview of the project
-    from the user's point of view.
+    from the user's point of view (`attached tutorial <https://www.youtube.com/watch?v=cRMIviobF_c>`_).
+
 
 
 New to the project? How about a quick dive?
@@ -53,8 +55,9 @@ Use a model (basic user)
    Perform inference locally <howto/inference-locally>
 
 ..
-  // commment because right now deepaas serverless is not working correctly,
-  // thus this can be potentially confusing
+  // comment because right now deepaas serverless is not working correctly,
+  // thus this guide can be potentially confusing.
+  // TODO: uncomment when ready
   Perform inference remotely <howto/inference-remotely>
 
 Train a model (intermediate user)

source/user/overview/cookiecutter-template.rst

@@ -4,6 +4,12 @@
 DEEP Modules Template
 =====================
 
+.. admonition:: Useful video demos
+   :class: important
+
+    - `Data science cookiecutter template <https://www.youtube.com/watch?v=mCxz8LQJWWA&list=PLJ9x9Zk1O-J_UZfNO2uWp2pFMmbwLvzXa&index=7>`__
+
+
 Overview
 --------
 
@@ -16,9 +22,7 @@ There are two versions of this template:
   looking for. Simple, minimal template, with the minimum requirements to integrate your code in DEEP.
 * `advanced <https://github.com/deephdc/cookiecutter-deep/tree/advanced>`_: this is a more advanced template.
   It makes more assumptions on how to structure projects and adds more files than those strictly needed for integration.
-  It also comes with additional files to implement tests (coverage, code quality, ...) on your module.
-  Unless you are looking for one of these features, you are probably safer using master.
-  This is the template that was used originally in the project.
+  Unless you are looking for some specific feature, you are probably safer using master.
 
 In order to create your project based on the template, one has to `install <https://cookiecutter.readthedocs.io/en/latest/installation.html>`_
 and then run the `cookicutter <https://cookiecutter.readthedocs.io>`_ tool as follows
@@ -78,7 +82,11 @@ The content of these files is populated based on your answer to the questions.
     │   │
     │   ├── __init__.py        <- Makes {{cookiecutter.repo_name}} a Python module
     │   │
-    │   └── api.py             <- Main script for the integration with DEEP API
+    │   ├── api.py             <- Main script for the integration with DEEP API
+    │   │
+    │   ├── misc.py            <- Misc functions that were helpful across projects
+    │   │
+    │   └── tests              <- Scripts to perform code testing
     │
     └── Jenkinsfile            <- Describes basic Jenkins CI/CD pipeline
 

source/user/overview/user-roles.rst

@@ -53,7 +53,6 @@ model and retrains it to perform `seed classification <https://marketplace.deep-
 
 * :doc:`How to train a model locally <../howto/train-model-locally>`
 * :doc:`How to train a model remotely <../howto/train-model-remotely>`
-* :doc:`How to add your module to the DEEP Marketplace <../howto/add-to-DEEP-marketplace>`
 
 
 The advanced user
@@ -82,4 +81,3 @@ models.
 * :doc:`How to use the DEEP Cookiecutter template for model development <cookiecutter-template>`
 * :doc:`How to develop your own machine learning model <../howto/develop-model>`
 * :ref:`How to integrate your model with the DEEPaaS API <user/overview/api:Integrate your model with the API>`
-* :doc:`How to add your model to the DEEP Marketplace <../howto/add-to-DEEP-marketplace>`