Skip to content
This repository has been archived by the owner on Jun 14, 2023. It is now read-only.

Commit

Permalink
fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
ignacio committed Jan 10, 2019
2 parents ed5be44 + 38c1e4e commit 24d5d33
Show file tree
Hide file tree
Showing 6 changed files with 105 additions and 38 deletions.
62 changes: 37 additions & 25 deletions source/user/develop-model.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
.. highlight:: console

**************************************
Develop a model using DEEP UC template
======================================
**************************************

.. highlight:: console

Prepare DEEP UC environment
---------------------------
Expand All @@ -18,7 +20,6 @@ Run the DEEP UC cookiecutter template
Answer all questions from DEEP UC cookiecutter template with attentions to
``repo_name`` i.e. the name of your github repositories, etc.

This creates two project directories:
::

Expand All @@ -27,59 +28,70 @@ This creates two project directories:
Go to ``github.com/your_account`` and
create corresponding repositories: ``DEEP-OC-your_project`` and ``your_project``

Do ``git push origin master`` in both created directories. This puts your initial code to ``github``.


Develop a model according to DEEP UC template
---------------------------------------------
2. Improve the initial code of the model
----------------------------------------

The structure of ``your_project`` created using
`DEEP UC template <https://github.com/indigo-dc/cookiecutter-data-science>`__ contains
the following core items needed to develop a model
the following core items needed to develop a DEEP UC model:
::

requirements.txt
data/
models/
{{cookiecutter.repo_name}}/dataset/make_dataset.py
{{cookiecutter.repo_name}}/features/build_features.py
{{cookiecutter.repo_name}}/models/model.py
**Installing development requirements**
{{repo_name}}/dataset/make_dataset.py
{{repo_name}}/features/build_features.py
{{repo_name}}/models/model.py


2.1 Installing development requirements
=======================================

Modify ``requirements.txt`` according to your needs (e.g. add more libraries) then run
::

$ pip install -r requirements.txt
You can modify and add more ``source files`` and put them
accordingly into the directory structure.

**Improve the initial code**

You can modify as well as add more source files and put them accordingly into the directory structure.
2.2 Make datasets
==================

**1. Make datasets:** source files in this directory aim to manipulate with raw dataset(s).
The output of this step is raw data, which can be cleaned and/or pre-formatted.
Source files in this directory aim to manipulate with raw dataset(s).
The output of this step is also raw data, but cleaned and/or pre-formatted.
::

{{cookiecutter.repo_name}}/dataset/make_dataset.py
{{cookiecutter.repo_name}}/dataset/

**2. Build features** takes the output from the previous step (Make datasets) and
creates ML train, test as well as validation data from raw data.
The concrete realisation is depend on concrete UC, the aim of the application as well as
technological background (e.g. high-performance supports).

2.3 Build features
===================

This step takes the output from the previous step ``Make datasets`` and
creates train, test as well as validation ML data from raw but cleaned and pre-formatted data.
The realisation of this step depends on concrete UC, the aim of the application as well as
available technological backgrounds (e.g. high-performance supports for data processing).
::

{{cookiecutter.repo_name}}/features/build_features.py
{{cookiecutter.repo_name}}/features/

**3. Develop models** dealing with the most interesting ML core i.e. modelling.
The most important thing in the ``model.py`` are implementations for DEEP entry points,
which are defined according to :ref:`API methods <user/overview/api:Methods>`.
You don't need to implement all the methods, just the ones you need.

2.4 Develop models
==================

This step deals with the most interesting phase in ML i.e. modelling.
The most important thing of DEEP UC models is located in ``model.py``
containing DEEP entry point implementations.
DEEP entry points are defined using :ref:`API methods <user/overview/api:Methods>`.
You don't need to implement all of them, just the ones you need.
::

{{cookiecutter.repo_name}}/models/model.py
{{cookiecutter.repo_name}}/models/

2 changes: 1 addition & 1 deletion source/user/overview/api.rst
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
DEEPaaS API
===========


The `DEEPaaS API <https://github.com/indigo-dc/DEEPaaS>`_ enables a user friendly interaction with the underlying Deep
Learning models and can be used both for training and inference with the models.


Integrate your model with the API
---------------------------------

Expand Down
2 changes: 1 addition & 1 deletion source/user/overview/architecture.rst
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
Architecture overview
---------------------


There are several different components in the DEEP-HDC project that are relevant for the users. Later on you will see
how each :doc:`different type of user <user-roles>` can take advantage of the different components.


The marketplace
===============

Expand Down
5 changes: 5 additions & 0 deletions source/user/overview/train-model-locally.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,14 @@ Train a model locally
This is a step by step guide on how to train a model from the Marketplace with your own dataset.
You can also look at `video demo <https://www.youtube.com/watch?v=Mh6rdlqX-7I&feature=youtu.be>`_ on how to do this.


1. Get Docker
=============

The first step is having `Docker <https://www.docker.com>`_ installed. To have an up-to-date installation please follow
the `official Docker installation guide <https://docs.docker.com/install>`_.


2. Search for a model in the marketplace
========================================

Expand All @@ -18,6 +20,7 @@ the `official Docker installation guide <https://docs.docker.com/install>`_.
The next step is to look for a model in the marketplace you want to retrain on your own data.
Possible option include image classifiers, etc.


3. Get the model
================

Expand All @@ -40,11 +43,13 @@ For example if you wanted to download the image classifier model you would have
We are using the port ``5000`` to deploy the API and the port ``6006`` to monitor the training (for example using
`Tensorboard <https://www.tensorflow.org/guide/summaries_and_tensorboard>`_).


4. Upload your data to storage resources
========================================

.. todo:: Lara and Valentin: Fill this section. Look at the possibility of using rclone with Google Drive, Dropbox, etc.


5. Train the model
==================

Expand Down
1 change: 1 addition & 0 deletions source/user/overview/user-roles.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Our different user roles
DEEPaaS is focused with three different types of users in mind, whcih vary in their machine learning knowledge and their
usage of DEEPaaS.


The basic user
--------------

Expand Down
71 changes: 60 additions & 11 deletions source/user/try-model-locally.rst
Original file line number Diff line number Diff line change
@@ -1,25 +1,74 @@
.. highlight:: console

*******************
Try a model locally
===================
*******************


1. Get Docker
-------------

The first step is having `Docker <https://www.docker.com>`_ installed. To have an up-to-date installation please follow
the `official Docker installation guide <https://docs.docker.com/install>`_.


2. Search for a model in the marketplace
----------------------------------------

The next step is to look for a model
in the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_
you want to try locally.
The marketplace contains an extensible list of existing models e.g.
`DEEP OC Image Classification <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-image-classification-tensorflow.html>`_,
`DEEP OC Retinopathy <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-retinopathy.html>`_,
`DEEP OC Massive Online Data Streams <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-massive-online-data-streams.html>`_,
`DEEP OC Seed Classification <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-seed-classification-theano.html>`_,
`DEEP OC Phytopankton (Theano) <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-phytopankton-theano.html>`_,
`DEEP OC Conus Classification <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-conus-classification-theano.html>`_,
`DEEP OC dogs breed determination <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-dogs-breed-determination.html>`_,
and many more.


1. **Get Docker**
3. Get the model
----------------

You will find that each model has an associate Docker container in DockerHub
`DEEP OC Image Classification <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-image-classification-tensorflow.html>`_
is associated with `deephdc/deep-oc-image-classification-tf <https://hub.docker.com/r/deephdc/deep-oc-image-classification-tf>`_,
`DEEP OC Massive Online Data Streams <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-massive-online-data-streams.html>`_
is associated with `deephdc/deep-oc-mods <https://hub.docker.com/r/deephdc/deep-oc-mods>`_, etc.

2. **Search for a model in the marketplace**
Let call the model you selected ``deep-oc-model_of_interest``.
Please, download the container with:

3. **Get the model**
::

clone the repo
$ docker pull deephdc/deep-oc-model_of_interest

build the container or get from DockerHub
4. Run the model
----------------

Run the container with:
::

4. **Run the model**
$ docker run -ti -p 5000:5000 deephdc/deep-oc-model_of_interest

5. Go to the API, get the results
---------------------------------

5. **Go to the API, get the results**
Once running, point your browser to `http://127.0.0.1:5000/ <http://127.0.0.1:5000/>`_
and you will see the API documentation,
where you can test the model functionality, as well as perform other actions.

MODS - probs as outpus
All models in the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_
utilize `DEEPaaS API <https://github.com/indigo-dc/DEEPaaS>`_.
The API enables a user friendly interaction with the underlying Deep Learning models and
can be used both for training and inference with the models.

Need reviewing/improvements for other UCs
.. image:: ../_static/deepaas.png

The concrete results can vary from model to model e.g.
the results of ``deephdc/deep-oc-image-classification-tf`` are image type(s) and picture(s),
the results of ``deephdc/deep-oc-mods`` are predicted values (float numbers).

0 comments on commit 24d5d33

Please sign in to comment.