Skip to content
This repository has been archived by the owner on Jun 14, 2023. It is now read-only.

Commit

Permalink
try model locally
Browse files Browse the repository at this point in the history
  • Loading branch information
giangzuzana committed Jan 10, 2019
1 parent 9063bfd commit 38c1e4e
Show file tree
Hide file tree
Showing 2 changed files with 94 additions and 43 deletions.
72 changes: 40 additions & 32 deletions source/user/develop-model.rst
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
**************************************
Develop a model using DEEP UC template
======================================
**************************************

Prepare DEEP UC environment
---------------------------
1. Prepare DEEP UC environment
------------------------------

Install cookiecutter (if not yet done)
::
Expand All @@ -14,62 +15,69 @@ Run the DEEP UC cookiecutter template
Answer all questions from DEEP UC cookiecutter template with attentions to
``repo_name`` i.e. the name of your github repositories, etc.

This creates two project directories:
::
~/DEEP-OC-your_project
~/your_project
Go to ``github.com/your_account`` and
create corresponding repositories: ``DEEP-OC-your_project`` and ``your_project``

Do ``git push origin master`` in both created directories. This puts your initial code to ``github``.


Develop a model according to DEEP UC template
---------------------------------------------
2. Improve the initial code of the model
----------------------------------------

The structure of ``your_project`` created using
`DEEP UC template <https://github.com/indigo-dc/cookiecutter-data-science>`__ contains
the following core items needed to develop a model
the following core items needed to develop a DEEP UC model:
::
requirements.txt
data/
models/
{{cookiecutter.repo_name}}/dataset/make_dataset.py
{{cookiecutter.repo_name}}/features/build_features.py
{{cookiecutter.repo_name}}/models/model.py
{{repo_name}}/dataset/make_dataset.py
{{repo_name}}/features/build_features.py
{{repo_name}}/models/model.py
**Installing development requirements**
2.1 Installing development requirements
=======================================

Modify ``requirements.txt`` according to your needs (e.g. add more libraries) then run
::
$ pip install -r requirements.txt
You can modify and add more ``source files`` and put them
accordingly into the directory structure.

**Improve the initial code**
2.2 Make datasets
==================

You can modify as well as add more source files and put them accordingly into the directory structure.

**1. Make datasets:** source files in this directory aim to manipulate with raw dataset(s).
The output of this step is raw data, which can be cleaned and/or pre-formatted.
Source files in this directory aim to manipulate with raw dataset(s).
The output of this step is also raw data, but cleaned and/or pre-formatted.
::
{{cookiecutter.repo_name}}/dataset/make_dataset.py
{{cookiecutter.repo_name}}/dataset/
{{repo_name}}/dataset/
{{repo_name}}/dataset/make_dataset.py

2.3 Build features
===================

**2. Build features** takes the output from the previous step (Make datasets) and
creates ML train, test as well as validation data from raw data.
The concrete realisation is depend on concrete UC, the aim of the application as well as
technological background (e.g. high-performance supports).
This step takes the output from the previous step ``Make datasets`` and
creates train, test as well as validation ML data from raw but cleaned and pre-formatted data.
The realisation of this step depends on concrete UC, the aim of the application as well as
available technological backgrounds (e.g. high-performance supports for data processing).
::
{{cookiecutter.repo_name}}/features/build_features.py
{{cookiecutter.repo_name}}/features/
{{repo_name}}/features/
{{repo_name}}/features/build_features.py

2.4 Develop models
==================

**3. Develop models** dealing with the most interesting ML core i.e. modelling.
The most important thing in the ``model.py`` are implementations for DEEP entry points,
which are defined according to :ref:`API methods <user/overview/api:Methods>`.
You don't need to implement all the methods, just the ones you need.
This step deals with the most interesting phase in ML i.e. modelling.
The most important thing of DEEP UC models is located in ``model.py``
containing DEEP entry point implementations.
DEEP entry points are defined using :ref:`API methods <user/overview/api:Methods>`.
You don't need to implement all of them, just the ones you need.
::
{{cookiecutter.repo_name}}/models/model.py
{{cookiecutter.repo_name}}/models/
{{repo_name}}/models/
{{repo_name}}/models/model.py

65 changes: 54 additions & 11 deletions source/user/try-model-locally.rst
Original file line number Diff line number Diff line change
@@ -1,25 +1,68 @@
*******************
Try a model locally
===================
*******************

1. **Get Docker**
1. Get Docker
-------------

The first step is having `Docker <https://www.docker.com>`_ installed. To have an up-to-date installation please follow
the `official Docker installation guide <https://docs.docker.com/install>`_.

2. **Search for a model in the marketplace**


3. **Get the model**
2. Search for a model in the marketplace
----------------------------------------

clone the repo
The next step is to look for a model
in the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_
you want to try locally.
The marketplace contains an extensible list of existing models e.g.
`DEEP OC Image Classification <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-image-classification-tensorflow.html>`_,
`DEEP OC Retinopathy <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-retinopathy.html>`_,
`DEEP OC Massive Online Data Streams <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-massive-online-data-streams.html>`_,
`DEEP OC Seed Classification <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-seed-classification-theano.html>`_,
`DEEP OC Phytopankton (Theano) <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-phytopankton-theano.html>`_,
`DEEP OC Conus Classification <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-conus-classification-theano.html>`_,
`DEEP OC dogs breed determination <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-dogs-breed-determination.html>`_,
and many more.


build the container or get from DockerHub
3. Get the model
----------------

You will find that each model has an associate Docker container in DockerHub
`DEEP OC Image Classification <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-image-classification-tensorflow.html>`_
is associated with `deephdc/deep-oc-image-classification-tf <https://hub.docker.com/r/deephdc/deep-oc-image-classification-tf>`_,
`DEEP OC Massive Online Data Streams <https://marketplace.deep-hybrid-datacloud.eu/models/deep-oc-massive-online-data-streams.html>`_
is associated with `deephdc/deep-oc-mods <https://hub.docker.com/r/deephdc/deep-oc-mods>`_, etc.

4. **Run the model**
Let call the model you selected ``deep-oc-model_of_interest``.
Please, download the container with:
::
$ docker pull deephdc/deep-oc-model_of_interest

4. Run the model
----------------

5. **Go to the API, get the results**
Run the container with:
::
$ docker run -ti -p 5000:5000 deephdc/deep-oc-model_of_interest

MODS - probs as outpus
5. Go to the API, get the results
---------------------------------

Need reviewing/improvements for other UCs
Once running, point your browser to `http://127.0.0.1:5000/ <http://127.0.0.1:5000/>`_
and you will see the API documentation,
where you can test the model functionality, as well as perform other actions.

All models in the `DEEP Open Catalog marketplace <https://marketplace.deep-hybrid-datacloud.eu/>`_
utilize `DEEPaaS API <https://github.com/indigo-dc/DEEPaaS>`_.
The API enables a user friendly interaction with the underlying Deep Learning models and
can be used both for training and inference with the models.

.. image:: ../_static/deepaas.png

The concrete results can vary from model to model e.g.
the results of ``deephdc/deep-oc-image-classification-tf`` are image type(s) and picture(s),
the results of ``deephdc/deep-oc-mods`` are predicted values (float numbers).

0 comments on commit 38c1e4e

Please sign in to comment.