Skip to content

Commit

Permalink
Added sphinx docs to /docs Fixes #988 (#1005)
Browse files Browse the repository at this point in the history
* WIP #988 Added "Getting Started" section

* WIP #988 Added some reference docs for cli and config

* WIP #988 Added quick start

* WIP #988 moved installation to deployment section

* WIP #988 started adding initial tutorial

* WIP #988 Finished hello, cifar10

* WIP #988 Added custom operation info

* WIP #988 Added operation feedback reference

* WIP #988 added custom layer docs

* WIP #988 bolded the setter fn best practice

* WIP #988 'DeepForge Fundamentals' -> 'Fundamentals'

* WIP #988 fixed the inline code snippets

* WIP #988 added custom data types

* WIP #988 added images to custom operation docs

* WIP #988 updated operation feedback section

* WIP #988 Added example of finished train operation

* WIP #988 Added custom layer images

* WIP #988 Added image for editing prim types

* WIP #988 deepforge->DeepForge and updated styling

* WIP #988 Added images for getting started

* WIP #988 Added more images to hello world example

* WIP #988 added test results
  • Loading branch information
brollb committed Apr 10, 2017
1 parent 012c9fe commit 4d7a158
Show file tree
Hide file tree
Showing 29 changed files with 1,002 additions and 0 deletions.
1 change: 1 addition & 0 deletions docs/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
_build
20 changes: 20 additions & 0 deletions docs/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = deepforge
SOURCEDIR = .
BUILDDIR = _build

# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
159 changes: 159 additions & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
import sphinx_rtd_theme
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# deepforge documentation build configuration file, created by
# sphinx-quickstart on Mon Mar 13 18:56:27 2017.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))


# -- General configuration ------------------------------------------------

# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = []

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'

# The master toctree document.
master_doc = 'index'

# General information about the project.
project = 'DeepForge'
copyright = '2017, Brian Broll'
author = 'Brian Broll'

# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = ''
# The full version, including alpha/beta/rc tags.
release = ''

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']

# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'

# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False


# -- Options for HTML output ----------------------------------------------

# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]

# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']


# -- Options for HTMLHelp output ------------------------------------------

# Output file base name for HTML help builder.
htmlhelp_basename = 'deepforgedoc'


# -- Options for LaTeX output ---------------------------------------------

latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',

# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',

# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',

# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}

# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'deepforge.tex', 'deepforge Documentation',
'Brian Broll', 'manual'),
]


# -- Options for manual page output ---------------------------------------

# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'deepforge', 'deepforge Documentation',
[author], 1)
]


# -- Options for Texinfo output -------------------------------------------

# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'deepforge', 'deepforge Documentation',
author, 'deepforge', 'One line description of project.',
'Miscellaneous'),
]



134 changes: 134 additions & 0 deletions docs/deployment/installation.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
Installation
============

DeepForge Component Overview
----------------------------
DeepForge is composed of four main elements:

- *Server*: Main component hosting all the project information and is connected to by the clients
- *Database*: MongoDB database containing DeepForge, job queue for the workers, etc
- *Worker*: Slave machine performing the actual machine learning computation
- *Client*: The connected browsers working on DeepForge projects.

Of course, only the *Server*, *Database* (MongoDB) and *Worker* need to be installed. If you are not going to execute any machine learning pipelines, installing the *Worker* can be skipped.

Preinstallation
---------------
Installing dependencies
~~~~~~~~~~~~~~~~~~~~~~~
The following dependencies are required for each component:

- *Server* (NodeJS v6.2.1)
- *Database* (MongoDB v3.0.7)
- *Worker*: NodeJS v6.2.1 (used for job management logic) and `Torch <http://torch.ch/docs/getting-started.html#>`_ (this will be installed automatically by the cli when needed)
- *Client*: We recommend using Google Chrome and are not supporting other browsers (for now). In other words, other browsers can be used at your own risk.

Installation
------------
Database
~~~~~~~~
Download and install MongoDB from the `website <https://www.mongodb.org/>`_. If you are planning on running MongoDB locally on the same machine as DeepForge, simply start `mongod` and continue to setting up DeepForge.

If you are planning on running MongoDB remotely, set the environment variable "MONGO_URI" to the URI of the Mongo instance that DeepForge will be using:

.. code-block:: bash
MONGO_URI="mongodb://pathToMyMongo.com:27017/myCollection" deepforge start
Server
~~~~~~
The DeepForge server is included with the deepforge cli and can be started simply with

.. code-block:: bash
deepforge start --server
By default, DeepForge will start on `http://localhost:8888`. However, the port can be specified with the `--port` option. For example:

.. code-block:: bash
deepforge start --server --port 3000
Worker
~~~~~~
The DeepForge worker can be started with

.. code-block:: bash
deepforge start --worker
The worker will install dependencies the first time it is run (including torch, if it is not already installed).

To connect to a remote deepforge instance, add the url of the DeepForge server:

.. code-block:: bash
deepforge start --worker http://myaddress.com:1234
Updating
~~~~~~~~
DeepForge can be updated with the command line interface rather simply:

.. code-block:: bash
deepforge update
By default, this will update both DeepForge and the local torch installation. To only update DeepForge, add the `--server` flag:

.. code-block:: bash
deepforge update --server
For more update options, check out `deepforge update --help`!

Manual Installation (Development)
---------------------------------
Installing DeepForge for development is essentially cloning the repository and then using `npm` (node package manager) to run the various start, test, etc, commands (including starting the individual components). The deepforge cli can still be used but must be referenced from `./bin/deepforge`. That is, `deepforge start` becomes `./bin/deepforge start` (from the project root).

DeepForge Server
~~~~~~~~~~~~~~~~
First, clone the repository:

.. code-block:: bash
git clone https://github.com/dfst/deepforge.git
Then install the project dependencies:

.. code-block:: bash
npm install
To run all components locally start with

.. code-block:: bash
./bin/deepforge start
and navigate to `http://localhost:8888` to start using DeepForge!

Alternatively, if jobs are going to be executed on an external worker, run `./bin/deepforge start -s` locally and navigate to `http://localhost:8888`.

DeepForge Worker
~~~~~~~~~~~~~~~~
If you are using `./bin/deepforge start -s` you will need to set up a DeepForge worker (`./bin/deepforge start` starts a local worker for you!). DeepForge workers are slave machines connected to DeepForge which execute the provided jobs. This allows the jobs to access the GPU, etc, and provides a number of benefits over trying to perform deep learning tasks in the browser.

Once DeepForge is installed on the worker, start it with

.. code-block:: bash
./bin/deepforge start -w
Note: If you are running the worker on a different machine, put the address of the DeepForge server as an argument to the command. For example:

.. code-block:: bash
./bin/deepforge start -w http://myaddress.com:1234
Updating
~~~~~~~~
Updating can be done the same as any other git project; that is, by running `git pull` from the project root. Sometimes, the dependencies need to be updated so it is recommended to run `npm install` following `git pull`.

Configuration
-------------
After installing DeepForge, it can be helpful to check out `configuring DeepForge <getting_started/configuration.rst>`_
Binary file added docs/fundamentals/action_layer.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
13 changes: 13 additions & 0 deletions docs/fundamentals/custom_data_types.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
Custom Data Types
=================

As operation inputs and outputs are strongly typed, DeepForge supports the creation of custom data types to promote flexibility when designing complex pipelines and operations. DeepForge data types can be either primitive types or custom classes. Custom DeepForge primitive types are relatively straight-forward; they can inherit from other types and must implement a serialization and deserialization methods (which may be as simple as :code:`torch.save` and :code:`torch.load`). Custom classes are also relatively simple to define but actually contain their own methods along with serialization and deserialization functions.

New data types can be defined from the operation editor from the dialog for selecting input or output data for the operation. After defining a new class, this class is available from within any of the operations in the DeepForge project.

.. figure:: model_data_editor.png
:align: center
:scale: 55 %

Editing the serialization and deserialization for the "model" type

0 comments on commit 4d7a158

Please sign in to comment.