-
Notifications
You must be signed in to change notification settings - Fork 78
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* WIP #988 Added "Getting Started" section * WIP #988 Added some reference docs for cli and config * WIP #988 Added quick start * WIP #988 moved installation to deployment section * WIP #988 started adding initial tutorial * WIP #988 Finished hello, cifar10 * WIP #988 Added custom operation info * WIP #988 Added operation feedback reference * WIP #988 added custom layer docs * WIP #988 bolded the setter fn best practice * WIP #988 'DeepForge Fundamentals' -> 'Fundamentals' * WIP #988 fixed the inline code snippets * WIP #988 added custom data types * WIP #988 added images to custom operation docs * WIP #988 updated operation feedback section * WIP #988 Added example of finished train operation * WIP #988 Added custom layer images * WIP #988 Added image for editing prim types * WIP #988 deepforge->DeepForge and updated styling * WIP #988 Added images for getting started * WIP #988 Added more images to hello world example * WIP #988 added test results
- Loading branch information
Showing
29 changed files
with
1,002 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
_build |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# Minimal makefile for Sphinx documentation | ||
# | ||
|
||
# You can set these variables from the command line. | ||
SPHINXOPTS = | ||
SPHINXBUILD = sphinx-build | ||
SPHINXPROJ = deepforge | ||
SOURCEDIR = . | ||
BUILDDIR = _build | ||
|
||
# Put it first so that "make" without argument is like "make help". | ||
help: | ||
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) | ||
|
||
.PHONY: help Makefile | ||
|
||
# Catch-all target: route all unknown targets to Sphinx using the new | ||
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). | ||
%: Makefile | ||
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,159 @@ | ||
import sphinx_rtd_theme | ||
#!/usr/bin/env python3 | ||
# -*- coding: utf-8 -*- | ||
# | ||
# deepforge documentation build configuration file, created by | ||
# sphinx-quickstart on Mon Mar 13 18:56:27 2017. | ||
# | ||
# This file is execfile()d with the current directory set to its | ||
# containing dir. | ||
# | ||
# Note that not all possible configuration values are present in this | ||
# autogenerated file. | ||
# | ||
# All configuration values have a default; values that are commented out | ||
# serve to show the default. | ||
|
||
# If extensions (or modules to document with autodoc) are in another directory, | ||
# add these directories to sys.path here. If the directory is relative to the | ||
# documentation root, use os.path.abspath to make it absolute, like shown here. | ||
# | ||
# import os | ||
# import sys | ||
# sys.path.insert(0, os.path.abspath('.')) | ||
|
||
|
||
# -- General configuration ------------------------------------------------ | ||
|
||
# If your documentation needs a minimal Sphinx version, state it here. | ||
# | ||
# needs_sphinx = '1.0' | ||
|
||
# Add any Sphinx extension module names here, as strings. They can be | ||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom | ||
# ones. | ||
extensions = [] | ||
|
||
# Add any paths that contain templates here, relative to this directory. | ||
templates_path = ['_templates'] | ||
|
||
# The suffix(es) of source filenames. | ||
# You can specify multiple suffix as a list of string: | ||
# | ||
# source_suffix = ['.rst', '.md'] | ||
source_suffix = '.rst' | ||
|
||
# The master toctree document. | ||
master_doc = 'index' | ||
|
||
# General information about the project. | ||
project = 'DeepForge' | ||
copyright = '2017, Brian Broll' | ||
author = 'Brian Broll' | ||
|
||
# The version info for the project you're documenting, acts as replacement for | ||
# |version| and |release|, also used in various other places throughout the | ||
# built documents. | ||
# | ||
# The short X.Y version. | ||
version = '' | ||
# The full version, including alpha/beta/rc tags. | ||
release = '' | ||
|
||
# The language for content autogenerated by Sphinx. Refer to documentation | ||
# for a list of supported languages. | ||
# | ||
# This is also used if you do content translation via gettext catalogs. | ||
# Usually you set "language" from the command line for these cases. | ||
language = None | ||
|
||
# List of patterns, relative to source directory, that match files and | ||
# directories to ignore when looking for source files. | ||
# This patterns also effect to html_static_path and html_extra_path | ||
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] | ||
|
||
# The name of the Pygments (syntax highlighting) style to use. | ||
pygments_style = 'sphinx' | ||
|
||
# If true, `todo` and `todoList` produce output, else they produce nothing. | ||
todo_include_todos = False | ||
|
||
|
||
# -- Options for HTML output ---------------------------------------------- | ||
|
||
# The theme to use for HTML and HTML Help pages. See the documentation for | ||
# a list of builtin themes. | ||
# | ||
html_theme = 'sphinx_rtd_theme' | ||
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] | ||
|
||
# Theme options are theme-specific and customize the look and feel of a theme | ||
# further. For a list of options available for each theme, see the | ||
# documentation. | ||
# | ||
# html_theme_options = {} | ||
|
||
# Add any paths that contain custom static files (such as style sheets) here, | ||
# relative to this directory. They are copied after the builtin static files, | ||
# so a file named "default.css" will overwrite the builtin "default.css". | ||
html_static_path = ['_static'] | ||
|
||
|
||
# -- Options for HTMLHelp output ------------------------------------------ | ||
|
||
# Output file base name for HTML help builder. | ||
htmlhelp_basename = 'deepforgedoc' | ||
|
||
|
||
# -- Options for LaTeX output --------------------------------------------- | ||
|
||
latex_elements = { | ||
# The paper size ('letterpaper' or 'a4paper'). | ||
# | ||
# 'papersize': 'letterpaper', | ||
|
||
# The font size ('10pt', '11pt' or '12pt'). | ||
# | ||
# 'pointsize': '10pt', | ||
|
||
# Additional stuff for the LaTeX preamble. | ||
# | ||
# 'preamble': '', | ||
|
||
# Latex figure (float) alignment | ||
# | ||
# 'figure_align': 'htbp', | ||
} | ||
|
||
# Grouping the document tree into LaTeX files. List of tuples | ||
# (source start file, target name, title, | ||
# author, documentclass [howto, manual, or own class]). | ||
latex_documents = [ | ||
(master_doc, 'deepforge.tex', 'deepforge Documentation', | ||
'Brian Broll', 'manual'), | ||
] | ||
|
||
|
||
# -- Options for manual page output --------------------------------------- | ||
|
||
# One entry per manual page. List of tuples | ||
# (source start file, name, description, authors, manual section). | ||
man_pages = [ | ||
(master_doc, 'deepforge', 'deepforge Documentation', | ||
[author], 1) | ||
] | ||
|
||
|
||
# -- Options for Texinfo output ------------------------------------------- | ||
|
||
# Grouping the document tree into Texinfo files. List of tuples | ||
# (source start file, target name, title, author, | ||
# dir menu entry, description, category) | ||
texinfo_documents = [ | ||
(master_doc, 'deepforge', 'deepforge Documentation', | ||
author, 'deepforge', 'One line description of project.', | ||
'Miscellaneous'), | ||
] | ||
|
||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,134 @@ | ||
Installation | ||
============ | ||
|
||
DeepForge Component Overview | ||
---------------------------- | ||
DeepForge is composed of four main elements: | ||
|
||
- *Server*: Main component hosting all the project information and is connected to by the clients | ||
- *Database*: MongoDB database containing DeepForge, job queue for the workers, etc | ||
- *Worker*: Slave machine performing the actual machine learning computation | ||
- *Client*: The connected browsers working on DeepForge projects. | ||
|
||
Of course, only the *Server*, *Database* (MongoDB) and *Worker* need to be installed. If you are not going to execute any machine learning pipelines, installing the *Worker* can be skipped. | ||
|
||
Preinstallation | ||
--------------- | ||
Installing dependencies | ||
~~~~~~~~~~~~~~~~~~~~~~~ | ||
The following dependencies are required for each component: | ||
|
||
- *Server* (NodeJS v6.2.1) | ||
- *Database* (MongoDB v3.0.7) | ||
- *Worker*: NodeJS v6.2.1 (used for job management logic) and `Torch <http://torch.ch/docs/getting-started.html#>`_ (this will be installed automatically by the cli when needed) | ||
- *Client*: We recommend using Google Chrome and are not supporting other browsers (for now). In other words, other browsers can be used at your own risk. | ||
|
||
Installation | ||
------------ | ||
Database | ||
~~~~~~~~ | ||
Download and install MongoDB from the `website <https://www.mongodb.org/>`_. If you are planning on running MongoDB locally on the same machine as DeepForge, simply start `mongod` and continue to setting up DeepForge. | ||
|
||
If you are planning on running MongoDB remotely, set the environment variable "MONGO_URI" to the URI of the Mongo instance that DeepForge will be using: | ||
|
||
.. code-block:: bash | ||
MONGO_URI="mongodb://pathToMyMongo.com:27017/myCollection" deepforge start | ||
Server | ||
~~~~~~ | ||
The DeepForge server is included with the deepforge cli and can be started simply with | ||
|
||
.. code-block:: bash | ||
deepforge start --server | ||
By default, DeepForge will start on `http://localhost:8888`. However, the port can be specified with the `--port` option. For example: | ||
|
||
.. code-block:: bash | ||
deepforge start --server --port 3000 | ||
Worker | ||
~~~~~~ | ||
The DeepForge worker can be started with | ||
|
||
.. code-block:: bash | ||
deepforge start --worker | ||
The worker will install dependencies the first time it is run (including torch, if it is not already installed). | ||
|
||
To connect to a remote deepforge instance, add the url of the DeepForge server: | ||
|
||
.. code-block:: bash | ||
deepforge start --worker http://myaddress.com:1234 | ||
Updating | ||
~~~~~~~~ | ||
DeepForge can be updated with the command line interface rather simply: | ||
|
||
.. code-block:: bash | ||
deepforge update | ||
By default, this will update both DeepForge and the local torch installation. To only update DeepForge, add the `--server` flag: | ||
|
||
.. code-block:: bash | ||
deepforge update --server | ||
For more update options, check out `deepforge update --help`! | ||
|
||
Manual Installation (Development) | ||
--------------------------------- | ||
Installing DeepForge for development is essentially cloning the repository and then using `npm` (node package manager) to run the various start, test, etc, commands (including starting the individual components). The deepforge cli can still be used but must be referenced from `./bin/deepforge`. That is, `deepforge start` becomes `./bin/deepforge start` (from the project root). | ||
|
||
DeepForge Server | ||
~~~~~~~~~~~~~~~~ | ||
First, clone the repository: | ||
|
||
.. code-block:: bash | ||
git clone https://github.com/dfst/deepforge.git | ||
Then install the project dependencies: | ||
|
||
.. code-block:: bash | ||
npm install | ||
To run all components locally start with | ||
|
||
.. code-block:: bash | ||
./bin/deepforge start | ||
and navigate to `http://localhost:8888` to start using DeepForge! | ||
|
||
Alternatively, if jobs are going to be executed on an external worker, run `./bin/deepforge start -s` locally and navigate to `http://localhost:8888`. | ||
|
||
DeepForge Worker | ||
~~~~~~~~~~~~~~~~ | ||
If you are using `./bin/deepforge start -s` you will need to set up a DeepForge worker (`./bin/deepforge start` starts a local worker for you!). DeepForge workers are slave machines connected to DeepForge which execute the provided jobs. This allows the jobs to access the GPU, etc, and provides a number of benefits over trying to perform deep learning tasks in the browser. | ||
|
||
Once DeepForge is installed on the worker, start it with | ||
|
||
.. code-block:: bash | ||
./bin/deepforge start -w | ||
Note: If you are running the worker on a different machine, put the address of the DeepForge server as an argument to the command. For example: | ||
|
||
.. code-block:: bash | ||
./bin/deepforge start -w http://myaddress.com:1234 | ||
Updating | ||
~~~~~~~~ | ||
Updating can be done the same as any other git project; that is, by running `git pull` from the project root. Sometimes, the dependencies need to be updated so it is recommended to run `npm install` following `git pull`. | ||
|
||
Configuration | ||
------------- | ||
After installing DeepForge, it can be helpful to check out `configuring DeepForge <getting_started/configuration.rst>`_ |
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
Custom Data Types | ||
================= | ||
|
||
As operation inputs and outputs are strongly typed, DeepForge supports the creation of custom data types to promote flexibility when designing complex pipelines and operations. DeepForge data types can be either primitive types or custom classes. Custom DeepForge primitive types are relatively straight-forward; they can inherit from other types and must implement a serialization and deserialization methods (which may be as simple as :code:`torch.save` and :code:`torch.load`). Custom classes are also relatively simple to define but actually contain their own methods along with serialization and deserialization functions. | ||
|
||
New data types can be defined from the operation editor from the dialog for selecting input or output data for the operation. After defining a new class, this class is available from within any of the operations in the DeepForge project. | ||
|
||
.. figure:: model_data_editor.png | ||
:align: center | ||
:scale: 55 % | ||
|
||
Editing the serialization and deserialization for the "model" type | ||
|
Oops, something went wrong.