Add step-by-step guides to docs. Closes #1895 (#1896)

* Full walkthrough added

* Fix viewing-executions link

* Fix multiple hyperlinks

* Add link to walkthrough main page

* Added nav links to all pages

* Fixed centering on nav links

* Fixed previous tutorial link

* resolving requested changes (minus style)

* minor fixes and restructured toctree

* Remove section links from cifar/redshift guides

* removed some icon color references

docs/index.rst

@@ -30,6 +30,19 @@ Welcome to DeepForge's documentation!
    deployment/overview.rst
    deployment/native.rst
 
+.. toctree::
+   :maxdepth: 1
+   :caption: Step-by-Step Guides
+
+   walkthrough/introduction.rst
+   walkthrough/creating-pipelines.rst
+   walkthrough/creating-operations.rst
+   walkthrough/creating-neural-networks.rst
+   walkthrough/executing-pipelines.rst
+   walkthrough/viewing-executions.rst
+   walkthrough/CIFAR-10-classifier.rst
+   walkthrough/redshift-estimator.rst
+
 .. toctree::
    :maxdepth: 1
    :caption: Tutorials and Examples

docs/walkthrough/creating-neural-networks.rst

@@ -0,0 +1,116 @@
+Creating Neural Networks
+------------------------
+
+This page will guide you through the steps needed to create a neural network architecture for use in pipelines.
+
+Importing Resource Libraries
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Neural networks and other models can be created from the *Resources* tab on the sidebar. Before any models can be created in this tab, you must import the associated library into the project using the red floating button in the bottom right of the workspace.
+
+.. figure:: images/resources-blank.png
+    :align: center
+    :scale: 50%
+
+In the box that appears, you will see a list of libraries that are available for import.
+
+.. figure:: images/resources-import-keras.png
+    :align: center
+    :scale: 50%
+
+Clicking the download icon will install that library and allow creation of associated models. The keras library, for instance, allows for the creation of neural network models.
+
+.. figure:: images/resources-import-keras-after.png
+    :align: center
+    :scale: 50%
+
+Creating a New Architecture
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+After any library has been imported, new models can be created by hovering over the import library button and clicking the floating blue button that appears. This will generate a blank model and automatically open up that model's workspace.
+
+.. figure:: images/resources-new.png
+    :align: center
+    :scale: 50%
+
+Clicking anywhere in the workspace will add the first layer of the architecture, which will always be an input layer. Just as with pipelines, these architectures are represented by a flowchart, with each node representing a single layer in the neural network.
+
+.. figure:: images/vgg-blank.png
+    :align: center
+    :scale: 50%
+
+Editing Network Layers
+~~~~~~~~~~~~~~~~~~~~~~
+Clicking on a layer allows for changing the parameters of that layer. Many of these parameters can be left undefined, but many layers require that some specific parameters be given. If a layer has not been supplied with the necessary parameters, or if there is some other error encountered at that layer when building the network, the layer will be highlighted with a red border. Hovering the mouse over the layer will reveal the error. Hovering over a newly created **Input** layer, for example, shows us that the layer requires that either the shape or batch_shape parameter must be defined.
+
+.. figure:: images/vgg-input-error.png
+    :align: center
+    :scale: 50%
+
+Some parameters may not be immediately clear on their effects from the name alone. For unfamiliar parameters, hovering over the name of the parameter will reveal a short description.
+
+.. figure:: images/vgg-input-hover.png
+    :align: center
+    :scale: 50%
+
+In addition, clicking on the **?** icon in the top right of the expanded layer will display documentation on the layer as a whole, including descriptions of all available parameters.
+
+.. figure:: images/vgg-input-doc.png
+    :align: center
+    :scale: 50%
+
+Adding Additional Layers
+~~~~~~~~~~~~~~~~~~~~~~~~
+To add additional layers, you can click on the arrow icons on the top or bottom of any layer. The icon should become a + icon and clicking again will open a menu from which the desired layer type can be chosen.
+
+.. figure:: images/vgg-add-layer.png
+    :align: center
+    :scale: 50%
+
+.. figure:: images/network-new-layer.png
+    :align: center
+    :scale: 50%
+
+Layers can also be removed from the network by expanding the layer and clicking the red X icon in the top left. Two layers that already exist in the network can also be linked by clicking on the output icon on one layer and the input icon on another. A given layer can have any number of other layers as inputs or outputs. Some layers, such as the **Dense** layer, however, only expect one input and will give an error when multiple inputs are detected.
+
+.. figure:: images/network-multi-io.png
+    :align: center
+    :scale: 50%
+
+It is optional, though recommended, that the network be concluded with an **Output** layer. A network may include multiple outputs, in which case all outputs must be given an **Output** layer. If no **Output** layer is included, the last layer in the network will be treated as the sole output.
+
+.. figure:: images/network-multi-out.png
+    :align: center
+    :scale: 50%
+
+Connections Between Layers
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+When two layers are connected, they will be joined by a black arrow that indicates the flow of data through the network. Hovering over these arrows will reveal the shape of the data, which can help with analyzing the network to ensure that the data is being transformed as desired.
+
+.. figure:: images/network-connect-hover.png
+    :align: center
+    :scale: 50%
+
+Connections can also be removed and layers separated by clicking on the unwanted arrow and then clicking on the red X icon that appears.
+
+.. figure:: images/network-connect-delete.png
+    :align: center
+    :scale: 50%
+
+Exporting Architectures
+~~~~~~~~~~~~~~~~~~~~~~~
+With keras models, another feature exists to export the model as python code. Clicking the red arrow button in the bottom right of the workspace will display a window generating the code. After making any optional changes to the configuration, clicking run will generate the code.
+
+.. figure:: images/vgg-generate-keras.png
+    :align: center
+    :scale: 50%
+
+After successful generation, hovering over the red arrow button and clicking on the floating gray list button will provide a list of all exported architectures.
+
+.. figure:: images/vgg-gen-keras-view-res.png
+    :align: center
+    :scale: 50%
+
+Clicking on *Details* will provide some metadata about the export, as well as a link to download the generated file. This file can then be incorporated into a python project.
+
+.. figure:: images/vgg-gen-keras-view-res-details.png
+    :align: center
+    :scale: 50%

docs/walkthrough/creating-operations.rst

@@ -0,0 +1,65 @@
+Creating Operations
+-------------------
+
+When adding an operation to a pipeline, new operations can be created by clicking the *New Operation* option. This will open the operation editor for the new operation.
+
+.. figure:: images/cifar-pipeline-blank.png
+    :align: center
+    :scale: 50%
+
+This editor can also be reached for existing operations by clicking the **</>** icon when editing an operation's attributes.
+
+.. figure:: images/cifar-operation-io.png
+    :align: center
+    :scale: 50%
+
+This editor has two primary views for editing the operation. The left view allows editing the underlying code of the operation directly. The right view provides a graphical means of adding inputs, outputs, and attributes.
+
+.. figure:: images/new-operation.png
+    :align: center
+    :scale: 50%
+
+Editing the Operation Interface
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. figure:: images/new-operation-orig.png
+    :align: center
+    :scale: 50%
+
+Clicking on the operation in the right view will allow editing the operation interface. The operation interface consists of design time parameters (attributes and references) as well as inputs and outputs generated at runtime.
+
+Attributes can be added by clicking the *New Attribute* label, which will open a dialog box where you can define the name, type, default value, and other metadata about that attribute. This dialog box can be viewed again to edit the attribute by clicking on the name of the attribute in the right-side view.
+
+.. figure:: images/train-cifar-epochs.png
+    :align: center
+    :scale: 50%
+
+Inputs and outputs can be added using the blue arrow icons. Any number of inputs and outputs can be added to an operation, but each should be given a unique name.
+
+.. figure:: images/new-operation-io.png
+    :align: center
+    :scale: 50%
+
+Using the plus icon, referencess to resources can be added to the operation. These resources will usually be some form of neural network. As with inputs and outputs, any number of resources can be added to an operation.
+
+.. figure:: images/train-cifar-io.png
+    :align: center
+    :scale: 50%
+
+The paint brush icon allows editing the color of the operation, but this is purely aesthetic and does not affect the operation's underlying logic.
+
+Implementing the Operation
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. figure:: images/operation-code-editor.png
+    :align: center
+    :scale: 50%
+
+In the left-side view, the underlying logic of the operation is implemented using Python. The code here can be edited freely. All operations are defined by a class with the same name as the operation. This class has two primary functions associated with it. The first is the *__init__* function, which will appear automatically when creating the operation's first attribute. This function will run when the operation is initialized and is primarily used for the creation of class variables and the processing of attributes. Note that operation attributes will not be accessible from other functions and must be assigned to a class variable in this function to be utilized elsewhere. The second primary function is the *execute* function. This is the function that is executed when the operation is running. Any number of other classes and functions can be created in the code editor, but they will not be executed if they are not called within the execute function. The outputs of the execute function will also be the outputs of the operation.
+
+Importing Libraries
+~~~~~~~~~~~~~~~~~~~
+.. figure:: images/operation-depen.png
+    :align: center
+    :scale: 50%
+
+Python libraries can be used within an operation by importing them, which is usually done above the operation class. Any library that is installed on the compute backend's python environment can be imported as normal, but more niche libraries that are available through pip or anaconda need to be specified as dependencies for the operation by clicking the *Environment* tab on the right side. The dependencies described here should be defined using the same syntax as in a `conda environment file <https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-file-manually>`_.
+

docs/walkthrough/creating-pipelines.rst

@@ -0,0 +1,27 @@
+Creating Pipelines
+------------------
+From the home view in the *Pipelines* tab, you are presented with a list of the pipelines that have already been created. Clicking on any of these pipelines will allow editing of that pipeline. To create a new, empty pipeline, click on the red button in the bottom right corner of the workspace.
+
+.. figure:: images/pipelines-view.png
+    :align: center
+    :scale: 50%
+
+The basic unit of work in a pipeline is the operation. Operations can be added using the red button in the bottom right corner of the workspace.
+
+.. figure:: images/cifar-pipeline-blank.png
+    :align: center
+    :scale: 50%
+
+After an operation has been added, the attributes of that operation can be changed by clicking on the operation and then clicking on the current value of that attribute.
+
+.. figure:: images/cifar-gt.png
+    :align: center
+    :scale: 50%
+
+Operation inputs and outputs are represented by blue circles that are visible after clicking on the operation. Blue circles on the top on the operation represent inputs, while circles on the bottom represent outputs. The red X circle can be clicked to remove an operation from the pipeline. This does not remove it from the set of available operations. The **</>** icon will open the operation editor view. Holding alt while clicking this icon will instead create a copy of the operation and open the new copy's editor.
+
+.. figure:: images/cifar-operation-io.png
+    :align: center
+    :scale: 50%
+
+Operations can be connected by clicking on an input or output of an operation before clicking on an output or input respectively of another operation. All input and output connections are optional, though missing outputs may give errors depending upon the operation's internal logic.

docs/walkthrough/executing-pipelines.rst

@@ -0,0 +1,60 @@
+Executing Pipelines
+-------------------
+
+This page will guide you through the steps needed to execute a finished pipeline.
+
+Executing within DeepForge
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Finished pipelines can be conveniently executed from within DeepForge. To do so, navigate to the desired pipeline's workspace, hover over the red + button in the bottom right, and click on the blue arrow button. This will open a dialog box for defining how to execute the pipeline. The configuration options are split into several sections. Once all information has been provided, clicking the blue *Run* button will begin execution. The information provided can also be saved for future executions by checking the box in the bottom left.
+
+.. figure:: images/cifar-execute-dialog.png
+    :align: center
+    :scale: 50%
+
+Basic Options
+^^^^^^^^^^^^^
+Here you will define the name of the execution. Execution names are unique identifiers and cannot be repeated. In the case that a name is given that has already been used for that project, an index will be added to the pipeline name automatically (i.e. *test* becomes *test_2*). Upon starting execution, the execution name will also be added to the project version history as a tag.
+
+The pipeline can also be chosen to run in debug mode here. This will allow editing the operations and re-running the pipelines with the edited operations after creation. Alternatively, the execution will only use the version of each operation that existed when the pipeline was first executed. This can be helpful when creating and testing pipelines before deployment.
+
+.. figure:: images/cifar-execute-basic.png
+    :align: center
+    :scale: 50%
+
+Credentials for Pipeline Inputs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+This section requires inputting the credentials for accessing all artifacts used in the pipeline. Because each artifact can be located in different storage backends or different accounts within that backend, each artifact must be provided with its own credentials. If no input artifacts are used in the pipeline, this section will not be present.
+
+.. figure:: images/redshift-execute-creds.png
+    :align: center
+    :scale: 50%
+
+Compute Options
+^^^^^^^^^^^^^^^
+In this section, you will select from the available compute backends. In the examples shown, for instance, the SciServer Compute service will be used. Each compute backend may require additional information, such as login credentials or computation resources that should be used.
+
+.. figure:: images/cifar-execute-compute.png
+    :align: center
+    :scale: 50%
+
+Storage Options
+^^^^^^^^^^^^^^^
+Here, the storage backend must be chosen from the available options. As with the compute options, SciServer's Files Service is used here as an example. Each backend may require additional input, such as login credentials and the desired storage location. This storage backend and location will be where all files created during execution will be stored. This will include both files used during execution, such as data passed between operations, as well as artifacts created using Output operations.
+
+.. figure:: images/cifar-execute-storage.png
+    :align: center
+    :scale: 50%
+
+Manual Execution
+~~~~~~~~~~~~~~~~
+If desired, pipelines can be executed manually by advanced users. Hovering over the red + icon in the pipeline's workspace and clicking the yellow export button that appears will open a dialog box for exporting the pipeline.
+
+.. figure:: images/export-pipeline.png
+    :align: center
+    :scale: 50%
+
+Any artifacts used in Input operations will require the login credentials for the backend and account where the artifact is stored. Clicking the blue *Run* button in the bottom right will generate the execution files for the pipeline and automatically download them in a zip file. In this zip folder are all the files normally generated for execution. The simplest way to execute this pipeline is to run the top-level *main.py* file. 
+
+.. figure:: images/export-pipeline-dialog.png
+    :align: center
+    :scale: 50%

docs/walkthrough/introduction.rst

@@ -0,0 +1,8 @@
+Introduction
+============
+This tutorial provides detailed instructions for creating a complete DeepForge project from scratch. The motivating examples for this walkthrough will be a simple image classification task using `CIFAR-10 <https://www.cs.toronto.edu/~kriz/cifar.html>`_ as our dataset and a more complex astronomical redshift estimation task using `Sloan Digital Sky Survey <https://www.sdss.org/dr13/>`_ as our dataset.
+
+The overall process of creating projects is centered around the creation of data processing **pipelines** that will be executed to generate the data, visualizations, models, etc. that we need. This guide begins with a detailed walkthrough on how to create pipelines and all their constituent parts. After this introductory walkthrough will be detailed walkthroughs on how to create a pair of useful pipelines using the motivating examples.
+
+.. figure:: images/pipelines-view.png
+    :align: center