Update/1702 restructure (#38)

* Add flowchart
* Restructuring
* remove curl refs
* add languages file
* Add versioning
* Add cli commands

architecture.rst

@@ -34,6 +34,6 @@ NStack is built using best-of-class technologies, such as:
 Diagrams
 ********
 
-.. image:: arch.png
+.. image:: resources/arch.png
 
-.. image:: arch2.png
+.. image:: resources/arch2.png

concepts.rst

@@ -1,16 +1,38 @@
 .. _concepts:
 
 NStack Concepts
-************
+***************
+
+Example
+-------
+
+.. image:: resources/readme-flowchart-example.svg
+
+
+.. code:: fsharp
+
+  module Demo:0.1.0 {
+    import NStack.Transforms:0.1.4 as T
+    import Acme.Classifier:0.3.0 as C
+
+    // our analytics workflow
+    def workflow = Source.postgresql<(Int, Int)> 
+                   | T.transform
+                   | C.classify
+                   | Sink.s3blob<Text>
+  }
+
 
 .. _module:
-**Modules**
+Modules
+-------
 
 A *module* is a piece of code that has been deployed to NStack, either by you or someone else. It has an input schema and an output schema, which defines what kind of data it can receive, and the kind of data that it returns.
 
 .. _sink_source:
 
-**Sources & Sinks**
+Sources & Sinks
+---------------
 
 * A *source* is something which emits a stream of data. 
 * A *sink* is something which can receive a stream of data.
@@ -19,11 +41,13 @@ Example sources and sinks are databases, files, message-queues, and HTTP endpoin
 
 .. _workflows:
 
-**Workflows**
+Workflows
+---------
 
 Modules, sources, and sinks can be combined together to build *workflows*. This is accomplished using the *NStack Workflow Language*, a simple, high-level language for connecting things together on the *NStack Platform*.
 
-**Processes**
+Processes
+---------
 
 When a workflow is started, it becomes a running `process`. You can have multiple processes of the same workflow.
 

conf.py

@@ -108,7 +108,7 @@
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-#html_theme = 'classic'
+html_theme = 'sphinx_rtd_theme' # 'classic'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the

advanced_start/workflow_power.rst → features/index.rst

@@ -1,14 +1,16 @@
-.. _workflow_power: 
+.. _features
 
-More Powerful Workflows
-=======================
+Features
+========
 
-Now that we have published our classifier to NStack as a module, we can use it to demonstrate some of the more powerful features of the workflow engine.
+In this section, we're going to describe some of the more advanced features you can do with NStack when building your modules and composing them together to build workflows.
 
-Multiple Steps
----------------
+.. _features_composition
 
-Workflows can contain as many steps as you like, as long as the output type of one matches the input type of the other. For instance, let's say we wanted to create the following workflow:
+Composition
+-----------
+
+Workflows can contain as many steps as you like, as long as the output type of one matches the input type of the other. For instance, let's say we wanted to create the following workflow based on the Iris example in :ref:`in_depth_tutorial` and available on `GitHub <https://github.com/nstack/nstack-examples/tree/master/iris>`_
 
 - Expose an HTTP endpoint which takes four ``Double``\s
 - Send these ``Double``\s to our classifier, ``Iris.Classify``, which will tell us the species of the iris
@@ -28,8 +30,10 @@ We could write the following workflow:
 
 .. note :: ``numChars`` and ``predict`` can be `composed` together because their types -- or schemas -- match. If ``predict`` wasn't configured to output ``Text``, or ``numChars`` wasn't configured to take ``Text`` as input, NStack would not let you build the following workflow.
 
-Partial workflows
------------------
+.. _features_reuse
+
+Workflow Reuse
+--------------
 
 All of the workflows that we have written so far have been `fully composed`, which means that they contain a source and a sink. Many times, you want to split up sources, sinks, and functions into separate pieces you can share and reuse. In this case, we say that a workflow is `partially composed`, which just means it does not contain a source and a sink. These workflows cannot be ``start``\ed by themselves, but can be shared and attached to other sources and/or sinks to become `fully composed`. 
 
@@ -114,3 +118,68 @@ Because ``roundedPetalsSource`` is a combination of a source and a function, it
 
 Because NStack functions, source, and sinks can be composed and reused, this lets you build powerful abstractions over infrastructure.
 
+
+.. _features_versioning
+
+Versioning
+----------
+
+Modules in NStack are versioned with a 3-digit suffix that is intended to follow semantic versioning, e.g.::
+  
+  Demo:0.0.1
+  
+This is specified in the ``nstack.yaml`` for code-based modules, and in ``workflow.nml`` for workflow modules.
+A module of a specific version is completely immutable, and it's not possible to build another copy of the module with the same version without deleting it first.
+
+Snapshots
+^^^^^^^^^
+
+When creating a new module, i.e. with ``nstack init``, your module will have the version number (``0.0.1-SNAPSHOT``). 
+The ``SNAPSHOT`` tag tells NStack to allow you to override it every time you build. 
+This is helpful for development, as you do not need to constantly increase the version number. 
+When you deem your module is ready for release, you can remove ``SNAPSHOT`` and NStack will create an immutable version of ``0.0.1``.
+
+.. _features_configuration
+
+Configuration
+-------------
+
+In addition to receiving input at runtime, modules, sources, and sinks often need to be able to configured by a workflow author. To do this, we use brackets and pass in a list of named records: ::
+
+   Sources.Postgres<Text> {
+        pg_host = "localhost", 
+        pg_port = "5432",
+        pg_user = "user", 
+        pg_password = "123456",
+        pg_database = "db", 
+        pg_query = "SELECT * FROM tbl;" 
+    }
+
+For sources and sinks, some parameters are mandatory, and some provide sensible defaults. This is documented in `Supported Integrations <supported_integrations>`_.
+
+To pass configuration parameters to a module, we use the same syntax ::
+
+  FirstLastName.full_name { first_name = "John" }
+
+NStack passes in configuration parameters as a dictionary, ``args``, which is added to the base class of your module.
+For instance, in Python you can access configuration parameters in the following manner:
+
+.. code :: python 
+  
+  class Service(nstack.BaseService):
+  
+      def full_name(self, second_name):
+        full_name = "{} {}".format(self.args.get("first_name", "Tux"), second_name)
+        return full_name
+
+
+.. _features_framework
+
+
+Framework Modules
+-----------------
+
+It is often useful to create a common parent module with dependencies already installed, either to save time or for standardisation. NStack supports this with *Framework Modules*. Simply create a new module similar to above, ``nstack init framework [parent]``, and modify the resulting ``nstack.yaml`` as needed.
+
+You can then build this module using ``nstack build``, and refer to it from later modules within the ``parent`` field of their ``nstack.yaml`` config file.
+

advanced_start/more.rst → in_depth_tutorial.rst

@@ -1,8 +1,13 @@
-.. _more:
+.. _in_depth_tutorial
 
-Productionising a Classifier as an NStack Module
+
+In-Depth Tutorial - Productionising a Classifier
 ================================================
 
+In this section, we're going to productionise a Random Forest classifier written with `sklearn <http://scikit-learn.org/>`_, deploy it to the cloud, and use it in a more sophisticated workflow. 
+
+By the end of the tutorial, you will learn how to build modules with dependencies, write more sophisticated workflows, and build abstractions over data-sources. Enjoy!
+
 So far, we have built and published a Python module with a single function on it, ``numChars``, and built a workflow which connects our function to an HTTP endpoint. This in itself isn't particularly useful, so, now that you've got the gist of how NStack works, it's time to build something more realistic!
 
 In this tutorial, we're going to create and productionise a simple classifier which uses the famous `iris dataset <https://en.wikipedia.org/wiki/Iris_flower_data_set>`_. 
@@ -158,14 +163,10 @@ In this instance, it is running as process ``2``. We can test our classifier by
 
 .. code :: bash
 
-   ~/Iris.Classify/ $ curl -X PUT -d '{ "params" : [4.7, 1.4, 6.1, 2.9] }' localhost:8080/irisendpoint 
-   Msg Accepted
+   ~/Iris.Classify/ $ nstack send "/irisendpoint" '[4.7, 1.4, 6.1, 2.9]'
+   Message Accepted
    ~/Iris.Classify/ $ nstack log 2  
    Feb 17 10:32:30 nostromo nstack-server[8925]: OUTPUT: "Iris-versicolor"
 
-Our classifier is now productionised. Next, we're going explore some of the more sophisticated workflows you can build using NStack.
-
-
-
-
+Our classifier is now productionised.
 

index.rst

@@ -4,20 +4,33 @@
    contain the root `toctree` directive.
 
 NStack Documentation
-====================================
+====================
 
-Welcome to the NStack Documentation!
+Welcome to the NStack Documentation! Please also look at our `main GitHub page <https://www.github.com/nstack/nstack>`_ , and our `online examples <https://www.github.com/nstack/nstack-examples>`_.
+
+What is NStack?
+^^^^^^^^^^^^^^^
+
+.. image:: resources/readme-flowchart-intro.svg
+
+- Productionise your models and data integration code to the cloud as reusable functions that can be monitored, versioned, shared, and updated
+- Effortlessly compose workflows to integrate your functions with your data and event sources, without dealing with any infrastructure
+- NStack containerises and orchestrates your workflows on your cloud provider and makes sure they always work 
+
+Contents
+^^^^^^^^
 
 ..  toctree::
     :maxdepth: 2
 
-    overview
     usecases
     concepts
     installation
     quick_start/index
-    advanced_start/index
-    architecture
+    in_depth_tutorial
+    features/index
+    languages
+    .. architecture
     reference/index
 
 

languages.rst

@@ -0,0 +1,63 @@
+.. _languages
+
+
+Supported Languages
+===================
+
+NStack is language-agnostic and allows you to write modules in multiple languages and connect them together -- currently we support Python, with R and Java coming soon. More languages will be added over time -- if there's something you really need, please let us know!
+
+Python
+------
+
+
+Basic Structure
+^^^^^^^^^^^^^^^
+
+NStack services in Python inherit from a base class, called ``BaseService`` within the ``nstack`` module::
+
+  import nstack
+
+  class Service(nstack.BaseService):
+      def numChars(self, msg):
+          return len(msg)
+
+.. note:: Ensure you import the nstack module in your service, e.g. ``import nstack`` 
+
+Any function that you export within the ``API`` section of your ``nstack yaml`` must exist as a method on this class (you can add private methods on this class for internal use as expected in Python).
+
+Data comes into this function via the method arguments - for ``nstack`` all the data is passed within a single argument that follows the ``self`` parameter. For instance, in the example above there is a single argument ``msg`` consisting of a single ``string`` element that may be used as-is. However if your function was defined as taking ``(Double, Double)``, you would need to unpack the tuple in Python first as follows, ::
+
+  def test(self, msg):
+      x, y = msg
+      ...
+
+Similarly, to return data from your NStack function, simply return it as a single element within your Python method, as in the top example above.
+
+The NStack object lasts for the life-time of the workflow, so if there is any initialisation you need to do within your service you can perform this within the object ``__init__`` method, e.g. open a database connection, load a data-set.
+However remember to call the parent object ``__init__`` method to ensure NStack is initialised correctly, i.e.. ::
+
+
+  import nstack
+
+  class Service(nstack.BaseService):
+      def __init__(self):
+          super().__init__()
+          # custom initialisation here
+
+
+Notes
+^^^^^
+
+* Anything your``print`` will show up in ``nstack log`` to aid debugging. (all output on ``stdout`` and ``stderr`` is sent to the NStack logs)
+* Extra libraries from pypi using ``pip`` can be installed by adding them to the ``requirements.txt`` file in the project directory - they will be installed during ``nstack build``
+
+
+R
+-
+
+Coming soon
+
+Java
+---- 
+
+Coming soon

quick_start/module.rst

@@ -67,7 +67,7 @@ In ``service.py``, there is a ``Service`` class. This is where we write the func
   stack: python
 
   # Parent Image
-  parent: NStack.Python:0.24.0
+  parent: NStack.Python:0.25.0
 
   api: |
     numChars : Text -> Integer
@@ -101,14 +101,4 @@ We can check that our ``numChars`` function is live by running the suggested ``n
 
 That's it! Our ``numChars`` function is live in the cloud, and is ready to be connected to input and output data streams, which the next tutorial will cover.
 
-Advanced: Framework Modules
----------------------------
-
-You may want to create a common parent module that has lots of complex dependencies already installed, either to save time or for standardisation. NStack supports this with _Framework Modules_. Simply create a new module similar to above, `nstack init framework [parent]`, and modify the resulting `nstack.yaml` as needed.
-
-You can then build this module using `nstack build`, and refer to it within your future modules within the `parent` field of their `nstack.yaml` config file.
-
-
-
-