Updating quick-start for new module system, and other improvements (#32)

quick_start/index.rst

@@ -10,7 +10,7 @@ By the end of the tutorial, you will learn how to publish your code to NStack an
 
 .. note:: To learn more about modules, sources, and sinks, read :ref:`Concepts<concepts>`
 
-Make sure you have :doc:`installed NStack </installation>`.
+Make sure you have :doc:`installed NStack </installation>` and let's get going. These instructions are for the Linux and Mac OSX versions of the NStack CLI, so adjust accordingly if you are on Windows.
 
 .. toctree::
 

quick_start/module.rst

@@ -3,9 +3,9 @@
 Building a Module
 =========================
 
-NStack Modules contain the functions that can be used on the NStack platform. They are the building blocks which can be used to build workflows and applications.
+*Modules* contain the *functions* that you want to publish to the NStack platform. 
 
-After this tutorial, we will have a simple Python module deployed to our NStack instance, where it can be hooked up to event and data-sources. This module has a single function in it which simply counts the number of characters in some text. 
+After this tutorial, we will have a simple Python module deployed to our NStack instance. This module will have a single function in it which counts the number of characters in some text. 
 
 .. note:: Before starting, check that NStack is installed by running ``nstack --version`` in your terminal. If you got information about the version of NStack you have, you're good to go. If that didn't work, check out :doc:`Installation </installation>` again.
 
@@ -15,7 +15,7 @@ Step 1: ``init``
 
 We want to create a new Python module.
 
-Create a directory called ``demo`` where you would like to build your module (NStack uses the name of the directory as the default name of the module) and ``cd`` into that directory using your terminal.
+Create a directory called ``Demo`` where you would like to build your module and ``cd`` into that directory using your terminal. NStack uses the name of the directory as the default name of the module
 
 To create a new module, run ``nstack init python``.
 You should see the following output confirming that this operation was successful.
@@ -25,27 +25,29 @@ You should see the following output confirming that this operation was successfu
   ~> mkdir Demo
   ~> cd Demo
   ~/Demo> nstack init python
-  python module 'Demo:0.0.1' successfully initialised at ~/Demo
+  python module 'Demo:0.0.1-SNAPSHOT' successfully initialised at ~/Demo
 
-A successful ``init`` will have created some files in the directory.
+Because NStack versions your modules, it has given ``Demo`` a version number (``0.0.1-SNAPSHOT``). Because the version number has a ``SNAPSHOT`` appended to it, this means NStack allows you to override it everytime you build. This is helpful for development, as you do not need to constantly increase the version number. When you deem your module is ready for "release", you can remove ``SNAPSHOT`` and NStack will create a immutable version of ``0.0.1``.
+
+A successful ``init`` will have created some files.
 
 .. code:: bash
 
- > ls
+ ~/Demo> ls
  nstack.yaml  requirements.txt  service.py  setup.py
 
 This is the skeleton of an NStack module. ``nstack.yaml`` is the configuration file for your module, and ``service.py`` is where the code of your module lives (in this case, it's a Python class). ``requirements.txt`` and ``setup.py`` are both standard files for configuring Python. 
 
 We're going to be concerned with ``nstack.yaml`` and ``service.py``. For a more in-depth look at all these files, refer to :doc:`Module Structure </reference/module_structure>`.
 
-In ``service.py``, there is a ``Service`` class. This is where we would write the functions we want to use on NStack. It is pre-populated with a sample function, ``numChars``, that counts the number of characters in some text.
+In ``service.py``, there is a ``Service`` class. This is where we write the functions we want to use on NStack. It is pre-populated with a sample function, ``numChars``, that counts the number of characters in some text.
 
 .. code:: python
 
   #!/usr/bin/env python3
   # -*- coding: utf-8 -*-
   """
-  demo Service
+  Demo Service
   """
   import nstack
 
@@ -59,7 +61,7 @@ In ``service.py``, there is a ``Service`` class. This is where we would write th
 .. code:: yaml
 
   # Service name (a combination of lower case letters, numbers, and dashes)
-  name: Demo:0.0.1
+  name: Demo:0.0.1-SNAPSHOT
 
   # The language stack to use
   stack: python
@@ -70,37 +72,34 @@ In ``service.py``, there is a ``Service`` class. This is where we would write th
   api: |
     numChars : Text -> Integer
 
-We're going to focus on the ``api`` section, where you tell NStack which of the functions in your ``service.py`` you want to turn into functions on NStack,
+We're going to focus on the ``api`` section, where you tell NStack which of the functions in ``service.py`` you want to publish as functions on NStack,
 and their input and output schemas (also known as types).
+In this instance, we are telling NStack to publish one function, ``numChars``, which takes ``Text`` and returns an ``Integer``.
 
 .. note:: The schema -- or type -- system is a key feature of NStack that lets you define the sort of data your function can take as input, and produce as output. This helps you ensure that your module can be reused and works as intended in production.
 
-In this instance, we want to expose one function, ``numChars``, which takes ``Text`` and returns an ``Integer``.
-
-
 Step 2: ``build``
 -------------
 
 To build and publish our module on NStack, we use the ``build`` command. 
 
 .. code:: bash
 
-  ~/demo> nstack build
-  Building NStack Container module demo. Please wait. This may take some time.
-  Module demo built successfully. Use `nstack list functions` to see all available functions
+  ~/Demo> nstack build
+  Building NStack Container module Demo:0.0.1-SNAPSHOT. Please wait. This may take some time.
+  Module Demo:0.0.1-SNAPSHOT built successfully. Use `nstack list functions` to see all available functions
 
-When we run this, the code in the directory is packaged up and sent to the server, where NStack transforms it into a module.
-
-.. note:: Learn more about how NStack packages and runs your module using containers in the :ref:`Architecture <architecture>` section.
+When we run ``build``, the code is packaged up and sent to the server.
 
 We can check that our ``numChars`` function is live by running the suggested ``nstack list functions`` command:
 
 .. code:: bash
 
   ~/Demo> nstack list functions
-  Demo.numChars : Text -> Integer
+  Demo:0.0.1-SNAPSHOT
+    numChars : Text -> Integer
 
-Now that our ``numChars`` function is live on NStack, we can productionise it by connecting it to input and output data. We do this by attaching it to an event *source* and an event *sink* using NStack's Workflow Language. 
+That's it! Our ``numChars`` function is live in the cloud, and is ready to be connected to input and output data streams, which the next tutorial will cover.
 
 Advanced: Framework Modules
 ---------------------------

quick_start/workflow.rst

@@ -3,36 +3,72 @@
 Building a Workflow
 =========================
 
-In the previous tutorial, we built and published a Python module `Demo` using NStack.
-This module had a single function on it, ``numChars``, which counted the number of characters in some text. Although it has been published, we cannot talk to it until we connect it to a `source` and a `sink`.
-In this tutorial, we're going to do this, and we'll be using an HTTP endpoint as a `source`, and NStack's built-in log as a `sink`. When we're finished, we will be able to send some text to an HTTP endpoint, and see the length of the text in our log. 
+In the previous tutorial, we built and published a Python module, `Demo`.
+This module had a single function on it, ``numChars``, which counted the number of characters in some text. Although it has been published, it needs to be connected to a `source` and a `sink`.
 
 .. note:: Sources generate data which gets sent to your function, and sinks receive the data which your function outputs. Learn more in :ref:`Concepts<concepts>`
 
-Let's first refresh ourselves on what the input and output types of our function were by asking NStack:
+Let's refresh ourselves on what the input and output types of our function were by asking NStack:
 
 .. code:: bash
   
   > nstack list functions
-  Demo.numChars : Text -> Integer
+  Demo:0.0.1-SNAPSHOT.numChars : Text -> Integer
 
 This means that our function can be connected to any source which generates ``Text``, and can write to any sink which can take an ``Integer`` as input. 
 
 One of the sources that NStack provides is ``http``;
-if you use this source, NStack sets up an HTTP endpoint which you can send JSON-encoded data to.
+if you use this source, NStack sets up an HTTP endpoint which you can send ``JSON``-encoded data to.
 As a sink, we are going to use the NStack ``log``,
-which is a simple sink for seeing the output from your function.
+which is a sink for seeing the output from your function. We are going to use these two integrations in our tutorial.
 
 .. note:: See a list of available sources and sinks in :ref:`Supported Integrations <supported_integrations>`
 
-In full, our workflow is going to look like this:
+Creating a workflow module
+---------------------------
 
-.. code:: bash
-  import Demo:0.0.1 as Demo;
-  Sources.http<Text> { http_path = "/demo" } | Demo.numChars | Sinks.log<Integer>
+To write workflows, we create a special NStack workflow module, which we create in the same way we create a Python module -- by using ``init``.
+
+Let's create a new directory called ``DemoWorkflow``, ``cd`` into the directory, and create a new workflow module.
+
+.. code :: bash
+  
+  ~/DemoWorkflow/ nstack init workflow
+  Module 'DemoWorkflow:0.0.1-SNAPSHOT' successfully initialised at /home/nstack/Demo/DemoWorkflow
+
+``init`` has created a single file, ``workflow.nml``, which is where we write our workflow module using NStack's scripting language. If we look inside the file, we see that NStack has created an example module for us. 
+
+.. note :: Just like Python modules, workflow modules are versioned.
+
+.. code :: java
+
+ module DemoWorkflow:0.0.1-SNAPSHOT {
+   import Python.Hello:0.0.1 as Hello;
+   // A sample workflow
+   def w = Sources.http<Text> { http_path = "/demo" } | Hello.numChars | Sinks.log<Integer>;
+ }
+
+This currently has a single workflow on it, ``w``, which uses a function imported from a module called ``Python.Hello`` with the version number of ``0.0.1``.
+Like the workflow we will create, this example workflow creates an HTTP endpoint which pipes data to a function, and pipes data from the function to the NStack log.
+
+When we created our Python module, we defined the input and output types of our function in our API. On NStack, sources and sinks also have types: this workflow specifies that the HTTP source only receives and passes on ``Text``, and the log only accepts ``Integer``\s. Because our Python function takes ``Text``, counts it, and returns ``Integer``\s, that means it can fit in the middle of the workflow.
+
+Writing our workflow
+--------------------
+
+First, let's change the import statement to import our *Demo* module instead of *Python.Hello*.
+
+.. code :: java
+
+ import Demo:0.0.1-SNAPSHOT as Demo;
+
+Now, let's change our workflow to use the ``numChars`` function on ``Demo``, instead of the one on ``Python.Hello``. 
 
+.. code:: java
 
-NStack uses the ``|`` operator to connect statements together, just like in a shell such as ``bash``. We use it to connect together the parts to form our workflow.
+  def w = Sources.http<Text> { http_path = "/demo" } | Demo.numChars | Sinks.log<Integer>;
+
+.. note :: The http source is configured in this example to expose an endpoint on ``/demo``. If you are using the demo server, we would recommend changing ``/demo`` to something more unique -- as someone else may have already taken that endpoint.
 
 Let's break these parts to see what we're doing:
 
@@ -46,24 +82,53 @@ Part                                             Description
 ``Sinks.log<Integer>``                           Use NStack's log as a sink. The ``<Integer>`` statement means it can only accept Integers.
 ===============================================  ===========
 
-To start this workflow with NStack, we use NStack's ``start`` command. This opens a command prompt where you can type a snippet and press Ctrl+D to submit it. Alternatively, you can pipe this in from a file.
+NStack uses the ``|`` operator to connect statements together, just like in a shell such as ``bash``. 
 
-.. code:: bash
+Building our workflow
+---------------------
+
+Before we start our workflow, we need to build it in the cloud with NStack. We do this in the same way we build a Python module. We save our ``workflow.nml`` file and run:
+
+.. code :: bash
+
+ ~/DemoWorkflow/ nstack build
+ Building NStack Workflow module DemoWorkflow:0.0.1-SNAPSHOT. Please wait. This may take some time.
+ Workflow module DemoWorkflow:0.0.1-SNAPSHOT built successfully. Use `nstack list all` to see all available functions.
+
+We can now see our workflow is live by using the list command.
+
+.. code :: bash
 
- > nstack start 
- import Demo:0.0.1 as Demo;
- Sources.http<Text> { http_path = "/demo" } | Demo.numChars | Sinks.log<Integer>
- [Ctrl + D]
- Started Sources.http<Text> { http_path = "/demo" } | Demo.numChars | Sinks.log<Integer> as process 1
+  ~/DemoWorkflow/ nstack list workflows
+  DemoWorkflow:0.0.1-SNAPSHOT
+    w : Workflow
 
-We now have a live HTTP endpoint on ``localhost:8080/demo``, running as process ``1`` on NStack. The HTTP endpoint is configured to accept JSON-encoded values. We defined it to use an input schema of ``Text``, so we will be able to send it any JSON ``string``. In our JSON, we put ``params`` as the key, and our input as the value:
+This means our workflow is ready to be started.
+
+Starting and using our workflow
+---------------------------------
+
+To start our workflow in the cloud, we use the start command:
+
+.. code :: bash
+
+ ~/DemoWorkflow/ $ nstack start DemoWorkflow:0.0.1-SNAPSHOT.w
+
+We now have a live HTTP endpoint on ``localhost:8080/demo``. The HTTP endpoint is configured to accept JSON-encoded values. We defined it to use an input schema of ``Text``, so we will be able to send it any JSON ``string``. In our JSON, we put ``params`` as the key, and our input as the value:
 
 We can call it using ``curl``:
 
 .. code:: bash
 
- > curl -X PUT -d '{ "params" : "Foo" }' localhost:8080/demo 
- Success
+ ~/DemoWorkflow/ $ curl -X PUT -d '{ "params" : "Foo" }' localhost:8080/demo 
+ Msg Accepted
+
+When workflows are started, they become *processes* which have numerical ids. We can see the id of our process by running:
+
+.. code :: bash
+
+ ~/DemoWorkflow/ $ nstack ps 
+ 1
 
 And if we look at the log of our process, which we configured as the sink, we will be able to see the result. Because our process was started with an id of ``1``, we run the following:
 
@@ -72,4 +137,4 @@ And if we look at the log of our process, which we configured as the sink, we wi
  > nstack log 1
  Feb 17 09:59:26 nostromo nstack-server[8925]: OUTPUT: 3
 
-Great!
+Great - we can see that the output of our function (and the number of characters in "Foo") is 3.