Merge pull request #43 from nstack/post-idl-fixes

Update the docs to the new DSL and cross-module types

features/index.rst

@@ -19,14 +19,17 @@ Workflows can contain as many steps as you like, as long as the output type of o
 
 We could write the following workflow:
 
-.. code :: bash
+::
    
-  module Iris.Workflow:0.0.1-SNAPSHOT {
-    import Iris.Classify:0.0.1-SNAPSHOT as Classifier;
-    import Demo:0.0.1-SNAPSHOT as Demo;
+  module Iris.Workflow:0.0.1-SNAPSHOT
+  import Iris.Classify:0.0.1-SNAPSHOT as Classifier
+  import Demo:0.0.1-SNAPSHOT as Demo
 
-    def multipleSteps = Sources.http<(Double, Double, Double, Double)> { http_path = "/irisendpoint" } | Classifier.predict | Demo.numChars | sinks.log<Integer>;
-  }
+  def multipleSteps =
+    Sources.http<(Double, Double, Double, Double)> { http_path = "/irisendpoint" } |
+    Classifier.predict |
+    Demo.numChars |
+    sinks.log<Integer>
 
 .. note :: ``numChars`` and ``predict`` can be `composed` together because their types -- or schemas -- match. If ``predict`` wasn't configured to output ``Text``, or ``numChars`` wasn't configured to take ``Text`` as input, NStack would not let you build the following workflow.
 
@@ -39,14 +42,13 @@ All of the workflows that we have written so far have been `fully composed`, whi
 
 For instance, we could combine ``Iris.Classify.predict`` and ``demo.numChars`` from the previous example to form a new workflow ``speciesLength`` like so:
 
-.. code :: java
+::
   
-  module Iris.Workflow:0.0.1-SNAPSHOT {
-    import Iris.Classify:0.0.1-SNAPSHOT as Classifier;
-    import Demo:0.0.1-SNAPSHOT as Demo;
+  module Iris.Workflow:0.0.1-SNAPSHOT
+  import Iris.Classify:0.0.1-SNAPSHOT as Classifier
+  import Demo:0.0.1-SNAPSHOT as Demo
 
-    def speciesLength = Classifier.predict | Demo.numChars;
-  } 
+  def speciesLength = Classifier.predict | Demo.numChars
 
 Because our workflow ``Iris.Workflow.speciesLength`` has not been connected to a source or a sink, it in itself is still a function. If we build this workflow, we can see ``speciesLength`` alongside our other functions by using the ``list`` command:
 
@@ -62,57 +64,56 @@ Because our workflow ``Iris.Workflow.speciesLength`` has not been connected to a
 
 As we would expect, the input type of the workflow is the input type of ``Iris.Classify.predict``, and the output type is the output type of ``demo.numChars``. Like other functions, this must be connected to a source and a sink to make it `fully composed`, which means we could use this workflow it in *another* workflow.
 
-.. code :: bash
+::
 
-  module Iris.Endpoint:0.0.1-SNAPSHOT {
-    import Iris.Workflow:0.0.1-SNAPSHOT as IrisWF;
-    def http = Sources.http<(Double, Double, Double, Double)> | IrisWF.speciesLength | Sinks.log<Integer>;
-  } 
+  module Iris.Endpoint:0.0.1-SNAPSHOT
+  import Iris.Workflow:0.0.1-SNAPSHOT as IrisWF
+  def http = Sources.http<(Double, Double, Double, Double)> |
+    IrisWF.speciesLength |
+    Sinks.log<Integer>
 
 Often times you want to re-use a source or a sink without reconfiguring them. To do this, we can similarly separate the sources and sinks into separate workflows, like so:
 
-.. code :: java
+::
   
-  module Iris.Workflow:0.0.1-SNAPSHOT {
-    import Iris.Classify:0.0.1-SNAPSHOT as Classifier;
+  module Iris.Workflow:0.0.1-SNAPSHOT
+  import Iris.Classify:0.0.1-SNAPSHOT as Classifier
 
-    def httpEndpoint = sources.http<(Double, Double, Double, Double)> { http_path = "speciesLength" };
-    def logSink = sinks.log<Text>;
+  def httpEndpoint = sources.http<(Double, Double, Double, Double)> { http_path = "speciesLength" }
+  def logSink = sinks.log<Text>
 
-    def speciesWf = httpEndpoint | Classifier.predict | logSink;
-  }
+  def speciesWf = httpEndpoint | Classifier.predict | logSink
 
 Separating sources and sinks becomes useful when you're connecting to more complex integrations which you don't want to configure each time you use it -- many times you want to reuse a source or sink in multiple workflows. In the following example, we are defining a module which provides a source and a sink which both sit ontop of Postgres. 
 
-.. code :: java
+::
 
-  module Iris.DB:0.0.1-SNAPSHOT {
-    def petalsAndSepals = Sources.postgres<(Double, Double, Double, Double)> {
-      pg_database = "flowers",
-      pg_query = "SELECT * FROM iris"
-    };
+  module Iris.DB:0.0.1-SNAPSHOT
 
-    def irisSpecies = Sinks.postgres<Text> {
-      pg_database = "flowers",
-      pg_table = "iris"
-    };
-  }   
+  def petalsAndSepals = Sources.postgres<(Double, Double, Double, Double)> {
+    pg_database = "flowers",
+    pg_query = "SELECT * FROM iris"
+  }
+
+  def irisSpecies = Sinks.postgres<Text> {
+    pg_database = "flowers",
+    pg_table = "iris"
+  }
 
 If we built this module, ``petalsAndSepals`` and ``irisSpecies`` could be used in other modules as sources and sinks, themselves.
 
 We may also want to add a functions to do some pre- or post- processing to a source or sink. For instance:
 
-.. code :: java
+::
 
-  module IrisCleanDbs:0.0.1-SNAPSHOT {
+  module IrisCleanDbs:0.0.1-SNAPSHOT
 
-    import PetalTools:1.0.0 as PetalTools;
-    import TextTools:1.1.2 as TextTools;
-    import Iris.DB:0.0.1-SNAPSHOT as DB;
+  import PetalTools:1.0.0 as PetalTools
+  import TextTools:1.1.2 as TextTools
+  import Iris.DB:0.0.1-SNAPSHOT as DB
 
-    def roundedPetalsSource = DB.petalsAndSepals | PetalsTools.roundPetalLengths;
-    def irisSpeciesUppercase = TextTools.toUppercase | DB.irisSpecies; 
-  }   
+  def roundedPetalsSource = DB.petalsAndSepals | PetalsTools.roundPetalLengths
+  def irisSpeciesUppercase = TextTools.toUppercase | DB.irisSpecies
 
 Because ``roundedPetalsSource`` is a combination of a source and a function, it is still a valid source. Similarly, ``irisSpeciesUppercase`` is a combination of a function and a sink, so it is still a valid sink.
 

in_depth_tutorial.rst

@@ -51,13 +51,13 @@ Next, let's download our training data into this directory so we can use it in o
 Defining our API
 ----------------
 
-As we know what the input and output of our classifier is going to look like, let's edit the ``api`` section of ``nstack.yaml`` to define our API (i.e. the entry-point into our module). By default, a new module contains a sample function ``numChars``, which we replace with our definition. We're going to call the function we write in Python ``predict``, which means we fill in the ``api`` section of ``nstack.yaml`` as follows:
+As we know what the input and output of our classifier is going to look like, let's edit ``workflow.nml`` to define our API (i.e. the entry-point into our module). By default, a new module contains a sample function ``numChars``, which we replace with our definition. We're going to call the function we write in Python ``predict``, which means we write our ``workflow.nml`` as follows:
 
-.. code :: java
+::
 
-    api : |
-        predict : (Double, Double, Double, Double) -> Text
+  module Iris.Classify:0.1.0
 
+  fun predict : (Double, Double, Double, Double) -> Text
 
 This means we want to productionise a single function, ``predict``, which takes four ``Double``\s (the measurements) and returns ``Text`` (the iris species).
 

quick_start/module.rst

@@ -22,23 +22,23 @@ You should see the following output confirming that this operation was successfu
 
 .. code:: bash
 
-  ~> mkdir Demo
-  ~> cd Demo
+  ~> mkdir Demo.NumChars
+  ~> cd Demo.NumChars
   ~/Demo> nstack init python
-  python module 'Demo:0.0.1-SNAPSHOT' successfully initialised at ~/Demo
+  python module 'Demo.NumChars:0.0.1-SNAPSHOT' successfully initialised at ~/Demo.NumChars
 
-Because NStack versions your modules, it has given ``Demo`` a version number (``0.0.1-SNAPSHOT``). Because the version number has a ``SNAPSHOT`` appended to it, this means NStack allows you to override it every time you build. This is helpful for development, as you do not need to constantly increase the version number. When you deem your module is ready for release, you can remove ``SNAPSHOT`` and NStack will create an immutable version of ``0.0.1``.
+Because NStack versions your modules, it has given ``Demo.NumChars`` a version number (``0.0.1-SNAPSHOT``). Because the version number has a ``SNAPSHOT`` appended to it, this means NStack allows you to override it every time you build. This is helpful for development, as you do not need to constantly increase the version number. When you deem your module is ready for release, you can remove ``SNAPSHOT`` and NStack will create an immutable version of ``0.0.1``.
 
 A successful ``init`` will have created some files.
 
 .. code:: bash
 
- ~/Demo> ls
- nstack.yaml  requirements.txt  service.py  setup.py
+ ~/Demo.NumChars> ls
+ nstack.yaml  requirements.txt  service.py  setup.py  workflow.nml
 
-This is the skeleton of an NStack module. ``nstack.yaml`` is the configuration file for your module, and ``service.py`` is where the code of your module lives (in this case, it's a Python class). ``requirements.txt`` and ``setup.py`` are both standard files for configuring Python. 
+This is the skeleton of an NStack module. ``nstack.yaml`` is the configuration file for your module, ``workflow.nml`` describes the functions and types defined in your module, and ``service.py`` is where the code of your module lives (in this case, it's a Python class). ``requirements.txt`` and ``setup.py`` are both standard files for configuring Python.
 
-We're going to be concerned with ``nstack.yaml`` and ``service.py``. For a more in-depth look at all these files, refer to :doc:`Module Structure </reference/module_structure>`.
+We're going to be concerned with ``workflow.nml`` and ``service.py``. For a more in-depth look at all these files, refer to :doc:`Module Structure </reference/module_structure>`.
 
 In ``service.py``, there is a ``Service`` class. This is where we write the functions we want to use on NStack. It is pre-populated with a sample function, ``numChars``, that counts the number of characters in some text.
 
@@ -47,7 +47,7 @@ In ``service.py``, there is a ``Service`` class. This is where we write the func
   #!/usr/bin/env python3
   # -*- coding: utf-8 -*-
   """
-  Demo Service
+  Demo.NumChars Service
   """
   import nstack
 
@@ -56,24 +56,15 @@ In ``service.py``, there is a ``Service`` class. This is where we write the func
           return len(x)
 
 
-``nstack.yaml`` is where the configuration for this module lives. NStack fills in the ``service``, ``stack``, and ``parent`` for you, so we don't need to worry about them for now.
-
-.. code:: yaml
+``workflow.nml`` is where you tell NStack which of the functions in ``service.py`` you want to publish as functions on NStack,
+and their input and output schemas (also known as types).
 
-  # Service name (a combination of lower case letters, numbers, and dashes)
-  name: Demo:0.0.1-SNAPSHOT
+::
 
-  # The language stack to use
-  stack: python
+  module Demo.NumChars:0.0.1-SNAPSHOT
 
-  # Parent Image
-  parent: NStack.Python:0.25.0
+  fun numChars : Text -> Integer
 
-  api: |
-    numChars : Text -> Integer
-
-We're going to focus on the ``api`` section, where you tell NStack which of the functions in ``service.py`` you want to publish as functions on NStack,
-and their input and output schemas (also known as types).
 In this instance, we are telling NStack to publish one function, ``numChars``, which takes ``Text`` and returns an ``Integer``.
 
 .. note:: The schema -- or type -- system is a key feature of NStack that lets you define the sort of data your function can take as input, and produce as output. This helps you ensure that your module can be reused and works as intended in production.
@@ -85,18 +76,18 @@ To build and publish our module on NStack, we use the ``build`` command.
 
 .. code:: bash
 
-  ~/Demo> nstack build
-  Building NStack Container module Demo:0.0.1-SNAPSHOT. Please wait. This may take some time.
-  Module Demo:0.0.1-SNAPSHOT built successfully. Use `nstack list functions` to see all available functions
+  ~/Demo.NumChars> nstack build
+  Building NStack Container module Demo.NumChars:0.0.1-SNAPSHOT. Please wait. This may take some time.
+  Module Demo.NumChars:0.0.1-SNAPSHOT built successfully. Use `nstack list functions` to see all available functions
 
 When we run ``build``, the code is packaged up and sent to the server.
 
 We can check that our ``numChars`` function is live by running the suggested ``nstack list functions`` command:
 
 .. code:: bash
 
-  ~/Demo> nstack list functions
-  Demo:0.0.1-SNAPSHOT
+  ~/Demo.NumChars> nstack list functions
+  Demo.NumChars:0.0.1-SNAPSHOT
     numChars : Text -> Integer
 
 That's it! Our ``numChars`` function is live in the cloud, and is ready to be connected to input and output data streams, which the next tutorial will cover.

quick_start/workflow.rst

@@ -40,33 +40,26 @@ Let's create a new directory called ``DemoWorkflow``, ``cd`` into the directory,
 
 .. note :: Just like Python modules, workflow modules are versioned.
 
-.. code :: java
+::
 
- module DemoWorkflow:0.0.1-SNAPSHOT {
-   import Python.Hello:0.0.1 as Hello;
-   // A sample workflow
-   def w = Sources.http<Text> { http_path = "/demo" } | Hello.numChars | Sinks.log<Integer>;
- }
+ module DemoWorkflow:0.0.1-SNAPSHOT
 
-This currently has a single workflow on it, ``w``, which uses a function imported from a module called ``Python.Hello`` with the version number of ``0.0.1``.
-Like the workflow we will create, this example workflow creates an HTTP endpoint which pipes data to a function, and pipes data from the function to the NStack log.
-
-When we created our Python module, we defined the input and output types of our function in our API. On NStack, sources and sinks also have types: this workflow specifies that the HTTP source only receives and passes on ``Text``, and the log only accepts ``Integer``\s. Because our Python function takes ``Text``, counts it, and returns ``Integer``\s, that means it can fit in the middle of the workflow.
-
-Writing our workflow
---------------------
+ import Demo.NumChars:0.0.1-SNAPSHOT as D
 
-First, let's change the import statement to import our *Demo* module instead of *Python.Hello*.
+ // A sample workflow
+ def w = Sources.http<Text> { http_path = "/demo" } | D.numChars | Sinks.log<Integer>
 
-.. code :: java
-
- import Demo:0.0.1-SNAPSHOT as Demo;
-
-Now, let's change our workflow to use the ``numChars`` function on ``Demo``, instead of the one on ``Python.Hello``. 
+This currently has a single workflow on it, ``w``, which uses a function imported from a module called ``Demo.NumChars`` with the version number of ``0.0.1``.
+Like the workflow we will create, this example workflow creates an HTTP endpoint which pipes data to a function, and pipes data from the function to the NStack log.
 
-.. code:: java
+.. note ::
+  There is no need to create a separate module in order to define a
+  workflow. You could have included the definition of ``w`` in the
+  ``workflow.nml`` of the original Python module ``Demo.NumChars``.
+  In that case, you would not need to prefix ``numChars`` with ``D.``,
+  as it is defined in the same module.
 
-  def w = Sources.http<Text> { http_path = "/demo" } | Demo.numChars | Sinks.log<Integer>;
+When we created our Python module, we defined the input and output types of our function in our API. On NStack, sources and sinks also have types: this workflow specifies that the HTTP source only receives and passes on ``Text``, and the log only accepts ``Integer``\s. Because our Python function takes ``Text``, counts it, and returns ``Integer``\s, that means it can fit in the middle of the workflow.
 
 .. note :: The http source is configured in this example to expose an endpoint on ``/demo``. If you are using the demo server, we would recommend changing ``/demo`` to something more unique -- as someone else may have already taken that endpoint.
 

reference/index.rst

@@ -11,6 +11,5 @@ Reference
    module_structure
    supported_types
    workflow_language
-   idl_language
    supported_integrations