More complex quick start examples to introduce other concepts #17

lanthias · 2017-02-27T15:17:17Z

What does this PR do?

I've added another section in quick_start to:

Build a more complex module (a classifier that uses some system libraries)
Introduced new sources and sinks
Introduce partial workflows
Introduce workflow modules

Notes

It doesn't yet include more complex types
It is badly organised and needs to be split into separate files

NickPollard · 2017-03-08T11:20:34Z

quick_start/more.rst

+    Building NStack Container module irisclassify. Please wait. This may take some time.
+    Module irisclassify built successfully. Use `nstack list methods` to see all available methods.
+
+We can see our method, ``irisclassify.predict``. Including our ``demo.numChars`` method from the previous tutorial, we should now have two:


This line reads oddly - presumably you mean something like:

We can now see irisclassify.predict in the list of existing methods (along with previously built methods like demo.numChars) by running the suggested command nstack list methods

Yes, thank you

NickPollard · 2017-03-08T11:21:31Z

quick_start/more.rst

+.. code :: bash
+
+   ~/irisclassify/ $ curl -X PUT -d '{ "params" : [4.7, 1.4, 6.1, 2.9] }' localhost:8080/irisendpoint 
+   Success


Instead of Success we now see Msg Accepted here.

NickPollard · 2017-03-08T11:21:49Z

quick_start/more.rst

+   ~/irisclassify/ $ nstack log 2  
+   Feb 17 10:32:30 nostromo nstack-server[8925]: OUTPUT: "Iris-versicolor"
+
+Great! Our classifiier is now productionised.


typo in classifiier

NickPollard · 2017-03-08T11:22:23Z

quick_start/more.rst

+
+Great! Our classifiier is now productionised.
+
+Other Sources and Sinks


I wonder if a better header here would be Connecting to a Database - it's more user-story focused, perhaps more compelling?

NickPollard · 2017-03-08T11:33:58Z

quick_start/more.rst

+
+  ~/irisworkflow/ $ nstack start irisworkflow.completeWorkflow
+
+This paradigm can be helpful when we apply it to sources and sinks. Oftentimes, you -- or someone else in your company -- will want to create sources and sinks which are combined with modules, for instance in the following fictional example:


Maybe we should build a practical example here that someone can go through, for defining sources and sinks?

E.g. define a reusable http endpoint:

def irisEndpoint = source(http:///irisclassify : (Double,Double,Double,Double))

Which they can then use and test? Or we stick with Postgres but make sure we actually have a test db on the server they can use.

I like the former, I think we should mention the latter as well though.

… more content for partial workflows, add new syntax

lanthias · 2017-03-16T11:04:18Z

The last more complex example should probably be split into sections IMO

rjmk · 2017-03-17T17:50:26Z

quick_start/more.rst

+.. code :: python
+
+    def __init__(self):
+        train = pd.read_csv("train.csv")


Even our examples escape our type system!

Not a criticism as it seems like its actually the main use case

jonboulle · 2017-03-23T14:52:42Z

advanced_start/index.rst

+Advanced Tutorial
+******************
+
+In this section, we're going to productionise a Random Forest classifier written with `sklearn`, deploy it to the cloud, and use it in a more sophisticated workflow. 


link to what sklearn is (and probably likewise with random forest)

Do we really really have to use "productionise"?

Good point on the former. On the latter, it's quite a common term but is a bit clunky. Do you prefer operationalise?

jonboulle · 2017-03-23T14:53:26Z

advanced_start/more.rst

+    ~/irisclassify/ $ nstack init python
+    python module 'irisclassify' successfully initialised at ~/irisclassify
+
+Next, let's download our training data into this so we can use it in our module.


into this -> into this directory

(or just remove it)

jonboulle · 2017-03-23T14:54:41Z

advanced_start/more.rst

+Defining our API
+****************
+
+As we know what the input and output of our classifier is going to look like, let's edit the ``api`` section of ``nstack.yaml`` to define our API (i.e. the entry-point into our module). By default a new module contains a sample function ``numChars``, which we can replace with our definition. We're going to call the function we write in Python ``predict``, which means we can fill in the ``api`` section of ``nstack.yaml`` as follows:


we can fill in -> we fill in

jonboulle · 2017-03-23T14:55:22Z

advanced_start/more.rst

+
+.. note :: Python modules must also import ``nstack``
+
+Before we add our ``predict`` function, we're going to add ``__init__``, the Python contructor function which runs upon the creation of our module. It's going to load our data from ``train.csv``, and use it to train our Random Forest classifier:


contructor -> constructor

jonboulle · 2017-03-23T14:57:18Z

advanced_start/more.rst

+
+.. code :: yaml
+
+    packages: ['numpy', 'python3-scikit-learn.x86_64', 'scipy', 'python3-scikit-image.x86_64', 'python3-pandas.x86_64']


it sucks that we have to specify architecture for some of these - is that definitely the case?

Probably not - can it be left out typically with dnf?

jonboulle · 2017-03-23T14:59:15Z

advanced_start/more.rst

+    Building NStack Container module irisclassify. Please wait. This may take some time.
+    Module irisclassify built successfully. Use `nstack list functions` to see all available functions.
+
+We can now see ``irisclassify.predict`` in the list of existing functions (along with previously built functions like demo.numChars) by running the suggested command nstack list functions


demo.numChars -> demo.numChars perhaps

by running the suggested command

jonboulle · 2017-03-23T15:01:36Z

advanced_start/more.rst

+    irisclassify.predict : (Double, Double, Double, Double) -> Text
+    demo.numChars : Text -> Integer
+
+Our classifier is now published, but to use it we need to connect it to an event-source and sink. In the previous tutorial, we used HTTP as a source, and the NStack log as a sink. We can do the same here by starting the following workflow.


an event source and a sink

jonboulle · 2017-03-23T15:07:12Z

advanced_start/workflow_power.rst

+
+.. code :: bash
+
+  module irsiworkflow {


irsi -> iris

Good catch..

jonboulle · 2017-03-23T15:07:39Z

advanced_start/workflow_power.rst

+Partial workflows
+*****************
+
+All of the workflows that we have written so far have been `fully composed`, which means that they contain a source, one or more functions, and a sink. Many times, you want to split up sources, sinks, and functions into separate pieces you can share and reuse. In this case, we say that a workflow is `partially composed`, which just means it does not contain a source, one or more functions, and a sink. These workflows cannot be ``start``\ed by themselves, but can be shared and attached to other sources, sinks, or functions to become `fully composed`. 


Can I create a workflow that's just a source and a sink?

Yes, it's a common theme for data integration.

OK, so "one or more functions" should be "zero or more functions"? (but would suggest tweaking the wording)

Yes good catch

jonboulle

I think this is good for now (please squash)

lanthias · 2017-03-23T17:38:38Z

It's still probably incomplete at then end (it doesn't really have a conclusion), and it stills needs to be changed to use the newest UX syntax on modules and DSL.

rjmk

Just looked at the first file so far

rjmk · 2017-03-29T11:09:20Z

advanced_start/more.rst

+
+.. code:: bash
+
+    ~/ $ mkdir Irisclassify; cd Irisclassify


If we want to follow the directory structure of the examples, this should be Iris.Classify

Cool, I'll update everything to use this format.

rjmk · 2017-03-29T11:10:26Z

advanced_start/more.rst

+
+.. code:: bash
+
+    ~/Irisclassify/ $ curl -O https://raw.githubusercontent.com/nstackcom/nstack-examples/master/iris/irisclassify/train.csv


curl -O https://raw.githubusercontent.com/nstack/nstack-examples/master/iris/Iris.Classify/train.csv

rjmk · 2017-03-29T11:35:59Z

advanced_start/more.rst

+    api : |
+          predict : (Double, Double, Double, Double) -> Text
+
+This means we are exposing a single function ``predict``, which takes a record of four ``Double``\s (the measurements) and returns ``Text`` (the iris species).


Would be kind of cool to show off sum types here. How many species are there?

rjmk · 2017-03-29T11:42:43Z

advanced_start/more.rst

+
+.. code :: bash
+
+    ~/Irisclassify/ $ nstack start "sources.http<(Double, Double, Double, Double)> { http_path : "/irisendpoint" } | irisclassify.predict | sinks.log<Text>"


This is sadly still not possible. You need a complete workflow for nstack start which could be referenced unqualified. The current example has a workflow and a project yaml, which could be recreated?

…tured

lanthias · 2017-03-29T12:28:16Z

This is now ready for review to merge.

More complex quick start

edff96b

This was referenced Mar 1, 2017

Cookbook / Examples #2

Open

Bloomsbury Documentation #18

Closed

lanthias added 3 commits March 6, 2017 14:46

Merge branch 'master' into moreComplexQuickStart

40062f1

Merge new changes

7b8765d

Update to Roman's syntax

d9dd9f7

NickPollard reviewed Mar 8, 2017

View reviewed changes

lanthias added 4 commits March 9, 2017 10:41

Split up more complex examples, integrate nick's comments on #17, add…

7b91878

… more content for partial workflows, add new syntax

:Merge branch 'master' into moreComplexQuickStart

ade74bf

Fixed some typos, titles formatting

6b1e7da

Fixed line type

f8700f9

rjmk reviewed Mar 17, 2017

View reviewed changes

Methods -> Functions, and reorganisation

4d11e75

jonboulle reviewed Mar 23, 2017

View reviewed changes

JB changes

aa8226d

jonboulle approved these changes Mar 23, 2017

View reviewed changes

remove functions from fully composed workflows

3016d6a

lanthias added 2 commits March 24, 2017 19:20

Some updates to new UX

4875f67

Merge branch 'master' into moreComplexQuickStart

41afa3c

rjmk reviewed Mar 29, 2017

View reviewed changes

lanthias added 2 commits March 29, 2017 13:18

updated with new module versioning syntax, fixed a few typos, restruc…

52e082a

…tured

Merge

78d44ed

lanthias changed the title ~~[WIP] More complex quick start examples to introduce other concepts~~ More complex quick start examples to introduce other concepts Mar 29, 2017

Updated module names to use dots

10347f4

mands approved these changes Mar 31, 2017

View reviewed changes

mands merged commit 83f9a18 into master Mar 31, 2017

mands deleted the moreComplexQuickStart branch March 31, 2017 15:45


		Great! Our classifiier is now productionised.

		Other Sources and Sinks


		~/irisworkflow/ $ nstack start irisworkflow.completeWorkflow

		This paradigm can be helpful when we apply it to sources and sinks. Oftentimes, you -- or someone else in your company -- will want to create sources and sinks which are combined with modules, for instance in the following fictional example:


		.. note :: Python modules must also import ``nstack``

		Before we add our ``predict`` function, we're going to add ``__init__``, the Python contructor function which runs upon the creation of our module. It's going to load our data from ``train.csv``, and use it to train our Random Forest classifier:


		.. code :: yaml

		packages: ['numpy', 'python3-scikit-learn.x86_64', 'scipy', 'python3-scikit-image.x86_64', 'python3-pandas.x86_64']


		.. code:: bash

		~/Irisclassify/ $ curl -O https://raw.githubusercontent.com/nstackcom/nstack-examples/master/iris/irisclassify/train.csv


		.. code :: bash

		~/Irisclassify/ $ nstack start "sources.http<(Double, Double, Double, Double)> { http_path : "/irisendpoint" } \| irisclassify.predict \| sinks.log<Text>"

More complex quick start examples to introduce other concepts #17

More complex quick start examples to introduce other concepts #17

Uh oh!

Conversation

lanthias commented Feb 27, 2017

What does this PR do?

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lanthias commented Mar 16, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonboulle Mar 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonboulle left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jonboulle Mar 23, 2017 •

edited

Loading

jonboulle left a comment •

edited

Loading

lanthias commented Mar 23, 2017 •

edited

Loading