Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

Remove scrap files

  • Loading branch information...
commit c00ff7a6baa87909fcc35890a7c2fe98b524a7b8 1 parent fb05ee8
Tim Caswell authored April 10, 2010
144  article.markdown
Source Rendered
... ...
@@ -1,144 +0,0 @@
1  
-Title: The Step of the Conductor
2  
-Author: Tim Caswell
3  
-Date: Fri Apr 02 2010 16:28:54 GMT-0500 (CDT)
4  
-
5  
-There have been several async management libraries proposed and written.  I'm guilty of at least three of them.  The reason for this proliferation of code is that they're all trying to solve a very real problem with writing non-trivial applications that make heavy use of async callbacks.
6  
-
7  
-## Parallel and Serial
8  
-
9  
-Most the libraries to date help solve two main common patterns of function use.  They are parallel execution and serial execution.  In parallel execution you fire off several asynchronous functions and want a common callback to be called then they all finish.  The serial pattern is when you have a chain of steps that can't execute till the previous one is done.  Combining these two patterns gives some pretty flexible uses of async functions without excessive boilerplate or nesting.
10  
-
11  
-### Step
12  
-
13  
-A very small library that I've been using for these simple cases is based on the idea from Will Conant's [flow-js][].  I simplified the idea down to it's core and made some little assumptions to make it easier to use with node's error handling pattern.  I call it [step][].  
14  
-
15  
-Here is a snippet of using `Step` in the [wheat][] blogging engine I'm working on:
16  
-
17  
-    function loadArticle(name, callback) {
18  
-      var props;
19  
-      Step(
20  
-        function readFile() {
21  
-          Git.readFile(path.join("articles", name + ".markdown"), this);
22  
-        },
23  
-        function getAuthor(err, markdown) {
24  
-          if (err) return callback(err);
25  
-          props = markdownPreParse(markdown);
26  
-          props.name = name;
27  
-          loadAuthor(props.author, this);
28  
-        },
29  
-        function finish(err, author) {
30  
-          if (err) return callback(err);
31  
-          props.author = author;
32  
-          callback(null, props);
33  
-        }
34  
-      );
35  
-    }
36  
-
37  
-In this example, I pass three steps as functions to the `Step` helper.  The first two end in a call to an asynchronous function.  I pass the value `this` as the callback.  This hooks's into `Step`'s system so that it know to call the next step when the first is done.  The parameters given to the callback are passed through to the next step.  Notice that I created a closure variable `props`.  This is so that the third step has access to the props defined in the second step, but not passed through by the `loadAuthor` call.  The third step then does some final processing and calls the main callback to the outer function.
38  
-
39  
-In essence `loadArticle` is a composite asynchronous function that had two other asynchronous function calls mixed with other synchronous logic within it.  
40  
-
41  
-How about an example that makes use of the parallel feature of `Step`:
42  
-
43  
-    // Reads the authors in the authors directory and returns a data structure
44  
-    function loadAuthors(callback) {
45  
-      var names;
46  
-      Step(
47  
-        function getFileNames() {
48  
-          Git.readDir("authors", this);
49  
-        },
50  
-        function readFileContents(err, results) {
51  
-          if (err) return callback(err);
52  
-          var parallel = this.parallel;
53  
-          results.files.forEach(function (filename) {
54  
-            var name = filename.replace(/\.markdown$/, '');
55  
-            loadAuthor(name, parallel());
56  
-          });
57  
-        },
58  
-        function parseFileContents(err) {
59  
-          if (err) return callback(err);
60  
-          var authors = {};
61  
-          Array.prototype.slice.call(arguments, 1).forEach(function (author) {
62  
-            authors[author.name] = author;
63  
-          });
64  
-          callback(null, authors);
65  
-        }
66  
-      );
67  
-    }
68  
-
69  
-This example is similar, but with the new addition of the `this.parallel` function.  This parallel function generates a new callback when called and sets an internal counter in the `Step` system.  Though it's hard to see with this example, the arguments to parseFileContents are first a single `err` and then the second argument to each of the `loadAuthor` callbacks.
70  
-
71  
-Perhaps this example will be more clear:
72  
-
73  
-    Step(
74  
-      function loadData() {
75  
-        Git.getTags(this.parallel());
76  
-        loadAuthors(this.parallel());
77  
-      },
78  
-      function renderContent(err, tags, authors) {
79  
-        if (err) return response.simpleText(500, err.stack);
80  
-        var data = {...}; // Truncated for clarity
81  
-        renderTemplate('index', data, this);
82  
-      },
83  
-      function showPage(err, content) {
84  
-        if (err) return response.simpleText(500, err.stack);
85  
-        render(request, response, {
86  
-          title: "Index",
87  
-          content: content
88  
-        });  
89  
-      }
90  
-    )
91  
-
92  
-This is the route handler for the front page of the blog.  It needs data from two different async calls and can't render the main template till they're loaded.  Then after the main template is rendered, the layout can be rendered.  Both `Git.getTags` and `loadAuthors` output two arguments, but their errors arguments are compressed into a single `err`.  If both emitted errors that the latter would overwrite the first.
93  
-
94  
-## More Advanced Patterns
95  
-
96  
-You'll notice in these patterns that there is a fair bit of hacks to fit the cases where the logic isn't exactly parallel or serial.  The closure variables are a kind of limited scope global.  The repeated error handling code is redundant.  Wouldn't it be nice if we could specify which output went to what input and chain arbitrary flows?
97  
-
98  
-## Conductor is born!
99  
-
100  
-Last night while talking with [tmpvar][](Elijah Insua), we decided it would be great to make a system that could calculate arbitrary control flows when given a set of dependencies.  A few productive hours later [conductor][] was born.
101  
-
102  
-Instead of shoe-horning a problem into a preset pattern to make it easier on the computer, why don't we just explain the problem to the computer and let it figure out how to handle it for us?
103  
-
104  
-### Loading an Article
105  
-
106  
-The example from above that uses `Step` could be rewritten to use `Conduct` (the function exported by the [conductor][] library):
107  
-
108  
-<img src="example1.png" style="float:right;margin:0 0 10px 10px" />
109  
-
110  
-    // Define the loadArticle function using Conduct from conductor.
111  
-    var loadArticle = Conduct({
112  
-      M: ["_1", function (name, callback) {
113  
-        // Async function that loads the contents of the markdown file.
114  
-        var filename = path.join("articles", name + ".markdown");
115  
-        Git.readFile(filename, callback);
116  
-      }],
117  
-      P: ["_1", "M1", function (name, markdown) {
118  
-        // Sync function that parses the markdown and adds the name property
119  
-        var props = markdownPreParse(markdown);
120  
-        props.name = name;
121  
-        return props;
122  
-      }],
123  
-      A: ["P1", function loadAuthor(props, callback) {
124  
-        // Async function that loads the author based on props.author
125  
-        loadAuthor(props.author, callback);
126  
-      }],
127  
-      F: ["P1", "A1", function (props, author) {
128  
-        // Final sync function that attaches the author object.
129  
-        props.author = author;
130  
-        return props;
131  
-      }]
132  
-    }, "F1");
133  
-
134  
-
135  
-At first glance this looks like a classic case of over-engineering.  For this simple case you'd be right, but we're keeping it simple for purposes of explanation.
136  
-
137  
-See the [conductor][] README for details on what all the parameters mean.  Basically you're defining a set of functions and their dependencies.  Here we're saying that the "`M`" function needs the first argument to the `loadArticle` function that get's generated
138  
-
139  
-[example1]: example1.png
140  
-[conductor]: http://github.com/creationix/conductor
141  
-[tmpvar]: http://github.com/tmpvar
142  
-[wheat]: http://github.com/creationix/wheat
143  
-[step]: http://github.com/creationix/experiments/blob/master/step.js
144  
-[flow-js]: http://github.com/willconant/flow-js
23  example1.dot
... ...
@@ -1,23 +0,0 @@
1  
-digraph G {
2  
-  size="5,5";
3  
-  
4  
-	subgraph cluster_0 {
5  
-  	style=filled;
6  
-  	color=darkorange;
7  
-  	node [style=filled,color=white,fontcolor=gray6,fontname=Monaco];
8  
-
9  
-		M[label="loadMarkdown"];
10  
-		P[label="parseMarkdown"];
11  
-		A[label="loadAuthor"];
12  
-		F[label="finalize"];
13  
-	}
14  
-  input -> M[label=name];
15  
-  input -> P[label=name];
16  
-	M -> P[label=markdown];
17  
-	P -> A[label=props];
18  
-	P -> F[label=props];
19  
-	A -> F[label=author];
20  
-	F -> output[label=props];
21  
-	input [shape=Mdiamond,color=dodgerblue4,fontname=Monaco];
22  
-	output [shape=Msquare,color=crimson,fontname=Monaco];
23  
-}
BIN  example1.png

0 notes on commit c00ff7a

Please sign in to comment.
Something went wrong with that request. Please try again.