Asynchronous coffeescript made easy, part III

[timbertson]

previous issues:
http://github.com/jashkenas/coffee-script/issuesearch?state=closed&q=narrative#issue/241
http://github.com/jashkenas/coffee-script/issuesearch?state=closed&q=defer#issue/287

So, it's back!

For those who didn't read or can't remember the previous discussions, I have been working on adding a "defer" semantic into coffee-script. This is aimed at making asynchronous coffee-script less painful, by providing a continuation-like callback object for the current function (at compile-time).

For example:

myfunc: (cb) ->
  data: defer call_some_ajax('/path/to/relevant/data')
  alert("got $data")
  cb(data)

Would be transformed into the following javascript (or its equivalent):

function myfunc(cb) {
  call_some_ajax('/path/to/relevant/data', function(data) {
    alert("got" + data);
    cb(data);
  });
}

Special care has been taken such that the following use-cases work as expected:

• multiple return arguments ([err, result]: defer some_call())
• defer calls within:
  • if/else statements
  • switch statements
  • for loops
  • while loops
  • nested expressions.

These are non-trivial, so there may be bugs if you do something utterly weird - let me know! ;)

So, please do check out my deferred branch (http://github.com/gfxmonk/coffee-script/tree/deferred) and let me know what you think. Is it a good idea? Should it be (eventually, after some more cleanup) merged into master? Are there any glaring omissions or bugs? Be sure to look at test/test_deferred.coffee, it has over 30 different tests ensuring that as many language features as I could think of work when defers appear in various locations.

(note that the tests in this branch rely on my coffee testing project, coffee-spec (http://github.com/gfxmonk/coffee-spec). I couldn't have managed this many complex tests without it, and it's hopefully coming to coffee-script itself officially sometime soon)

Stuff not yet done

• I have not yet added the ability to use defer in place of a positional argument - it shouldn't be too hard though.
• There's a fair bit of cleanup that can be done on the code, as well as some possible optimisations on the output of code - I was going for correctness, not efficiency.
• Some form of trampolining to prevent the stack growing indefinitely. This is easy enough to do manually, and I still have no idea how often it will be necessary.
• (potentially) adding a callback argument to a function definition when the compiler detects that one is required. This was a controversial suggestion in my first proposal, so it's been left out for now.

So... thoughts?

[jashkenas]

Here's a direct link to the test_deferred.coffee suite, for folks who'd like to take a peek:

http://github.com/gfxmonk/coffee-script/blob/deferred/test/test_deferred.coffee

[mrjjwright]

Exciting, I hope to try this soon, great work!

[mrjjwright]

I couldn't get this branch to compile, http://gist.github.com/387224. Any ideas? I am sick of nesting code and using 3rd party libs like step and flow for this so I am anxious to try this.

[timbertson]

@mrjjwright: might have been a bad merge, I'll check it out tonight and let you know.

[timbertson]

done! (just pushed). Not entirely sure what the problem was, but it seems to be sorted now. Apologies for that.

[mrjjwright]

It compiles now, thanks.

[mrjjwright]

I tried it on a place today where I was planning to use flow.js. It worked fine. I liked it because I didn't need a library to pull this off. I am always worried that there is a hidden bug in flow.js or step.js. There could of course be bugs in the defer code as well but I am assured that all the code is right there to inspect if it is wrong, and it clear what it is doing. Both flow.js and step.js use underlying state machines that are harder to debug.

The defer keyword definitely illustrates at least theoretically the beauty of CoffeeScript to provide a language solution to an awkward JS usage problem. A developer would otherwise use a library but I think directly generated code is a lot easier to inspect and trust than libraries. I don't understand why developers are reluctant to try CoffeeScript but will gladly pop in any third party library that is barely documented.

I don't want to get too addicted to this yet but I can see how I could easily do so.

+1

[jashkenas]

mrjjwright: for the sake of discussion, it would be great if you could paste in a bit of one of your real-world conversions to the "defer" syntax. I think a lot of the discussion that we'll have will be better informed by "before" and "after" examples with real code...

[mrjjwright]

Ok I am going to try this on what is surely a great test case, a SQLite table migration where you dump all the results of one table to another and then re-import them. This involves about 15 levels of nesting as well as a nested function definition in a very deep defer. I have this working in flow.js and tried migrating it to defer but running into a compilation issue that I have pasted in the comments below the gist (I could be doing something really naive but couldn't find it).

http://gist.github.com/388519

I will probably need your help gfxmonk to get this compiling.

[DanielVF]

If this were in coffeescript, I'd be using it now. Even with the all the virtues of asynchronous programming there are still many of things that need to be done serially. This makes clear, readable code out of it.

+1

[mrjjwright]

gfxmonk helped me get it to compile and it worked like a charm.

http://gist.github.com/388519

A few things to summarize from my experience.

• My code has virtually no nesting now. It didn't have much before because of the flow.js library but now I know the underlying javascript is simply async nested code.
  As I kept adding defer after defer I was wondering if the code generator could handle it. It felt weird but I went with it and it generated all the right levels. This saves tons of keystrokes over plain JS with brackets.
• There is no built-in error handling. I still need to check for a return error from each async call after each invocation and handle them. Not a big deal but just saying.
• Will future readers of my code easily understand defer? Sure if it's documented but it's going to be a bit odd at first. It's a lot better than following nested calls. All you have to do is look up to see where the variables are declared and you can see that the variable was set as a return of an async call.
• I personally would prefer a word like async. I just think it would read cool and make me kind of smug in a good way every time I type it. defer sounds a bit inaccurate and weird to me.

jashkenas I am wanting this, it's going to be hard to not lobby you. But this is my first day with this and I haven't heard or read all the naysayers yet.

[weepy]

I made a suggestion about syntax in an old thread - I'll just repeat it here ...
.... instead of

[err, data] = defer get "/", {}
my_async(x, y)
[err, data] = defer more a, b
setTimeout(defer, 100)
final()    

use a loong arrow

[err, data] = get "/", {}, -->
my_async(x, y)
[err, data] = more a, b, -->
setTimeout -->, 100
final()

I think it neatly encapsulates the idea of 'look to the next line' and also that it's a bit like a function.

[timbertson]

mrjjwright - you might be interested reading the old threads for discussions we had around naming. async has been suggested, but I don't really feel it explains what the keyword does. "call-with-current-continuation" is the best name for it - that comes from smalltalk, but even there it was contracted to call_cc in actual code (which is almost meaningless)

I've been discussing this with a friend, and a couple things should also be pointed out as shortcomings:

• loops containing a defer statement will be transformed into recursive calls. If the deferred function you call is not actually asynchronous, you may well blow the stack. But since an asynchronous return (via ajax, or some other browser callback mechanism) throws away the existing stack anyways, this should not be a problem too often. It can be fixed manually if need be, with appropriately-placed deferred calls to setTimeout.
• as I think we've discussed before, there is no plan to handle exceptions (as you've seen, mrjjwright). They can't easily be handled on a global level - errors in callback-based functions must be caught and explicitly returned into the callback. I've been thinking about ways to make this easier, but I haven't come up with anything compelling yet.

weepy Being a python guy, I typically prefer meningful words over punctuation. Plus, I think your suggestion makes it too easy to accidentally start a function (by forgetting a "-").

But if we move to having the "defer" keyword replacing positional arguments (rather than prefixing function calls) then defer wouldn't necessarily be a great choice, so suggestions are welcomed.

[timbertson]

oh, and:

Will future readers of my code easily understand defer?

Most people dealing with javascript callbacks would already understand continuation-passing-style, and hopefully most have longed for or thought about a way to make that automatic. So hopefully this feature should seem like a fairly natural transformation to most javascript programmers that would encounter it.

[timbertson]

by way of further example, here's an actual javascript file I wrote using lawnchair (a heavily async datastore). I've ported it to both coffe and coffee+defer (reasonably idiomatically, aside from some unimportant string concatenation):

http://gist.github.com/389375
it's easier to see the difference if you open them side by side in a text editor - the bottom "run" method in particular is far more straightforward with defer.