Skip to content
This repository

Asynchronous coffeescript made easy, part III #350

Closed
gfxmonk opened this Issue May 02, 2010 · 95 comments
Tim Cuthbertson
gfxmonk commented May 02, 2010

previous issues:
http://github.com/jashkenas/coffee-script/issuesearch?state=closed&q=narrative#issue/241
http://github.com/jashkenas/coffee-script/issuesearch?state=closed&q=defer#issue/287

So, it's back!

For those who didn't read or can't remember the previous discussions, I have been working on adding a "defer" semantic into coffee-script. This is aimed at making asynchronous coffee-script less painful, by providing a continuation-like callback object for the current function (at compile-time).

For example:

myfunc: (cb) ->
  data: defer call_some_ajax('/path/to/relevant/data')
  alert("got $data")
  cb(data)

Would be transformed into the following javascript (or its equivalent):

function myfunc(cb) {
  call_some_ajax('/path/to/relevant/data', function(data) {
    alert("got" + data);
    cb(data);
  });
}

Special care has been taken such that the following use-cases work as expected:

  • multiple return arguments ([err, result]: defer some_call())
  • defer calls within:
    • if/else statements
    • switch statements
    • for loops
    • while loops
    • nested expressions.

These are non-trivial, so there may be bugs if you do something utterly weird - let me know! ;)

So, please do check out my deferred branch (http://github.com/gfxmonk/coffee-script/tree/deferred) and let me know what you think. Is it a good idea? Should it be (eventually, after some more cleanup) merged into master? Are there any glaring omissions or bugs? Be sure to look at test/test_deferred.coffee, it has over 30 different tests ensuring that as many language features as I could think of work when defers appear in various locations.

(note that the tests in this branch rely on my coffee testing project, coffee-spec (http://github.com/gfxmonk/coffee-spec). I couldn't have managed this many complex tests without it, and it's hopefully coming to coffee-script itself officially sometime soon)

Stuff not yet done

  • I have not yet added the ability to use defer in place of a positional argument - it shouldn't be too hard though.

  • There's a fair bit of cleanup that can be done on the code, as well as some possible optimisations on the output of code - I was going for correctness, not efficiency.

  • Some form of trampolining to prevent the stack growing indefinitely. This is easy enough to do manually, and I still have no idea how often it will be necessary.

  • (potentially) adding a callback argument to a function definition when the compiler detects that one is required. This was a controversial suggestion in my first proposal, so it's been left out for now.

So... thoughts?

Jeremy Ashkenas
Owner

Here's a direct link to the test_deferred.coffee suite, for folks who'd like to take a peek:

http://github.com/gfxmonk/coffee-script/blob/deferred/test/test_deferred.coffee

John Wright

Exciting, I hope to try this soon, great work!

John Wright

I couldn't get this branch to compile, http://gist.github.com/387224. Any ideas? I am sick of nesting code and using 3rd party libs like step and flow for this so I am anxious to try this.

Tim Cuthbertson
gfxmonk commented May 02, 2010

@mrjjwright: might have been a bad merge, I'll check it out tonight and let you know.

Tim Cuthbertson
gfxmonk commented May 03, 2010

done! (just pushed). Not entirely sure what the problem was, but it seems to be sorted now. Apologies for that.

John Wright

It compiles now, thanks.

John Wright

I tried it on a place today where I was planning to use flow.js. It worked fine. I liked it because I didn't need a library to pull this off. I am always worried that there is a hidden bug in flow.js or step.js. There could of course be bugs in the defer code as well but I am assured that all the code is right there to inspect if it is wrong, and it clear what it is doing. Both flow.js and step.js use underlying state machines that are harder to debug.

The defer keyword definitely illustrates at least theoretically the beauty of CoffeeScript to provide a language solution to an awkward JS usage problem. A developer would otherwise use a library but I think directly generated code is a lot easier to inspect and trust than libraries. I don't understand why developers are reluctant to try CoffeeScript but will gladly pop in any third party library that is barely documented.

I don't want to get too addicted to this yet but I can see how I could easily do so.

+1

Jeremy Ashkenas
Owner

mrjjwright: for the sake of discussion, it would be great if you could paste in a bit of one of your real-world conversions to the "defer" syntax. I think a lot of the discussion that we'll have will be better informed by "before" and "after" examples with real code...

John Wright

Ok I am going to try this on what is surely a great test case, a SQLite table migration where you dump all the results of one table to another and then re-import them. This involves about 15 levels of nesting as well as a nested function definition in a very deep defer. I have this working in flow.js and tried migrating it to defer but running into a compilation issue that I have pasted in the comments below the gist (I could be doing something really naive but couldn't find it).

http://gist.github.com/388519

I will probably need your help gfxmonk to get this compiling.

DanielVF

If this were in coffeescript, I'd be using it now. Even with the all the virtues of asynchronous programming there are still many of things that need to be done serially. This makes clear, readable code out of it.

+1

John Wright

gfxmonk helped me get it to compile and it worked like a charm.

http://gist.github.com/388519

A few things to summarize from my experience.

  • My code has virtually no nesting now. It didn't have much before because of the flow.js library but now I know the underlying javascript is simply async nested code.
    As I kept adding defer after defer I was wondering if the code generator could handle it. It felt weird but I went with it and it generated all the right levels. This saves tons of keystrokes over plain JS with brackets.

  • There is no built-in error handling. I still need to check for a return error from each async call after each invocation and handle them. Not a big deal but just saying.

  • Will future readers of my code easily understand defer? Sure if it's documented but it's going to be a bit odd at first. It's a lot better than following nested calls. All you have to do is look up to see where the variables are declared and you can see that the variable was set as a return of an async call.

  • I personally would prefer a word like async. I just think it would read cool and make me kind of smug in a good way every time I type it. defer sounds a bit inaccurate and weird to me.

jashkenas I am wanting this, it's going to be hard to not lobby you. But this is my first day with this and I haven't heard or read all the naysayers yet.

weepy
weepy commented May 04, 2010

I made a suggestion about syntax in an old thread - I'll just repeat it here ...
.... instead of

[err, data] = defer get "/", {}
my_async(x, y)
[err, data] = defer more a, b
setTimeout(defer, 100)
final()    

use a loong arrow

[err, data] = get "/", {}, -->
my_async(x, y)
[err, data] = more a, b, -->
setTimeout -->, 100
final()

I think it neatly encapsulates the idea of 'look to the next line' and also that it's a bit like a function.

Tim Cuthbertson
gfxmonk commented May 04, 2010

mrjjwright - you might be interested reading the old threads for discussions we had around naming. async has been suggested, but I don't really feel it explains what the keyword does. "call-with-current-continuation" is the best name for it - that comes from smalltalk, but even there it was contracted to call_cc in actual code (which is almost meaningless)

I've been discussing this with a friend, and a couple things should also be pointed out as shortcomings:

  • loops containing a defer statement will be transformed into recursive calls. If the deferred function you call is not actually asynchronous, you may well blow the stack. But since an asynchronous return (via ajax, or some other browser callback mechanism) throws away the existing stack anyways, this should not be a problem too often. It can be fixed manually if need be, with appropriately-placed deferred calls to setTimeout.
  • as I think we've discussed before, there is no plan to handle exceptions (as you've seen, mrjjwright). They can't easily be handled on a global level - errors in callback-based functions must be caught and explicitly returned into the callback. I've been thinking about ways to make this easier, but I haven't come up with anything compelling yet.

weepy Being a python guy, I typically prefer meningful words over punctuation. Plus, I think your suggestion makes it too easy to accidentally start a function (by forgetting a "-").

But if we move to having the "defer" keyword replacing positional arguments (rather than prefixing function calls) then defer wouldn't necessarily be a great choice, so suggestions are welcomed.

Tim Cuthbertson
gfxmonk commented May 04, 2010

oh, and:

Will future readers of my code easily understand defer?

Most people dealing with javascript callbacks would already understand continuation-passing-style, and hopefully most have longed for or thought about a way to make that automatic. So hopefully this feature should seem like a fairly natural transformation to most javascript programmers that would encounter it.

Tim Cuthbertson
gfxmonk commented May 04, 2010

by way of further example, here's an actual javascript file I wrote using lawnchair (a heavily async datastore). I've ported it to both coffe and coffee+defer (reasonably idiomatically, aside from some unimportant string concatenation):

http://gist.github.com/389375
it's easier to see the difference if you open them side by side in a text editor - the bottom "run" method in particular is far more straightforward with defer.

John Wright

Hey gfxmonk,

Ok, defer is fine as a keyword. I defer.

Looping issues, no big deal either, I am always really careful about async calls in a loop. (Btw, ever seen this: http://glathoud.easypagez.com/publications/tailopt-js/tailopt-js.xhtml. Any value here?)

Will you be merging all major bug fixes into deferred from the main line for a while so I can use this branch? I am kind of hooked. If not, I will go back to my old crack async dealer, flow.js.

weepy
weepy commented May 05, 2010

How about wrapping setTimeout for numerical values? E.g.

defer 50
# do something in 50ms

You might counter that it wouldn't work for variables that are numeric, but 99% of the time setTimeout's are used with a hard coded numeric value.

Tim Cuthbertson
gfxmonk commented May 05, 2010

weepy: I think that's more of a library issue - special casing in the parser to deal with deferred numbers instead of deferred calls seems a bit of a strange way to handle it. You could do this with a simple library function:

sleep: (delay, cb) -> setTimeout(cb, delay)

then just use it as:

defer sleep 50

mrjjwright: I plan to keep the deferred branch up to date (maybe at a lag of a week or so, because merging is not exactly my top priority). However I can't promise that the syntax won't change (that is, after all, part of the point of this issue / discussion).
Having said that, I plan to port my currently in-progress javascript-heavy webapp to using defer when I get the time.

Dr Nic Williams
drnic commented May 05, 2010

I'm so giddy with excitement over the possibilities for readable nodejs code (as an example)

Tim Cuthbertson
gfxmonk commented May 16, 2010

Okay, I've just pushed a fairly significant change, which cleans up and clarifies the tricky bits of the original attempt. I now believe my branch is in a good state to consider merging into the master. Pity that jeremy has just disappeared for a couple weeks, but at least I shouldn't have to do too many more big merges while he's gone (they are not much fun).

If you want to see the differences, you can take a peek here:
http://github.com/gfxmonk/coffee-script/compare/master...deferred#diff-8
(nodes.coffee is where most of the changes have been made)

It's a big diff, but it's also the first change of its kind. Most of the CS compiler translates one thing into a more primitive version of it (i.e javascript). The deferred machinery, on the other hand, does a lot of rewriting nodes into a manner that will work when calls are "resumed" after an asynchronous call.

The first part of the machinery is basically to pull out deferred calls to the beginning of an expression. That is, if a complex expression contains a deferred call then that call is executed first, and the rest of the expression gets evaluated inside the callback provided to the function. This makes all deferred results available at the time when they are needed - because you can't just pause execution in the middle of an addition operation, for example.

The second (and quite unsightly) part of the machinery is the rewriting. Just as you can't pause execution in the middle of an addition operation, you also can't fire off a call from within an if block and expect your control-flow to still work when your callback is called. In fact, the callback would have to include the rest of the if branch, as well as any operations that sit after the end of the if block in the original code.

So that's where make_control_flow_async comes in. This method is implemented on all nodes. By default it just propagates the call to its children. But in the case of nodes that affect control flow (IfNode, WhileNode, ForNode), it does something pretty invasive. Basically, there's a way to transform each of these nodes into a "flattened" version which manages control-flow by explicit continuations. This method (for each of those nodes) generates the CoffeeScript node objects after that transformation is applied. Because they build up a reasonably different source tree, I've introduced a couple of builder objects (down the bottom of nodes.coffee). I can move them into another file if people feel they clutter the already-huge nodes.coffee file. These builders have named methods for common rewriting operations that make it more obvious what transformation is happening, and why.

I'm happy to explain specific parts of the implementation, if people are interested.

Duncan McKee
mckeed commented June 18, 2010

I just came across this; it's very exciting. I'd like to share my vote for positional argument syntax:

defer setTimeout(continue, 100)

The defer introduces the call that is being deferred to, and (conceptually) binds continue to the continuation that begins on the next line. If you are using this syntax, it is because the function being called focusses on the callback, and I think continue is the clearest description of what the continuation-as-callback does.

Obviously, this could be a problem since continue is already a statement, but I don't think there will be overlap, and it is kind of similar, both being control constructs. Other possibilities are resume, continuation, or cc.

Tim Cuthbertson

mckeed: continue could be confusing because of the existing use (even if it's not ambiguous to the compiler).

jashkenas: any thoughts on the possibility of merging this in sometime? You haven't said a word on this since you came back, not sure if you're busy elsewhere...

weepy
weepy commented June 19, 2010

I was wondering about an alternative syntax using the <- symbol. Something like

err, data:  mongo.find {user: 'nicky'}, <-
throw err if err
process data

So the idea is that the arrow points the other way indicate that the tree branch is being unwound, but keeps parity with the normal CoffeeScript.
It also allows for arbitrary positioning of the callback, e.g.

setTimeout <-, 100
run_delayed_code()

I think it works nicely because it still mostly looks like normal coffeescript (I found defer to be a bit ugly)

Jeremy Ashkenas
Owner

Hey gfxmonk. I've been trying to knock out most the smaller tickets before tackling the big ones, like defer.

There are a couple things you could do to facilitate this...

  • Bring the patch up-to-speed with the latest master (a big pain in the ass, I know, I'm sorry).
  • Make sure the tests that are marked as "these ones cause compile errors at the moment" are now working.
  • Comment in here about some of the parts of the implementation that you think look tricky and/or debatable, and explain why they're necessary.
  • Although test_deferred.coffee is supposed to test all of the possible scenarios, it would be good to have a reference Gist that shows the ways you are supposed to use deferred.
weepy
weepy commented June 20, 2010

I use => quite alot, so it would be important (for me) to be able to be able to specify that the function should be bound.

Tim Cuthbertson

@weepy: I think it works nicely because it still mostly looks like normal coffeescript (I found defer to be a bit ugly)
It's probably unwise to make it look too much like coffee-script, as it still has different semantics (return, for example). I personally get too confused by a language having every imaginable type of ascii arrow, so I'd prefer to steer away from "<-".

As for "=>", I'm not sure what it would mean for a deferred function to be bound - that's about function definition, not a function call...

@jashkenas: geez, that's a lot of changes. I get distracted writing android apps and all the underscores die ;)

  • merging: I've had a go at it. but am finding it difficult to pinpoint why tests are now breaking. I'll keep going...

  • The ones that cause compiler errors aren't actually written yet. the functionality as it stands is complete, the only optional bit is to have a keyword stand in for a positional argument in the unlikely case of not having the callback argument as the last one. To be honest, I don't think that functionality is necessary, so I'd rather not spend time on it and all the merge hell it entails. If people want it, I would suggest adding it after the branch has been merged in (it should be a non-breaking change...)

  • as for the tricky parts of the code, I've detailed how the compiler transforms work in my comments above ( http://github.com/jashkenas/coffee-script/issues#issue/350/comment/242851 ); I don't think there's any tricky or debatable stuff that I haven't mentioned. Feel free to look at the diff and point out any bits that look suss to you though...

  • in terms on ways to use deferred, the actual usage is very simple (the tests are just for completeness). Here's a gist: http://gist.github.com/445525

weepy
weepy commented June 20, 2010

ah - after seeing your examples, i can see that my comment about => isn't really appropriate.
so am i right that:

  • comprehensions work
  • returns work also what is the value of this ? it would be great to see some more of those examples you've got there - perhaps along with the exact output JS ? -- so we can get a good feel for it.
Jeremy Ashkenas
Owner

gfxmonk: Can you explain how defer changes return semantics in your branch? (Presumably other semantics like break and continue) after a return as well... I can see how they'd be problematic, but I'm not precisely sure how you're handling it.

Tim Cuthbertson

weepy: yep, that is correct.
As for this, it's what you'd expect - it's restored after a defer - that is, whatever this is at the start of the function is maintained after making a deferred call. this inside a deferred function call will be whatever it normally is (i.e you can use the fat arrow to bind this at definition time, as you already do).

The exact JS output is not exactly beautiful, but I've now added it to the gist for illustration's sake ( http://gist.github.com/445525 )

jashkenas; regarding return. With defer, you're still writing your functions in a callback style. That is, you take a callback, and you call it with one or more arguments instead of returning a value. The defer machinery makes this look nicer on the call side, so that you don't have to pass an actual function as the callback, it is constructed for you.

Keeping that in mind, there's no difference in return semantics to asynchronous coffee-script. If you return instead of calling your callback, execution will silently cease. So it's the same as normal async code, but that's obviously different to plain-old-procedural coffee-script, which is why I was discouraging making that the defer resemble procedural code too closely.

I'm happy to report that continue and break are indeed problematic, but as far as I know, they are handled properly in all cases :)

Jeremy Ashkenas
Owner

gfxmonk: looks like a couple of things need to be cleaned up in the generated code. To quote:

        return _h(undefined);            //continue;
        return undefined;
      });
    };
    return _h(undefined);
    return _g(undefined);

No need to double-return in either of those places, is there?

Also, for the common-case defer, it would be great to write this:

myFunc = function(callback) {
  var _a;
  someFunction(1, 2, 3, function(_a) {
    var result;
    result = _a;
    doSomeThingWith(result);
    return callback(true);
  });
};

As this, without the variable juggling:

myFunc = function(callback) {
  someFunction(1, 2, 3, function(result) {
    doSomeThingWith(result);
    return callback(true);
  });
};
Tim Cuthbertson

whew, that was a tricky one. all merged and pushed now :)

weepy
weepy commented June 21, 2010

hmm. isn't it confusing that the return doesn't actually return - even though it looks a bit like vanilla coffeescript ?

I think the comprehension parallel processing is cool (something that's rather ugly to do by hand - and I have to do it quite alot) - but I'm not entirely sure what else defer brings to the table, other than "I don't need to indent my callbacks" ? Perhaps I'm missing the point .....

DanielVF

Weepy: defer's usefulness doesn't really show up until you've got lots of calls using it from the same function.

do_stuff: (input, outout_style, callback) ->
  [data, coding]: defer parse_input(input)
  processed_data: defer process_data(data, coding, 'random_argument')
  output: defer generate_output(processed_data, output_style)
  callback(output)
weepy
weepy commented June 21, 2010

is it really anymore readable than

do_stuff: (input, outout_style, callback) ->
  parse_input input, (data, coding) ->
    process_data data, coding, 'random_argument', (processed_data) ->
      generate_output processed_data, output_style, (output) ->
        callback output
Tim Cuthbertson

weepy:

hmm. isn't it confusing that the return doesn't actually return - even though it looks a bit like vanilla coffeescript ?

i'm not sure what you mean - you still have a callback paramater, so you still need to return into it. I have some ideas about making that look more normal, but don't want to pollute the core feature with optional enhancements before it's merged. This was also a large source of contention between matehat and I during the initial implementation, and we both came to the agreement that this was a side matter that should be dealt with independently (if at all).

weepy (#2): you're right, that example is mostly cosmetic. I still think it's useful, but doesn't necessarily justify itself in terms of complexity. The real advantage of the defer feature comes when you use a defer inside a loop, or a comprehension, or an if branch. I've done that manually myself in plain javascript, and it's utterly ridiculous (and very easy to do wrong).

jashkenas: yes, there's certainly some unoptimised code in there. For the first cut, my approach has been to favour simpler compiler code over concise output. I'm reluctant to do much optimisation just yet, because:

  • I'd rather get confidence (from usage) in its correctness before I got adding (potentially complex) optimisations to the compiler code
  • I'm trying to keep the merge footprint small - i.e not adding too much that isn't directly to support the defer feature.

Both of these points would become much less of an issue if the code gets merged into the mainline (I won't have to merge so hard, and changes that break deferred functionality should be found at commit time rather than at merge time).

I'll have a look at the specific examples there, but it may not be worth fixing at this point if it complicates the code futher.

weepy
weepy commented June 22, 2010

gfxmonk. yes - I agree it's certainly very useful inside a comprehension - it is horrid without. Would you mind providing some more examples inside a loop and some if branches? I think the real use cases are extremely useful in showing off your work.
Personally I'm still not sure about the defer in normal javascript outside of the loop: I think it greatly obfuscates the real meaning of the Javascript without really making it much prettier (indentation vs defer keyword).

Tim Cuthbertson

weepy: I don't have many real examples, the test suite is currently probably the best place to look (the coffee-script really is quite readable, with descriptions of what each block demonstrates): http://gist.github.com/448286
If you search for the description of the test you're interested in, you should be able to locate it in the compiled javascript fairly easily (here's a gist: http://gist.github.com/448286 )

Also I forgot to mention that another nicety of defer is that "defer some_call()" is a valid expression, and can therefore be placed within other expressions, rather than having to structure your statements around it.

I personally feel the lack of indentation is a good thing - especially when you do, say, 5 deferred things in a row (as I have). Your code should read top to bottom, not top to squishy-bottom-right ;). But the main drive behind wanting to do this was for the more complex cases.

Jeremy Ashkenas
Owner

gfxmonk: I think we're at a bit of an impasse on this ticket. I can't really merge it back to master until things like the erroneous double-returns and the semantics-of-returning-from-a-defer are fixed and settled. Is this something that you want to tackle, or should we just put defers out of their misery at this point?

Tim Cuthbertson

jashkenas: I've spent a lot of effort on this, and I really think it's a smashingly useful feature. I'm not directly using CS for anything any more, which is why my intensity has waned a little lately. But I think this could be a feature to really make CS shine for new uses (uses for which there currently exists no useful alternative). So I'd very much like to see it merged in so that everyone can actually start using it (as a handful here have already said they would like to).

with respect to those specific problems you mentioned:

  • the double returns are an optimisation issue - having a double return in there is still correct, functioning code. I will of course look into optimising it, but I don't want you to think it's creating incorrect code - just suboptimal.

  • the return semantics have not changed. When matehat and I discussed this a while ago, we disagreed about how (or if) returns should be changed within a deferred function. My view was that the presence of returns should add an additional callback paramater, and then any "return" statement inside a deferred function would be transformed into a call to that function. Some didn't like this because it constricted the use of a callback argument to being the last, which is fair enough. I have had further thoughts about it (see issue 351, which didn't exactly get much support). With that history, I haven't tried to shoehorn in any alternate return syntax into this deferred branch, because if anything is to change, it should be discussed and accepted or rejected on its own merits, rather than being part of this ticket.

To reiterate, there is nothing wrong or semantically awkward with leaving returns as they are - at least no more awkward than they already are in asynchronous-style code. There are opportunities to clean them up and make them nicer for some use cases, but that can be said about any language feature.

Tim Cuthbertson

On the subject of optimisations, I've been looking specifically at the duplicate return case this evening. It's doable to eliminate this oddity, but it introduces more functionality that is only for the sake of defer (I have been trying to keep the footprint on the compiler LOC small). It also only eliminates one (admittedly common) use case. Eliminating cases where a variable is unnecessarily assigned only to be returned immediately
For that gain, it adds nonessential logic which clouds the essential (i.e strictly required for the feature to function) logic. The logic required for the deferred functionality to even work is nontrivial, and I suspect that adding optimisation code into the mix will confuse matters greatly.

I am hardly a compiler writer, so I'm finding this challenging. I do know that old-school compilers (like gcc) perform compilation in a separate step. That is, code generation is responsible for correct code, and the generated AST is passed to the optimiser which is then responsible for simplifying all the patterns that are unnecessary (for example, double returns, or assigning to a new variable when the original value was already just a value instead of a complex expression). I really don't know if this could work for coffee-script, because it's so dynamic and in most cases very closely aligned with its idiomatic javascript. But I suspect that trying to perform both code generation and optimisation for nontrivial transformations such as defer is going to confuse the logic terribly.

Jeremy Ashkenas
Owner

Glad to hear you're still onboard with it, thanks for the notes. Now that 0.7.0 is out, I'll take another stab at a merge and poke around sometime this week.

Jeremy Ashkenas
Owner

Let's start with the return issue. It seems to me that this gets to the heart of defer. Imagine you have a function that uses readFileSync to return a file:

read: (path) ->
  code: fs.readFileSync path
  return code.toString()

puts read __filename

It will print out the contents of the file. You want to change it to make it asynchronous, and so you use defer:

read: (path) ->
  [err, code]: defer fs.readFile path
  return code.toString()

puts read __filename

The return, of course, fails, because even though it's got access to the code, it's caller has vanished because of the defer. gfxmonk: how would you deal with this?

Tim Cuthbertson

jashkenas:
currently, making a function async is not quite that simple. You would also have to add a callback paramater, and call it in your return. That is...

read: (path, cb) ->
  [err, code]: defer fs.readFileAsync path
  return cb code.toString()

this is no different from current async code, except that there is no explicit second argument given to readFileAsync (aside: you have to use readFileAsync with defer, you cant just use readFile). The defer mechanism constructs a continuation argument to pass into functions that expect a callback argument - it does not (currently) alter returns.

My original pass did change the returns in the calling function, by appending a new (hidden) callback paramater, and changing returns to always call it. I no longer think this is a good idea (because it's surprising and doesn't allow enough flexibility). We discussed this previously, here http://github.com/jashkenas/coffee-script/issues/closed#issue/287/comment/170719

we came to the conclusion (with support from others, not just matehat and I) that adding just the defer keyword would be a good first step, in order to let people get used to the feature and see how they want to use it or what else they need from it. As another practical aspect, there were differing opinions of how to alter returns when used with deferred calls (see that thread for some of them).

I still think it's a good idea to have these issues separated. Having said that, if you don't want to merge in the defer branch without also including a change to the way returns are handled, I still think my second proposal here ( http://github.com/jashkenas/coffee-script/issuesearch?state=closed&q=return#issue/351 ) is the best idea for denoting that returns should happen "into" a callback:

my_func: (arg1, arg2, return cb) ->
  value: do_some_stuff()
  return value

which will signify that "cb" is the designated return callback, and thus substitute "return expr" into "return cb(expr)"

Jeremy Ashkenas
Owner

gfxmonk: but this is the point I'm trying to make. The idea with defer is to make async code appear to be sync, syntactically. return, and other things like non-final-position callbacks, break this illusion. To go back to our example:

read: (path) ->
  [err, code]: defer fs.readFileAsync path
  manipulate code
  transform code
  return code

I understand that this doesn't work -- but look at what the function seems to be visually. To all appearances, it looks like you're still in the body of the read function. It looks like you should still be able to return a value. Needing to understand that you have to take an extra callback argument, and then call into it, and then use defer from the calling site, is an unacceptable level of complexity for this feature.

Defer is spooky action at a distance, having unobvious effects on the entire remaining body of the function -- and there's nothing wrong with that -- if we can encapsulate it cleanly, and hide the transformations. If it's too hard or impossible to hide the effects of the transformations at the language level, then I can't merge it in with a clean conscience.

For folks who want to play with this, the gfxmonk/deferred is in a really good place for mucking around with. I recommend checking it out and trying to run this test file:

fs: require 'fs'
[err, code]: defer fs.readFile __filename
puts code.toString()
Henry Greville

Since functions cannot return a value (in the traditional sense) after a defer statement, perhaps it would be prudent, for the moment, to forbid using both return and defer within a single function. This would make it easier to visualize what happens inside a function which uses defer, since a return statement would never be allowed to appear somewhere unless it represents an actual, synchronous return to the caller. Asynchronous and synchronous functions would be visibly distinct form one another. This would also leave open the possibility of reintroducing a semantically-different return statement for asynchronous functions in the future.

Jeremy Ashkenas
Owner

Making sync and async functions look visibly distinct from one another would seem to suggest using indentation (it's CoffeeScript after all) to delimit the scope of the defer. Here's an example that wants to return the event emitter from readFile to the caller as well as do work in the async call.

read: (path) ->
  eventEmitter: defer fs.readFile(path) err, code
    manipulate code
    transform code
  return eventEmitter

Oops, looks familiar, doesn't it?

read: (path) ->
  eventEmitter: fs.readFile path, (err, code) ->
    manipulate code
    transform code
  return eventEmitter

That being, of course, the current way to write it with a callback. In my opinion, defers are valuable if they can make async code appear to work synchronously, by transforming the surrounding code into continuations. If they work so differently that we have to distinguish them visually, then the battle is lost.

Ben Nolan
bnolan commented June 30, 2010

Do not want. In our use case we have seperate error and success callbacks, and the syntax is not obvious that you are using an async block - so I'm against.

Travis Swicegood

Adding my two cents here. This adds a layer of complexity on top of CoffeeScript that I'm not sure is necessary. My brain's wired to think of indentions as a callback - not seeing the indentions makes me think it's all happening synchronously.

I agree with @jashkenas' last assessment here. The two blocks of code are almost equivalent with one having an entirely new syntax that doesn't carry of from JavaScript. With the second one, at least I can apply the "this is how to define a function" knowledge inside CS to the the second parameter and realize it's a function, so it must be a callback.

Cool stuff, though, just not sure it belongs at the language level.

Tim Cuthbertson

I plan to write up a longer post (hopefully over the weekend) on the defer functionality, its rationale and why it's tremendously useful. I'm hoping giving a thorough overview of it will help people understand why I think it's so important. For now, I have a few specific points to respond to:

sethaurus: yep, it's probably a good idea to prohibit returning after a defer. it's always been my opinion that using both is a terible idea.

jashkenas: there are two issues with your trivial example suggesting indentation:

  • firstly, there should be no such thing as the "scope" of a defer - it is the rest of the control flow of the function. Ideally, (with runtime support from browsers, which isn't going to happen) it would be the rest of the program's execution, but that is unfortunately not possible.
  • secondly, you're right - there's no advantage (other than cosmetic) for that example. As I've said before, the real power of defer comes in when you have loops, or if branches - heaven forbid you have both. I hope to explain this better shortly, because it seems to be a point that keeps being missed.

bnolan: are you arguing against defer, or against the suggestion that it should be indistinguishable from synchronous code? In the current defer branch, you would still be taking a success and callback function - is that (plus the use of the very specific "defer" keyword) not sufficient to imply async?

twicegood: disregarding the additional code in the compiler, the extra complexity exists only for people writing async code. You could still write it with callbacks if you wanted, but once you actually write a significant chunk of programming using an asynchronous API, you will not want to.

Tim Cuthbertson

oh, and also this is unfortunately something that cannot really be done at a library level - I wouldn't be trying to change a compiler if it could be. See async.js for an attempt - it's one of the most clunky and error-prone libraries I've ever tried to use, by no fault of the designer.

Jeremy Ashkenas
Owner

gfxmonk: I'm very much looking forward to the (blog?) post.

Travis Swicegood

gfxmonk: I'd love to see the post showing some examples.

The quick examples I saw above -- and in fairness I gave them < 60 second of review -- looked as though they could have been simplified by abstracting out to a single function with the final callback.

I've run into something I consider a use-case for this -- doing basic queries against MongoDB. It requires that you establish a connection, then select a collection, then perform your query, then finally doing something with that query. A series of defer calls would "fix" the nested issue, but I addressed it by simply extracting out the boiler plate code and passing the final callback (the result of the insert/find/etc) to a storage.insert. My code was nested, but in other functions passing the callback around to compose the end-result.

By abstracting it, I was also able to give it an intent revealing name that helped with readability of someone looking at the code for the first time with no prior knowledge on their part where a series of defer calls would have added extra parsing to it on their part to fully appreciate what was happening.

It's too bad this can't be done as a library -- that's what this really feels like to me.

Stan Angeloff

I'd love to see an up-to-date diff for the defer functionality. The one provided earlier reports 74 changed files with 3799 additions and 2002 deletions which I believe is pretty outdated. Any way to review the changes?

Tim Cuthbertson

oops, looks like I'd pushed deferred without pushing master. That same diff link shows something much more reasonable now. It's still got the noise from the compiled js differences, but it's much better.

Tim Cuthbertson

okay, I've published my post on the motivation and need for defer, and I'd really appreciate it if you could have a read:

http://gfxmonk.net/2010/07/04/defer-taming-asynchronous-javascript-with-coffeescript.html

It's a long post, but I wanted to make sure I didn't skip any of my reasoning or lose anyone. I think I've got my head in asynchronous coding a lot, and a lot of the things I assume about how it should work aren't necessarily widely known or believed - so I wanted to explain them as thoroughly as possible so that people could understand why I care about defer so much. Please let me know what you think after reading this, either here or in the comments on that page (if it pertains directly to defers inclusion in coffee-script, here is probably better)

elpollodiablo

I would SO love to see this go into cs, the async programming style bugged me to no end until I found Deferred() in twisted ...

Ben Nolan
bnolan commented July 04, 2010

I understand why people want it - using the native mongo wrapper is an exercise in block indentation, and the idea of defer is good - but I think breaking the "indent a block" syntax is very confusing and a bit of a slippery slope. I'd prefer a solution that was as nice as possible - but without requiring compiler support, maybe something like the jasmine run/waits/run/waits syntax.

I note that promises were built into node.js, then removed in favour of people using their own solution.

edit: great job on the article btw, very well written :)

Jeremy Ashkenas
Owner

That's a seriously wonderful article. Nice work writing it up -- and for making it accessible to folks who don't deal with asynchrony in JavaScript on a day-to-day basis.

After reading it, however, my personal take on it is largely unchanged: defer would be great to have in the language if:

  • We generate the async code you would have written, and not 50 lines of tangle.

  • We have a way to propagate the "defer" status of the function -- automatically generating the callback argument, and enforcing that the callback is used, instead of a return.

Until that's done, I can't in good conscience merge it to master. Master becomes the next release, and people rely on it being in a usable state for their current CoffeeScript projects.

So -- agreeing that simple cases of defer are largely immaterial -- let's take for an example the function that you use in your blog post:

get_all_documents: (callback) ->
  documents: (defer database.get(id) for id in document_ids)
  callback(documents)

Using your deferred branch, that function compiles into this:

var get_all_documents;
get_all_documents = function(callback) {
  var _a, _c, _d, _e, _f, _h, _i, _j, _k, _l, _m, id;
  (function(_b) {
    _c = (function() {
      _j = []; _l = document_ids;
      for (_k = 0, _m = _l.length; _k < _m; _k++) {
        _a = _l[_k];
        _j.push(_a);
      }
      return _j;
    })();
    _d = [];
    (function(_g) {
      _h = function() {
        if (!(_c.length > 0)) {
          return _g(undefined);            //break;
        }
        id = _c.shift();
        database.get(id, function(_o) {
          _f = _o;
          _d.push(_f);
          return _h(undefined);            //continue;
          return undefined;
        });
      };
      return _h(undefined);
      return _g(undefined);
    })(function(_n) {
      _n;
      return _b(_d);
    });
  })(function(_i) {
    var documents;
    documents = (_i);
    return callback(documents);
  });
};

I don't think that's going to fly. Unlike other languages, where the size of the generated code is irrelevant, in JavaScript, every byte that gets sent down to the browser counts. If we want to get this slimmed down, it might be worth disallowing continue and break within the deferred body -- along with the currently disallowed return.

So going back to that get_all_documents function, I'd currently write it in CoffeeScript like this:

get_all_documents: (callback) ->
  documents: []
  for id in document_ids
    database.get id, (contents) ->
      documents.push contents
      callback documents if documents.length is document_ids.length
  true

Certainly more verbose than the original, but it compiles to this JS:

var get_all_documents;
get_all_documents = function(callback) {
  var _a, _b, _c, documents;
  documents = [];
  _b = document_ids;
  for (_a = 0, _c = _b.length; _a < _c; _a++) {
    (function() {
      var id = _b[_a];
      return database.get(id, function(contents) {
        documents.push(contents);
        if (documents.length === document_ids.length) {
          return callback(documents);
        }
      });
    })();
  }
  return true;
};

Which is JS that I can live with...

Tim Cuthbertson

Well, I was hoping you'd be a little more convinced by the article, but such is life. Let's tackle the issues shall we?

Generating the code you would have written is a fair point, and in most cases I believe this to be already mostly done. A for loop is by far the least optimised case, and the reason for that is quite simple: I am entirely terrified of having to re-implement ForNode.compile. In order to compile a node that uses defer, the node's control-flow must be flattened. This is done by breaking apart basic blocks into functions, and stringing them together explicitly. This allows for a loop iteration to continue from inside a callback, for example.

Doing this for ForNode was a daunting enough task that I've taken a shortcut - flattening a for-loop means turning it into a non-deferred for-loop to construct an array of the objects to iterate over, and then a fairly trivial while loop to actually iterate over those values. Having this initial pass be an actual for-loop means I don't have to re-implement all the logic for ForNode.compile, the self-described hairiest method in all of CoffeeScript.

So that can certainly be cleaned up, but until I understand ForNode.compile, it can't be done by me. Plus if the code were to be duplicated, it could then get out of sync, and features that work in a regular for loop won't work in a deferred one. That would be bad.

A possible solution would be if ForNode.compile could produce basic nodes for the individual component parts as an intermediate step, instead of spitting out strings directly. If we could split out the node generation from the convert-to-string step, then the code could be re-used for the construction of the flattened loop.


The second point (about returns) is the subject of another post I've just published. It discusses four approaches for dealing with returns and their pros and cons:

http://gfxmonk.net/2010/07/06/dealing-with-return-in-deferred-coffeescript.html

Can you folks please let me know what you think, which of these solutions are workable / preferable for your respective use cases?

Jashkenas: It sounds like you're wanting proposal #1 (which was the original implementation), but I think the downsides of that approach outweigh the conveniences.

Jeremy Ashkenas
Owner

As to cleaning up the ForNode compilation, etc. I can certainly lend a hand with that -- and it might lead to some nice-to-have refactors.

As to the how-to-deal-with-return issue, I think I'd prefer something like this: Any function that uses defer is tagged by the compiler as an async function. Inside of it, returns are calls to the callback.

get_document: (id) ->
  document: defer db.get(id)
  return document

Is the equivalent of this CoffeeScript:

get_document: (id, _cb) ->
  db.get id, (document) ->
    _cb document

This is your proposal #1. For the record, Python does something similar with the yield statement -- any function that contains yield becomes a generator, and does not execute like a normal function does.

The more explicit approaches (#2, #3, and #4) are just a little too nasty to justify. If you're at the point where you're annotating the argument list, you might as well just be using the callback function. I'm more in favor of defer being opinionated about the structure of the callbacks it allows (and compatible with Node.js's style) -- you can always use vanilla callbacks if you've got something fancy, (like onSuccess and onError in an options hash).

DanielVF

I prefer the returning defer (#1). It less code to write when used, less error prone, and lets you work at higher level of abstraction. Like the rest of coffeescript, it changes the useful but ugly, into the useful and beautiful.

Ben Nolan
bnolan commented July 06, 2010

The get_document example makes me feel very uncomfortable - but you're saying that if I ban defer from my apps - it won't change the execution flow of normal coffeescript?

Tim Cuthbertson

bnolan: correct. It would also be easy enough to implement a wrapper that took both ok and error callbacks, and called a defer-using function (and routed the responses appropriately).

But yes - if you don't use the defer keyword, the compiled code will be identical to the current output.

If we are to use #1, I would assume we would always want to generate a runtime check for every defer-using function to ensure that we received a callback argument that is of type Function. Is this too invasive / inefficient? We could turn it off with a compiler flag, but it's certainly something you will want for development.

Jeff Larson

I have two worries about the the defer keyword in coffeescript:

  1. Callbacks are a style in javascript not a fundamental basis of the language. Because of the fact that functions are first class objects, the use of callbacks as an idiom is almost a given, but the fact remains that by including defer coffeescript would be dictating a style. If we all agree that coffeescript should be "unfancy javascript" including it would betray coffeescript's underpinnings.
  2. As a corollary to the above, first class functions are JavaScript, so hiding functions, especially non-trivial functions like callbacks, seems a bit anti-javascript* to me. Essentially what defer does is make the code appear to be iterative when the underlying operation is actually recursive. And even further, implies that the final line of the program is the endpoint rather than the body of the callback being the endpoint.

The inspiration for this discussion seems to arrive out of two places: other languages (scheme), and other frameworks built on top of languages (twisted, eventmachine). In the former, the argument falls flat because javascript is not another language, and coffeescript is a superset of javascript and claims to be such**.

In the latter argument, I'd offer that the lack of braces and language cruft allows for an almost perfect language for dsls (as in ruby) and libraries. For example, is

defer database.get(id) for id in document_ids

that much different than:

defer cb, () ->
  document_ids.shift

Where defer would then be an iterator that consumes document_ids until it can consume no more and then calls cb. I realize that the pain of writing a library was a big part of beginnings of this discussion, but I'd propose that freezing a Way of Doing Things into a language is dangerous at best, and much more likely to be disastrous. Libraries come and go and are easily improved upon, Languages are for the ages.

* N.B. if a feature is against the language as it stands, is it a bug?

** This may seem to imply that I believe c is a superset of assembly, for example, however, c does not claim to be better assembly.

Tim Cuthbertson

thejefflarson:

1: It's not entirely dictating a style, since regular (non-deferred) coffeescript is still perfectly fine (and can be mixed / matched with defer-using coffeescript). It's also a style that is encouraged / used by most libraries and for all of node.js (which CS runs on, outside the browser).

As for unfancy javascript, coffeescript already adds features to JS (e.g list comprehensions). It's up to jashkenas obviously, but I see this as a useful feature to simplify and ease the awkwardness of javascript development, much like the other features CS has that JS does not.

  1. we're not proposing hiding functions, just functions-as-used-for-callbacks-when-you-use-defer. If you want the callback to be the endpoint, you can still use explicit callbacks just fine. Defer is not for all situations, just those where you want to emulate continuing after an async call.

javascript not being another language is an interesting point, because web development is pretty much stuck with javascript. If this were a server-side-only language, I probably wouldn't need to add defer - I'd just use a different language (or blocking operations in threads, which is not possible with JS).

   defer cb, () ->
     document_ids.shift

Where defer would then be an iterator that consumes document_ids until it can consume no more and then calls cb.

I'm not sure what you mean. Where do you put the call to db.get() ?

once you call db.get, your function cannot do anything but return - and then you cannot do any further work outside a callback. This cannot be done as a library, otherwise I would have already. That's the most important point to realise, really.

Karl O'Keeffe
karl commented July 08, 2010

I've been watching this thread with interest for a while, and I think I come down slightly on the side of excluding this from the core language for now. Mainly because it seems to bake in a very specific way of handling asynchronous callbacks into the language, without a definite consensus that it is the best way of handling asynchronous code.

I'd prefer to see defer as a library for now. I know that means it won't be able to work quite as smoothly, but that is a small price to pay to avoid the risk of accidentally baking an sub optimal way of handling asynchronous code into the language.

I've had a little play around and it seems like you can have some nice helper methods that make asynchronous comprehensions easy to deal with from a library level. e.g.

# Library method to handle asynchronous comprehensions
map_async: (collection, each_cb, final_cb) ->
  results: []
  for element in collection
    each_cb element, (new_element) ->
      results.push new_element
      final_cb results if results.length is collection.length
  true

# Using a library
get_all_documents_1: (callback) ->
  map_async document_ids, database.get, callback

# Using a library (more explicit passing of variables)
get_all_documents_2: (callback) ->
  map_async document_ids, (id, cb) ->
    database.get id, cb
  , (documents) ->
    callback documents

# The same code using the defer branch
get_all_documents_defer: (callback) ->
  documents: (defer database.get(id) for id in document_ids)
  callback(documents)

With these the library would be almost as nice to use as the defer branch, but without language having to commit to it permanently.

What are other peoples thoughts on this?

Michael Fellinger

I'd really like to see this added to the language, defer pretty much generates idiomatic code that simply cuts out nesting boilerplate and mental overhead.
Even with defer in CS, it's still possible to use other libraries if one chooses to, so i don't think there is anything to be lost.
I don't think anybody is claiming that it is the best way to go, but waiting indefinitely for a better way won't be beneficial either, so I'd rather be able to use it right now since it seems good enough for me.

Jeff Larson

gfxmonk:

Let me first say that the defer branch is a fantastic feat, there's some nice stuff in there.

But I will say that my example was the final api of a library, just as defer would be a final feature of coffeescript. I should have thought it out a bit more and I'd write what I meant, but I see karl has beaten me to the punch.

Three things:

1) I'm not saying to use another language. I know we're all polyglots, but by the very fact that we're here, we each love javascript a whole bunch. What I am saying is that hiding the nature of javascript because other languages have a feature is a non-argument.

2) In a list comprehension coffeescript combines primitives to accomplish its task. From the docs:

globals: (name for name of window)[0...10]

becomes this coffeescript:

var _a, _b, globals, name;
var __hasProp = Object.prototype.hasOwnProperty;
globals = (function() {
  _a = []; _b = window;
  for (name in _b) { if (__hasProp.call(_b, name)) {
    _a.push(name);
  }}
  return _a;
})().slice(0, 10);

There's nothing in there that is not explicitly defined in the ECMAscript spec. Now while the ECMAscript spec does talk about callbacks (forEach for example), it does not define a true nature for callbacks. To put it a little more clearly: callbacks are a corollary to the language not a primitive.

3) And this is the most important part. I still believe that it's a mistake to make code look iterative when it is recursive, which is the reason for being for defer. A traditional (iterative) program runs like so:

statement
statement
statement
statement
alwaysExecuted()
exit # end of program
neverExecuted()

A javascript (recursive-functional) program with callbacks runs like so:

var _cb = function(){ neverExecuted() }
func(args, function(){
  process.exit(); // end of program
  _cb()
});
alwaysExecuted(); // as long as func is truly async

Now with defer it runs like so:

defer func args
process.exit()
return neverExecuted()

alwaysExecuted() // uh oh

This is a cognitive problem. When I write async code, I assume that sometime in the future the code will be executed, so often I'll go about other business while I wait for it to execute. But what defer does is change the meaning of the code, it takes what I know to be async, and changes it into an iterative call which is a slight of hand I'm not entirely sure is necessary. I'd further propose that it's a special case (albeit a popular one), and to change the behavior of language for a special case seems extreme.

Tim Cuthbertson

thejefflarson::

I'm convinced that a library is destined to be too noisy and awkward. The above example of map has not one but two callbacks, which one would typically write inline (they're not likely to be named functions, just steps to perform). A function call that takes two inline callable blocks is far from readable. And that still only solves a very specific case - map. Implementing filter, and if branches and everything else is all possible, but it will look hideous and will be a pain to use. I'm happy to be proved wrong, but I'm tired of people telling me I should do it in a library when I have already tried, and do not consider it acceptable. (yes, I have written something identical to the map function above. actually using it is horrible)

1) I'm not saying to use another language. I know we're all polyglots, but by the very fact that we're here, we each love javascript a whole bunch. What I am saying is that hiding the nature of javascript because other languages have a feature is a non-argument.

The fact that a feature has been implemented and shown to be immensely useful in other programming languages is no guarantee that it's a good idea, but it's hardly irrelevant either.

As an aside, I don't love javascript in the least - and if everyone loved javascript, there would be little point in coffeescript...

2) In a list comprehension coffeescript combines primitives to accomplish its task. From the docs:

(snip)

There's nothing in there that is not explicitly defined in the ECMAscript spec. Now while the ECMAscript spec does talk about callbacks (forEach for example), it does not define a true nature for callbacks. To put it a little more clearly: callbacks are a corollary to the language not a primitive.

functions are a primitive. CS uses primitives to turn a for loop into a map / filter operation. defer uses primitives to provide a continuation to a function call. We're providing abstractions over the core primitives where it makes sense to do so, in order to create more meaningful syntax - sounds pretty similar to me.

The problem of deferred code appearing to be sync is only true if you ignore the defer keyword. Trying to understand code by its syntax alone will not get you very far if you don't understand the semantics. defer makes much nicer syntax for mostly the same (but simplified) semantics. Again, you don't have to use it - or you can use it only when it makes sense to. Just like list comprehensions don't remove the ability to write a for loop, defer doesn't prevent you from managing your callbacks yourself.

Joshua Peek
josh commented July 15, 2010

I really love the idea of defer and I'm patiently waiting for it in CS master.

My syntactic proposal is to change the assignment operator instead of the defer keyword. Haskell has something very similar regarding sequencing events with its do block. They have a special <- assignment-like operator that cleans up staircasing.

with defer:

append: (filename, str) ->
  [err, data]: defer fs.readFile filename
  err: defer fs.writeFile filename, data + 'bar'
  return;
  console.log 'done'

<- operator:

append: (filename, str) ->
  [err, data] <- fs.readFile filename
  err <- fs.writeFile filename, data + 'bar'
  return;
  console.log 'done'

translates to:

append: (filename, str, cb) ->
  fs.readFile filename, (err, data) ->
    fs.writeFile filename, data + 'bar', (err) ->
      cb()
      console.log 'done'

Whats nice is that <- is not a valid function name like defer could be. It also makes it clear this isn't a normal assignment. And it has this nice parity with the -> lambda shorthand.

(Sorry if you already discussed something like this, I'm jumping into this discussion for the first time and there's alot of backlog ;)

Jeremy Ashkenas
Owner

Plus one to Josh's suggestion.

<- used to be a different CoffeeScript operator, serving the equivalent purpose to ECMA5's Function#bind. But we've removed it from recent versions of CoffeeScript, freeing it up to be put to other uses -- and in my opinion this is an excellent use indeed.

It visually signifies the inverted call order of the deferred function, and makes it obvious that normal assignment isn't happening. In fact, we could lose the destructuring array assignment, and just do this:

(err, data) <- fs.readFile filename

It doesn't solve our return problem, but it brings things very close...

Henry Greville

Another +1 for the <- syntax. I like that it emphasizes how the assignment itself is asynchronous, and is intermediated by a callback function. Also agree that we use parameter-list-style parens (potentially with splats) instead of array brackets to capture the return value/s.

getOne: (callback) ->
    callback 'one'
getTwo: (callback) ->
    callback 'one', 'two'
getMany: (callback) ->
    callback 'one', 'two', 'three'
(x) <- getOne() # x is 'one'
(x, y) <- getTwo() # y is 'two'
(x...) <- getMany() # x is ['one', 'two', 'three']
Joshua Peek
josh commented July 15, 2010

Had an observation about some simple CPS statements

I've seen this pattern frequently to curry on arguments to an async call.

getPeople: (callback) ->
  ajax.get "/people", callback

You could write something that accomplishes the same thing with the new syntax.

getPeople: () ->
  data <- ajax.get "/people"
  return data

# compiled
getPeople: (callback) ->
  ajax.get "/people", (data) ->
    callback data

The compiled code isn't quite the same because of the extra function wrapper. I'm not sure how you'd want to write it to take advantage of the implicit continuation argument. Maybe we could optimize away these identity functions?

More involved example

getAuthor: () ->
  post <- posts.get "1"
  person <- people.get post.author_id
  return person

# compiled
getAuthor: (callback) ->
  posts.get "1", (post) ->
    people.get post.author_id, (person) ->
      callback person


# Something similar would
getAuthor: () ->
  post <- posts.get "1"
  people.get post.author_id

# generate this ???
getAuthor: (callback) ->
  posts.get "1", (post) ->
    people.get post.author_id, callback

I really like the idea of making return work nicely, but it brings up alot of issues like this.

Tim Cuthbertson

I was a fan of <- for currying, but if it's free now then I think it fits well with defer (especially since continuation is a monad in haskell, so we can all pretend to be using haskell ;))

josh: there's certainly some amount of work that can be done towards optimisation. Personally, I'm not that interested in it (I think you should probably just use a JS optimiser), but it's certainly possible. Although without a clear separation between the AST and code generation, the optimisations are likely to make for ever-more-confusing compiler code. But yes, we can hopefully tackle the most obvious of inefficiencies at least.

Tim Cuthbertson

hmm, some problems just occurred to me with "<-". It's nice for an assignment, like so:

foo <- defer someOperation()

however when not used in an assignment, it would look pretty odd. e.g:

someFunc(<- db.get(id))

whereas

someFunc(defer db.get(id))

is still pretty readable.

And also, "<-" would cause havok with implied calls (the ones without parens). The following code:

foo <- someFunc()

would actually parse (naively) as:

foo(<-someFunc)

but foo is likely to be undefined, and not at all callable. I think the easiest way to fix this would be to restrict "<-" to not be an expression (make it only valid as a statement), but I'm pretty sure that's not what we want at all :/

or you could require the assignment still occur:

foo: <- someFunc

but that's super easy to forget, and doesn't look like haskell any more ;)

DanielVF

-1 for arrow.

Making a simple, often used feature (like defining a function) extremely short is wonderful. It's why I use coffeescript. But making every advanced language feature into punctation is bad.

Brian Slesinsky

I've been watching on the sidelines (and don't even use CoffeeScript yet) so take this with a grain of salt, but based on using Python generators, I'd like to see #1 with some special syntax:

get_document: (id, *) ->
  document: defer db.get(id)
  return document

The * stands for the compiler-generated callback parameter that will be called instead of returning. (Justification: it's concise, obvious that there's an extra parameter, obvious that it's a special parameter, and you don't have to scan the function body for 'defer'. The naive user will likely realize that there's something funny going on and read the CoffeeScript manual before calling this function.)

For consistency, it seems like we should do the same thing for the special function calls that defer makes:

get_document: (id, *) ->
  document: defer db.get(id, *)
  return document

I also think 'defer' is a somewhat unfortunate name since it doesn't sound like a return statement. (It also does something entirely different in Go.) If we can't have 'yield', maybe 'pause' would get the point across?

get_document: (id, *) ->
  document: pause db.get(id, *)
  return document

'pause' implies both that the function will stop running and resume later, and I think that's the most important thing to get across about an async function call. (It also becomes obvious why pausing within a loop might not be good for performance. Since the generated code isn't that good yet, it might be better to disallow that in the initial implementation.)

Passing along an error callback then becomes straightforward if we allow the '*' to appear at any position:

get_document: (id, *, error) ->
  document: pause db.get(id, *, error)
  return document
Brian Slesinsky

Also, on the client side, perhaps 'pause' with no argument could expand to:

 pause window.setTimeout(*, 0)
Greg Fodor
gfodor commented July 21, 2010

I'm sadly going to have to come out against the incorporation of defer into the language. Not that I'd not want to use it, but it would legitimately decrease the main argument that I'm using to push CS in my workplace: it's javascript with some great sugar. The semantics are 1-1 and the code generation is intuitive. It's has an hour long learning curve.

Up to this point the new syntax has been both intuitive and welcome. However, if we introduce this new defer mechanism, the learning curve for any body of code using it jumps up by an order of magnitude compared to what it is now, since it's a programming construct that would be hard to digest in any language. For this reason alone I feel like it will stand out as a feature that doesn't belong with the others, and would be a inflection point in the overall barrier to entry of CS for Javascript programmers.

Tim Cuthbertson

Hi all,

So I've spent a while on this, and obviously I'd love to see it merged in to coffeescript core. But it's looking like that won't happen, at least not by me. There are a few reasons:

  • differences in approach: jaskenas doesn't want it on core until it's finished, whereas maintaining a diff becomes much harder the more optimisations I make to defer (and the code around it)
  • difficulty / scope. defer is a pretty significant departure from the rest of CoffeeScript, which mostly deals with localised transformation of x into y, rather than function-wide transformations of control flow. The AST is unfortunately somewhat tangled up with the compilation output; the content of a node can change depending on how it is compiled.
  • forgiveness of the language. Features like implicit return and trying hard to enable more things to be statement takes its toll on the compiler, and make code transformations like defer have to deal with many more corner-cases.

So it really seems that perhaps defer isn't as good a fit for coffeescript as I had hoped. I'd love for someone to take over and prove me wrong, but I've reached the point of thinking it's not worth trying to squeeze defer into a language that has such different goals.

Although I no longer have a vested interest in seeing defer adopted (since I have no plans to write anything big in javascript any more), I'll certainly be taking a closer look at oni and stratified javascript - perhaps in the future stratified JS might be an optional compile target for coffeescript?

Anyways, if anyone wants to take over defer, please do get in touch and I'll do what I can to help out. But I don't expect I'll be doing any more work on it.

Nathan

That is very sad. A language that does not add new constructs to help with async programming is 'missing the point' of what a compile-to-javascript language should address imo.

When I saw coffeescript I thought, "That's kind of neat, but doesn't address the problem that I wish it did." When I saw defer, I was excited.

Greg Fodor
gfodor commented July 22, 2010

To me, the way something like this could come into fruition would be a CoffeeScript fork that takes the project in a different direction with different goals. There's been more than a few things Jeremy has avoided doing with good reason (for example, anything involving compilation into eval), but that's not to say there isn't room for a Javascript-compiled language that breaks these rules to put more power in the hands of the compiler. Defer would fit nicely into such a project, as would other enhancements that introduce fundamentally new concepts on top of Javascript. These are large tradeoffs of course, but I see it as a worthy project once there are more than a few of these types of features that would greatly increase programmer productivity.

Jeremy Ashkenas
Owner

It's not sad at all -- defer was a valiant and long-running effort to prove that code with sync semantics can be written in a sync style, and transformed transparently into an async implementation. If it didn't make it in the end, I believe it's because you can't accomplish the transformation and preserve semantics, while at the same time generating something close to the async code you would have written in the first place.

Case in point, Stratified JS: A virtuoso performance of JavaScript compilation, but look at what it compiles into. Taking the Stratified JS version of our getDocument function above...

var getDocument = function(){ 
  waitfor(document) { 
    resume(db.get(id)); 
  } 
  return document; 
};

It compiles into this JavaScript:

var getDocument;
__oni_rt.exec(__oni_rt.Seq(0,__oni_rt.Seq(0,__oni_rt.Nblock(
function(arguments){
  getDocument=function (){
    return __oni_rt.exec(__oni_rt.Seq(1,__oni_rt.Suspend(
    function(arguments, resume){
      return __oni_rt.exec(__oni_rt.Seq(0,__oni_rt.Fcall(0,__oni_rt.Nblock(
      function(arguments){
        return resume;
      }),__oni_rt.Nblock(function(arguments){
        return db.get(id)
      })
    )),arguments,this)}, 
    function() {
      document=arguments[0];
    }),__oni_rt.Fcall(0,__oni_rt.Return,__oni_rt.Nblock(
    function(arguments){
      return document;
    })
  )),arguments, this)};
}))), this.arguments, this);

I don't think we want to take CoffeeScript down that path. Open the Pandora's box of injecting special functions into the runtime, and ... suddenly you have to worry about being orders of magnitude slower than normal JS.

So, pour one on the ground for defer. This series of tickets will continue to be a reference point.

Sveinung Røsaker

To bad, I would have really loved if some kind of language feature like defer would help me solve/save daily problems/loc.

Henry Greville

An exact analogue of defer is slated for inclusion in *C#*5, as the await keyword.

I'm wondering if try/catch/finally could be used to achieve the readble callback-less sync flow attempted here - as in

try
    something_first()
catch error
    console.log error
finally
    do_something()

I'm assuming a bunch of stuff here but
Q1. Doesn't "finally" always enforce completion of the "try" block or are there any exceptions? (pardon the pun)
Q2. Any pros/cons or gotchas with this approach if it should indeed work?

any ideas?

Tim Cuthbertson

Nope, finally does not apply to asynchronous events, which is why asynchronous exception handling is so painfully difficult. This section of my post explains async programming as it relates to control flow more thoroughly: http://gfxmonk.net/2010/07/04/defer-taming-asynchronous-javascript-with-coffeescript.html#callbacks

Amir Shimoni

I find myself wishing CS had defer everyday... it's could be such a valuable paradigm, and simplify the way async code is written. Given that, I have a comment and question...

Jeremy: I'm not sure what the issue is with the size and/or speed of the generated code.
1) For speed, I don't think it's much of an issue, since (at least on client side), the places you would user defer would usually be where either you want to wait for IO or user interaction. Both of those are orders of magnitude slower than the speed of the generated code.
2) For size, you end up writing almost as much (albeit cleaner) code if you want to handle many of the edge cases. Plus... if someone is trying to save every last bit, then they can simply decide not to use defer.

Given that, I do understand the need to keep CS's learning curve low. Being able to learn it in one hour is one of it's (many) great strengths.

Unfortunately, I haven't had a chance to look at much of CS's internals, but was wondering if there are enough hooks to implement defer as a compiler plugin. And... whether the compiler architecture is stable enough (meaning it's not gonna change much) to make it worthwhile to create such a plugin now.

Yang Zhang
yang commented July 21, 2011

Is there a .js that I can include in my pages to just use the latest coffeescript-fork-with-defer (whichever fork that might be)?

Sveinung Røsaker

@yang This was not included after all, after long discussions. Coffee-script is staying close to js core features...

Yang Zhang
yang commented July 21, 2011

Right, that's why I'm looking past coffeescript and asking about forks.

weepy
weepy commented July 21, 2011
Tim Cuthbertson

@Yang: my fork is the most up to date that I know of for defer specifically (https://github.com/gfxmonk/coffee-script/tree/deferred), but it hasn't been touched in over a year (and I have no plans to do so), so Kaffeine (or tame, or stratified JS) is probably a better bet.

Amir Shimoni

You can also use coffeescript and then tamejs: https://github.com/maxtaco/tamejs
although it won't be pretty... you'll have to do something like:
`twait {` ; setTimeout defer(), 100; `}`

Amir Shimoni

The main drawback (IMHO) with using tamejs, as was one of the main reasons it didn't get into coffeescript, is debugging ability. But putting that aside, it works really well.

I created a gist with a very basic tamejs example written in coffeescript, compiled to tamejs:
https://gist.github.com/1098605

It does get pretty mangled. 3 lines of coffee script -> 7 lines in "tamejs" javascript -> 44 lines in "expanded" javascript

Devon Govett devongovett referenced this issue December 17, 2011
Closed

Tame #1942

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.