Defer, for transforming async callbacks #241

Closed
weepy opened this Issue Mar 8, 2010 · 65 comments

Comments

Projects
None yet
9 participants

weepy commented Mar 8, 2010

Was just reading about the yield operator from NarrativeJS. It has some interesting ideas about how to make async JS more readable.

"Narrative JavaScript is a small extension to the JavaScript language that enables blocking capabilities for asynchronous event callbacks. This makes asynchronous code refreshingly readable and comprehensible."

More here: http://www.neilmix.com/narrativejs/doc/index.html

Could be something for CS, though it might be a bit wacky?

Owner

jashkenas commented Mar 9, 2010

It's a really neat idea. I took a brief scroll through the source, but didn't quite catch how he's doing the transformation. Doing the easy case would be easy (with a hypothetical sync keyword):

json: sync jQuery.get('/api')
print json

Into:

jQuery.get '/api', (json) ->
  print json

But doing it in the middle of something like this:

for element in list
  json: sync jQuery.get('/$element')
  print json

I don't understand how you could transform the loop into a callback that knows how to resume the loop at the proper iteration. Variables may have changed out from under you while the async request was running, as well.

Any ideas for how this could be compiled, in general? If it was really so easy to implement, then I would imagine that a lot more languages would offer it.

weepy commented Mar 9, 2010

well the JS it makes sure ain't purty :

function sleep(millis) {
        var notifier = new EventNotifier();
        setTimeout(notifier, millis);
        notifier.wait->();
    }

===>>>

function sleep(millis){

  var njf1=njen(this,arguments,"millis");
  nj:while(1) {
    njf1.noex=1;
    switch(njf1.cp) { 
      case 0:
      njf1._notifier=new EventNotifier();
    setTimeout(njf1._notifier,njf1._millis);
      njf1.pc(1, njf1._notifier,"wait",[]);
    case 1:
      with(njf1)
         if((rv1=f.apply(c,a))==NJSUS){ 
            return fh;
         }
       break nj;
    } 
   }
}  

hen-zone commented Mar 9, 2010

To quote from the developer, for perspective:

My thinking about such things has since evolved, and I no longer think narrativejs is the best way to approach the problems I was trying to solve. As a result, I’m no longer actively developing the project.

Owner

jashkenas commented Mar 9, 2010

Good -- technically infeasible. Closing the ticket.

Owner

jashkenas commented Mar 16, 2010

Reopening the ticket because gfxmonk is taking a shot at something very similar, with promising early results. Take a look at his branch here:

http://github.com/gfxmonk/coffee-script/tree/deferred

Contributor

drnic commented Mar 17, 2010

Hiding the nested asynchronous callbacks with CS syntax is a delightful goal.

weepy commented Mar 17, 2010

I'm intrigued to know how it might be achieved. I cannot think of a neat mechanism ...

Contributor

timbertson commented Mar 18, 2010

I feel I should pipe up here ;)

So what I'm planning is essentially an automatic transformation of procedural-style code into continuation-passing style. The code that comes out should look (more or less) like a human would write javascript to deal with an asynchronous API, but the input cofeescript will me much easier to read.

For example, if you have a library function "func" that takes arguments (a,b,c, callback)
you could write:

myFunc: ->
    x: defer func(a,b,c)
    [more code here]
    return x;

which should generate something like

myFunc = function(__callback) {
    func(a, b, c, function(x) {
        [more code here]
        __callback(x);
        return null;
    }
}

Dealing with branches and loops will be tricky, but I think it can be solved with a judicious application of anonymous functions ;)

weepy commented Mar 18, 2010

From emails with gfxmonk:
While I think it looks good with the defer before the function call, I think it might be more flexible to use the keyword to replace the function call in question, e.g.

$.get "/posts", defer
# do stuff with posts

setTimeout defer, 10
# do something in 10ms

The other syntax question that needs considering is how to handle function arguments. Perhaps something like:

$.get "/posts", (a,b) --->
# use the variable a and b here after get call is complete

Here I'm using a long arrow to indicate a deferred call.

Contributor

timbertson commented Mar 19, 2010

I think doing some detection of a splat / destructured array assignment would be good here. i.e to have multiple args, you would just do:

a,b = defer get("/posts")

But I have no answer for what to do if the callback argument is not the last. I think it's extremely rare, and writing a wrapper is simple enough for each instance you will encounter.

There's also the issue of error-callbacks. I have no current plan to deal with them, but it's probably worth considering.

Contributor

timbertson commented Mar 19, 2010

I am pretty sure I've seen most of the possibilities, but don't actually remember running jwacs. I think I tried and it didn't work straight off, but I should probably give it another try. I think I decided that strands was a better option ( http://www.xucia.com/strands-doc/compile.html ). An array of bugs soon killed that idea...

jwacs has a lot of good features (like the error handling), but I think that makes it somewhat incompatible with everything-that-doesn't-use-jwacs. I'm hoping to maintain a reasonable level of compatibility with library code you might actually write in javascript ;)

weepy commented Mar 19, 2010

This is an interesting read: http://hyperstruct.net/2008/5/17/synchronous-invocation-in-javascript-part-1-problem-and-basic-solution

Is dependent on the yield operator however.

Contributor

timbertson commented Mar 19, 2010

Yep, async.js ( http://eligrey.com/blog/post/pausing-javascript-with-async-js ) makes a proper library out of this approach. I used it for a little while, but:

  • as noted, it depends on yield. As far as I know, nobody other than mozilla plans to implement yield
  • it's still very complicated to actually try and write code with. You have to wrap your head around how it works, and then you have to remember all these nuances when writing the code. If a chain breaks, debugging to find out which one is problematic.

I made some longer-winded musings about trying to make it simpler to use here: http://gfxmonk.net/2010/02/12/making-sense-of-async-js.html
(I don't think I succeeded, but it may be an interesting read nonetheless)

Owner

jashkenas commented Mar 20, 2010

gfxmonk: A question. Because you'll never know where a defer might occur, every single function call you make is going to have to pass its continuation as a closure, right? There would be no such thing as calling two functions sequentially...

If that's the case, then I think that the code this generates will be a few stops past our tolerance for nastiness...

Contributor

noonat commented Mar 20, 2010

Python deals with that by actually treating functions using the yield keyword as a different type at compile time (they essentially end up wrapping it and using syntactic sugar to hide the fact that it's wrapped).

Owner

jashkenas commented Mar 20, 2010

noonat: But it can occur through multiple levels of function calls. (I think). For example.

async: ->
  defer 
  ...

inner: ->
  async()
  ...

outer: ->
  inner()
  ...

So, really, all of the code following the inner() call at the top level needs to be passed into inner() as a callback, even though inner itself doesn't contain a defer. In the presence of eval() and with(), we can't guarantee that a function with a particular name will be guaranteed to contain a defer or not.

So, I think you end up having to pass continuations into every function call. If you have more insight into how Python has magically sidestepped it, I'd love to hear it.

Contributor

timbertson commented Mar 20, 2010

Jashkenas: You should always know when a defer is going to happen, because it's a keyword. I wasn't planning to deduce when a defer-using function is being called and automatically make that call a deferred one - rather, if you call a function that internally uses defer, its function signature will require a callback to be passed - and the best way to do that is to use the "defer" keyword where you call it.

So nothing here is transitive or dynamic, it's all resolveable at compile-time. If you call a defer-using function without the defer keyword, things won't work (hopefully we can make a nice error message there, because the callback argument will be undefined).

It's a convenience for not having to write callbacks - it sadly can't prevent you from needing to know where they are necessary (but hopefully error messages can help).

to rewrite your example as it would need to be (in order to work):

async: ->
  defer 
  ...

inner: ->
  defer async()
  ...

outer: ->
  defer inner()
  ...

that is, every call that uses a defer somewhere in its stack will have to be a "defer" call. This is still a fairly small inconvenience compared to the current state of javascript (writing callbacks explicitly, all the way down the call stack). Especially since it can hopefully deal with the awkwardness of branching and looping for you.

noonat: changing the function at compile time is one part, but you still have to do something at runtime to treat a deferred function differently. That's outside the scope for coffeescript (and my plans ;))

weepy commented Mar 20, 2010

how about sync as an alternative keyword ?

Contributor

timbertson commented Mar 20, 2010

"sync" seems a little too common of a word to reserve. I may be biased, as I'm currently writing a RSS sync application ;)

also, i'm not sure that sync fits. What it is is an asynchronous call, that will automatically resume when it is done. It is not actually synchronous, this is simply shorthand for saying "resume this function when this asynchronous call is complete". I personally think "defer" captures that nicely, as it indicates background (async) work that the current function will depend on. But that could just be me.

weepy commented Mar 20, 2010

yes i think you're right.

Contributor

noonat commented Mar 20, 2010

@jashkenas: oh, I misunderstood. Yeah, I don't think you could map calling scope to the continuation automatically... Continuation style development in Python can't even do this, and it has language support for yield.

Twisted Python's deferreds (e.g. NodeJS promises) have a helper for continuation style stuff. It allows you to yield deferreds from within the function, and resume where you left off. So, this:

def foo(url):
    def callback(result):
        print result
    deferred = bar()
    deferred.addCallback(callback)
    return deferred

Becomes this:

@defer.inlineCallbacks
def foo(url):
    result = yield bar()
    print result

This ends up creating a wrapper function which basically keeps continuing the foo() method until it stops yielding deferreds, then triggers the callback passed to the wrapper function (or the errback, if an exception occurs in the continuations). The wrapper adds itself as a callback/errback to the yielded deferred, and then passes the results off to the continuation when it gets a result.

weepy commented Mar 21, 2010

gfxmonk: can you convert your example into what the JS might look like - so we can see how you imagine it to work ? I mean this

async: ->
  defer 
  ...

inner: ->
  defer async()
  ...

outer: ->
  defer inner()
  ...
Contributor

timbertson commented Mar 21, 2010

sure, it would look something like:

function async(_cb) {
_cb();
return null;
}

function inner(_cb) {
async(_cb);
}

function outer(_cb) {
inner(_cb);
}

although the code might be a bit more verbose, and the callback variables would end up as _a, _b, etc..

Here's an actual example from my test cases, for a slightly more complex depiction of what's possible:

input code:

two_defers: ->
   2 * defer return_arg(5) + defer return_arg(2)

(return_arg is a test helper that just returns the given argument via a callback)

compiled code:

 two_defers = function two_defers(_a) {
   return_arg(5, function(_b) {
     return_arg(2, function(_c) {
       _a(2 * _b + _c)
     })
   })
 };

it ain't pretty, but that's kind of the point ;)

Owner

jashkenas commented Mar 21, 2010

gfxmonk: Thanks for the explanation. It's a very exciting approach to taming Node's asynchrony.

Here's a modified snippet from the current cake test task:

fs.readdir 'test', (err, files) ->
  files.forEach (file) ->
    source: path.join 'test', file
    fs.readFile source, (err, code) ->
      CoffeeScript.run code, {source: source}

When your branch lands on master, I'll be able to write it like this, if I understand it correctly:

files: defer fs.readdir 'test'
files.forEach (file) ->
  source: path.join 'test', file
  code: defer fs.readFile source
  CoffeeScript.run code, {source: source}

Although that ignores the error-as-first-argument bit of the Node API. Have you thought about how you're going to handle callback arguments, and expose them to the function using the defer?

Am I right in thinking that, as an unintended consequence of the defer keyword, code like the following would also work?

list: "one two three four".split " "
item: defer list.forEach()
print item
#one, two, three, four

...since the callback/continuation would be invoked multiple times by forEach. I realize that such code would be unreadably confusing, but would it work in principle?

matehat commented Mar 21, 2010

gfxmonk: I don't know if your code currently handles it, but to be consistent with current coffeescript, I think a proper handling of callbacks with multiple arguments could be, referring to jashkenas example :

[err, files]: defer fs.readdir 'test'
files.forEach (file) ->
  source: path.join 'test', file
  [err, code]: defer fs.readFile source
  CoffeeScript.run code, {source: source}

matehat commented Mar 21, 2010

Also, if it's going to look like we're continuing execution in the same context, each passed callbacks would need to be bound to that context, so we don't get weird errors (would it make sense for this to change its value in the same virtual block?)

tav commented Mar 21, 2010

Hey guys,

I just stumbled across this issue while looking to add similar functionality to coffeescript. My interest is in adding some kind of generic support to make "functional reactivity" much much easier, e.g.

Not sure if you guys have seen them before, but definitely worth taking a look! Do let me know if you think it might be worth supporting such things natively in coffeescript...

It'd also be nice if the mechanism added to support defer could also be somehow leveraged to support arbitrary DSLs like in this proposal:

Thoughts?

weepy commented Mar 21, 2010

Hey tav - can you explain what you mean by "functional reactivity" ? It's not clear from the links you gave (for me at least)

Owner

jashkenas commented Mar 21, 2010

tav a couple thoughts on blocks:

We already have them. Unlike Python lambdas, they are not limited to single-line expressions, and unlike Ruby lambdas, they share identical syntax to any other function definition. For example, so translate a couple of examples from the page you linked...

Pseudo-Python:

using webapp.runner do (config):
  config.time_zone = 'UTC'
  config.log_level = 'debug'

Into:

webapp.runner (config) ->
  config.time_zone: 'UTC'
  config.log_level: 'debug'

Ruby:

employees.select {|e| e.salary > developer.salary}

Into:

employees.select (e) -> e.salary > developer.salary

matehat commented Mar 21, 2010

tav: The projects you mentioned aim to provide useful tools for managing events in application development, I can't really see how such support could be implemented natively into an isomorphism of the javascript language.

Currently, the code coffeescript produces is not bound to any external libraries, it only rewrite stuff to standalone javascript. It could however be implemented as a set reusable modules. Thanks to its developer, coffeescript already provides most of the syntax needed to make such project look like it's native.

matehat commented Mar 21, 2010

About my earlier thoughts on handling multiple arguments for generated callbacks, I just noticed my suggestion breaks the inline construct :

two_defers: ->
  2 * defer return_arg(5) + defer return_arg(2)

We could also distinguish between an assignment syntax and an expression, where the former would allow multiple arguments for the generated callback and the latter would allow only one.

Contributor

timbertson commented Mar 22, 2010

Jashkenas:

Although that ignores the error-as-first-argument bit of the Node API. Have you thought about how you're going to handle callback arguments, and expose them to the function using the defer?

Not explicitly, no. I think splats should be sufficient though (read on ;))

sethaurus:
yes, that code would compile and run as you say. I don't think there's any way to be smarter and detect such cases (you and I know what forEach does, but the compiler can't)

matehat:
yes, I am planning to implement a special case when a splat is assigned the result of a deferred call. This would only happen when a deferred result is directly assigned to a splat, so it shouldn't break the case of using a deferred result inside an expression.

An alternative would be to return the arguments as an array if the callback is called with more than one argument, but if it's called with one argument then you just get the single argument back (instead of an array with one element). That way splats and expressions would work the same. The downside is confusion for functions that will sometimes call back with one argument and sometimes with multiple. I would not be surprised if some libraries do this, though I think they really shouldn't ;).

regarding this, that's a great point. We could capture a reference to this at the start of the function, and restore it at the start of each deferred callback. The only problem then is if you wanted to use the this that the callback gave you (I think this is a terrible design decision, but I know it happens in jQuery and probably elsewhere).

matehat commented Mar 22, 2010

gfxmonk:
I wouldn't worry about the use of this given by the caller because when that happens, we clearly know we are changing context. As in the jQuery example, when I want to use this given for a particular mouse event, I am aware I am running code on that context, not anymore on the context where I've bound a callback to the mouse event, so it wouldn't make sense to look like I am continuing execution. A normal anonymous function is more appropriate for these situations.

Also, about detecting what's the number of arguments passed on to the callback, I don't think it is possible to reliably determine behavior on top of that. I think it's more flexible and less error-prone to just transpose assignment structure to callback arguments structure :

[arg1, arg2]: defer asyncFunction()
=> asyncFunction (arg1, arg2) -> ...

arg1: defer asyncFunction()
=> asyncFunction (arg1) -> ...

[arg1, rest...]: defer asyncFunction()
=> asyncFunction (arg1, rest...) -> ...

2 * defer asyncFunction() + 2
=> asyncFunction (_a1) -> 2 *_a1 + 2 ...

leaving the choice to the developer as to how the arguments are accessed and which one is left unused

Contributor

timbertson commented Mar 22, 2010

After a brief thought, I'm thinking:

  1. capturing (and reassigning) this is less surprising than not doing so, so I think we should.
  2. forget about special behaviour for splat assignments, just return the array of arguments (when arguments.length != 1) or the first argument (when arguments.length == 1)

weepy commented Mar 22, 2010

Here's an piece on Ruby Fibers and trying to improve readability of async code: http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/

It's not directly applicable, but it's an interesting read.

Contributor

timbertson commented Mar 23, 2010

interesting indeed. That looks like a pretty cool library, I'm keen to investigate the magc behind it. It sounds (on the surface) a lot like async.js, but with much less awkwardness because it doesn't have to shoehorn it in to javascript.

Ruby's fibers themselves sound a lot like python's generators. Is that accurate, or have I missed an important difference?

matehat commented Mar 24, 2010

After some deep thoughts about the whole synchrony question, I've come to think there's way too many ambiguities and uneasy assumptions, at least the way it is currently presented. I think, in particular, that some key concepts are mistakenly assumed as being some aspects of the same thing. So I'm going for some brainstorming :

First, I don't think we should change the return statement for a call to an implicit callback. They are fundamentally different and do not make the code behave in the same way. This would lead to contradictory situations, that we'd only be able to overcome by applying unneeded assumptions. I think we should introduce a yield statement that would call the implicit callback. This allows things that we otherwise couldn't with the return statement, like calling the implicit callback multiple times in the same control flow, without leaving the function. With that, I think a keyword like callback_given? should be available so the function knows whether it should yield or not.

Implicit callback as assumed to be at the end, both for deferring and defining, implies comptatibility problems. A more flexible syntax could be based on a character, and I'm thinking ~. I think it's a good one, since the curliness reminds us of non-direct control flow. So some interesting use of it could be :

# defining function with implicit callback
asyncFunction: (x, y, z) ~>
  # some code
  yield

# defining alternate implicit callback position or giving it a name
readFirstFile: (dir, ~, error) ~>
  [err, files]: fs.readdir dir, ~
  return error(err) if err
  [err, out]:   fs.readFile files[o], ~
  return error(err) if err
  yield content

# deferring a function continuation
content: fs.readFirstFile dir, ~, (err) -> sys.puts err
# do something with the content

That way, deferring calls are not tightly bound to implicit callback. They are really seperate stuff and can be used in situations where they're not both needed :

cacheData: (data) -> # Notice, no implicit callback
  [code, message]: backend.cache data, ~
  log(message) if code is 'error'

map: (list) ~> # Implicit callback, but does not contain deferring calls!
  res: []
  list.forEach (o) -> 
    res.push(if callback_given? then yield o else o)
  list

So .. Ideas, anyone?
(yes, I initially posted this message in the wrong discussion)

weepy commented Mar 24, 2010

I like the , but not so sure about >. Leaving these syntax ideas aside - I'm interested to see how yield might work under the hood. It would be enormously useful if you could convert these examples to Javascript, so we can see exactly what might be going on ?

matehat commented Mar 24, 2010

Sure, here's the above example, together with the equivalent vanilla coffeescript and its compiled javascript code : http://gist.github.com/342570

Also, I was just reading the past ticket about the yield operator and wanted to point out a few things about my previous comment. My suggestion here is by no mean an attempt to introduce some magical generator-like process in coffeescript. It's only a synonym for call the implicit callback, so the task of handling such callback becomes clearer, because I don't think rewriting based on the return statement is a good idea, due to its very divergent usual behavior.

Contributor

timbertson commented Mar 25, 2010

Cool, it's good to have more people thinking about how this should work :)

I personally have seen very few examples of callback as anything but the last argument, so I am keen to avoid the "~" magic variable. "defer" is much more explanatory and since it comes at the start of a call, you're less likely to miss its use (it does change control flow significantly, so I think it should be obvious).

The "~>" operator is an interesting idea. One problem that I see is that an async function is very frequently (not in your examples, but I believe most of the time in actual usage) entirely determined by whether or not you use "defer".

I'm quite worried about making the use of this mechanism too easy to get wrong. For example, consider the following function:

def some_normal_function ->
    file = defer load_some_file_async("readme.txt")
    return file.read()

This seems like entirely reasonable code, but the return will actually have no effect (because you cannot usefully return from callback-based async code, you can only yield). Having the rule "always yield results instead of returning them after you use the "~" variable" is a pretty awkward thing to have to remember. And the penalties for returning instead of yielding are high - your program simply ceases execution, and you don't really know where things stopped working.

Having said that, your final example there with the yield inside the anonymous function (forEach's iteration) is a good one. I don't think it ever could work with the current return-replacing behaviour, because it would be ambiguous what you meant. ~> could be useful for that, since I'm assuming that yield would look for the closest implicit-callback function in scope, rather than just the function it appears in.

Although to be honest, i don't think a continuation should ever be used for anything other than a callback-based return. Using it for a looping construct is likely to be very confusing, as per sethaurus' example.

matehat commented Mar 25, 2010

I see what you mean, though I still think it could be useful to have a more general approach towards callback-based control flow (going back and forth between execution points, though not limited to async functions). We should make it so that it makes sense and is consistent, then build the async sugar on top of that. The whole point around ~> and yield is really about that control flow, and they're only meant to make it more readable. I don't think these operators are more confusing than those implied with ruby's iterator for instance. Since it is meant to be more general, I thought limiting the callback position in arguments to the last was an unnecessary restriction, though it's not the core issue, I agree.

That said, I think it is more confusing for people to see a return statement in a function, and not getting anything from it, just by calling it. A yield statement makes it obvious that it will not trivially return its operand. It will yield to something, and it needs that something to be passed as an argument. For example, a function might sequentially yield values to its callback; nothing prevents it from continuing execution after the first yield and yield another one. A return statement should never be able to do this: returning has always been a break point in a function.

Here, yielding to callback does not yet refer to asynchronous-oriented programming style, such as continuation. I think the latter can be implemented in coffeescript more easily if we have a readable callback semantic, such as yielding. I also agree that continuation-style should never be used for anything else than for asynchronous context or the like. For example, a function that successively fires its callback with different arguments, a continuation does not capture that iterative process and makes things confusing. A continuation should only be used in cases where it looks as if the execution was sequential.

I've tried to uncouple callback-based (read ~> and yield) and continuations (read ~ in function call arguments and code transformation that follows) because I strongly believe neither of them is exclusive and that led to situations hard to handle otherwise. One could be used without the other, and be really useful in real-world situations. That uncoupling stroke me when I thought about giving a shot at an implementation. Doing so really led to large simplifications. I'll push my work to a branch probably tonight when it gets stable enough.

Contributor

timbertson commented Mar 25, 2010

Also note that in your second example, you use:

return error(err) if err

this will return an error into whatever called your continuation. Since the async function thinks of the passed-in callback (your continuation) as a place to put return values, there is nothing meaningful it could do if your continuation were to actually return anything (every async function I've seen will completely ignore that error value).

What you would probably want to do in practice is something along the lines of yield(err, result). Since it's unlikely we can make exceptions via callbacks actually work, the next best thing is to use node's approach where the first return value is reserved for any errors.

If you were to do that, then you would have no use for both return and yield (in that second case).

matehat commented Mar 25, 2010

There, the return statement was rather to stop execution, while doing something with the error, than to return anything. This was kind of to illustrate that these two keywords might be used for different things in the same function.

I agree that, in that case, it would be better to yield the error, but I don't know about every situation and I'm not in favor of hard-coding what we think is better practice into language constructs.

Owner

jashkenas commented Mar 25, 2010

Just to pipe in about the error handling... I find Node's current error-handling convention completely unworkable. Errors are swallowed silently by default if you don't watch for them, and the code you need just to get a sane level of logging is absurd.

Since the problem, fundamentally, is the difficulty of reconnecting an asynchronous error with the call stack(s) where it originated ... I think it would be worth considering ditching standard synchronous exceptions, and just defining top-level handlers for all of the exception types you want to cover. Instead of passing them to the callback, you'd attach the callback to the error before triggering the top-level handler. The handler can determine if it should be called or not. The default handler would error out with the message, the same as if an exception bubbles up to the top level.

Just my two cents -- this isn't in scope for regular CoffeeScript, but it might be for your deferred branch, if you're trying to solve the async error problem.

Potential code might look vaguely like:

write_to_disk: ->
  directory.read_files (files) ->
    files.each (file) ->
      file.write contents, ->
        throw new FileLocked(file)

exception FileLocked (error) ->
  puts "failed to write to ${ error.file.name }"
Contributor

timbertson commented Mar 25, 2010

jashkenas: yes, I'd love to fix this as well. I think it's fundamentally difficult though. Consider the simple case of:

window.setTimeout((-> throw new Error()), 0)

your exception will bubble up through setTimeout, but I'm pretty sure there's absolutely nothing we can do to catch it. There is a chance that we could decorate all our continuations with error-handling code (assuming the [err, result] convention) but I'm not sure how we could connect the stacks. The global handler may be possible, but I'm not too sure what use it would be in terms of actually recovering from exceptions (and if you don't recover, is there any point catching it?).

matehat: cool, I'm keen to see your branch :)

a small note:
(disclaimer: I'm awful at keeping notes small)

For example, a function might sequentially yield values to its callback; nothing prevents it from continuing execution after the first yield and yield another one.

I'm planning to prevent this, actually ;)
"return x" will be transformed into the equivalent of "__callback(x); return null;" on my current branch. This means it acts just like a return, only it is delivered via a callback. You can never do anything else after you "return", whether it be a deferred function or a regular one.

we already have a solution for multiple callbacks (a-la ruby's yield), and that is anonymous functions - and coffeescript makes them just as easy to write as a ruby block.

I'm not sure you really gain much by allowing the following:

map: (list, cb) -> cb(x) for x in list

to be written as

map: (list) ~> yield x for x in list

It's a little nicer perhaps, but also more magic (and just another way to do something we already can). The magic I'm proposing is for a very specific use case (continuation-based returning), because I think the awkwardness in doing it any other way warrants such magic.

Owner

jashkenas commented Mar 25, 2010

gfxmonk: that's the idea. Because there is no way to correctly re-connect the stacks without resorting to twisty error-argument passing through every callback, instead you're responsible for attaching the information needed to recover from the error to the error object itself. Your handler should know what to do with it.

A FileNotWritable error might have file, mode, and contents, and callback properties, for example, so the handler could retry, prompt for sudo, or whatever you'd like to do.

Still, not saying that the idea is any good, but just throwing it out there. I don't think it would work because it would have to be a convention at the Node.js level, and it simply isn't.

Contributor

timbertson commented Mar 25, 2010

ahh, yes fair enough - I was assuming you're getting errors from code you didn't write, which makes it difficult to attach the necessary metadata. But yeah I think it's an interesting idea, and it'd be good to get some more if others have thoughts :)

weepy commented Mar 25, 2010

Some thoughts

  1. do you need both ~> and yield? Perhaps any function with a yield within it's body is defined as a async function?
  2. Perhaps it's also useful to be able to return from the function without firing the callback via return?
  3. Just to be sure we're all talking about the same thing here - node has recently changed it's API wrt to callbacks. Gone are the promises and addError and friends... all the functions simply have a single callback, e.g. (err, result, ...) ->

matehat commented Mar 25, 2010

If we remove ~>, there's no way to tell which function has an implicit callback, in the case of nested functions.

As for return, in my implementation, you can still do return yield obj if you want to return at that moment or yield obj if you don't.

Contributor

timbertson commented Mar 25, 2010

matehat: for all the cases where you might want to continue execution after a yield, are these also the exact same cases where you'd give it an anonymous function explicitly, rather than using defer or ~?

Because if you pass someone a continuation, and they yield to it more than once, I'm pretty sure all manner of confusion will break loose.

Contributor

timbertson commented Mar 25, 2010

re weepy's point 1: we don't need anything like ~> for my proposed solution, but that's simply because we wouldn't deal with nested callbacks (you'd have to defer the call to the function using them, and then you become an async function yourself).

point 2:, if you were to return without calling the callback, then you're effectively stopping execution silently - from the perspective of your caller, the function never returns - not even with an error! Imagine if this were normal procedural code - you would either return (implicitly or explicitly), or you would raise an error. You can't simply stop the program from continuing, and even if you could then I doubt you would want to ;)

Owner

jashkenas commented Mar 25, 2010

Check out how Go is planning to add exceptions-via-defers ... seems relevant:

http://groups.google.com/group/golang-nuts/browse_thread/thread/1ce5cd050bb973e4#

weepy commented Mar 25, 2010

Here's a simpler example from http://www.go-program.com/go-defer :

# The defer statement executes a function (or method) when the 
# enclosing function returns. The arguments are evaluated at the
# point of the defer; the function call happens upon return.

func data(name string) string {
  f := os.Open(name, os.O_RDONLY, 0);
  defer f.Close();
  contents := io.ReadAll(f);
  return contents;
}

# Useful for closing fds, unlocking mutexes, etc. 

weepy commented Mar 25, 2010

forgot to say : So as you can see - it's only really relevant for clearing up at the end of the function (i.e. not synchronicity)

Contributor

noonat commented Mar 25, 2010

Re ~> and ~: I'm against these, at least in their current forms. ~> looks far too much like ->, and isn't nearly visible enough for the weird side effects the presence/lack of it can cause. weepy suggested detecting function bodies using a yield -- this is what Python does.

In general, I think using character magic is a dangerous path. Succinct operators like these can be really nice, when you know the language. But if you go too far, as I think this does, the language can be incredibly hard to read for newcomers (Perl and Go are both examples of this... I think Ruby strikes a good balance).

weepy commented Mar 25, 2010

I'm getting a bit confused about all these proposed yields and defers etc etc.
Could you fix up the following code !

sleep: (x, fn) -> yield setTimeout(fn, x)

getAfterWait: (url) -> 
  sleep(1000)
  posts: defer $.get "/posts"
  yield posts

[e, data] = defer getAfter1Sec "/posts"
($ @).html data

cheers

matehat commented Mar 25, 2010

In my proposed paradigm, that's

sleep: (x) ~> setTimeout x, -> yield 
getAfterWait: (url) ~>
    sleep 1000, ~
    posts: $.get url, ~
    yield posts

[e, data] = getAfterWait "/posts", ~
($ @).html data

It currently compiles to 42 LOC :)

Contributor

tim-smart commented Mar 25, 2010

Ahh... all the ~ is confusing the heck out of me :D It seems a little.... cryptic?

weepy commented Mar 25, 2010

How about swapping for defer and meaning that any function that defines yield is a > type function. Hence this would be :

sleep: (x) -> setTimeout x, -> yield 
getAfterWait: (url) ->
    sleep 1000, defer
    posts: $.get url, defer
    yield posts

[e, data] = getAfterWait "/posts", defer
($ @).html data

Could that work ?

Contributor

timbertson commented Mar 25, 2010

I was thinking about it last night, and I'm happy to adopt ~ for the unlikely case where the continuation argument is not the last. That is:

defer func(a, b, c)

would become syntactic sugar for

defer func(a,b,c, ~)

but if you need to, you can specify the tilda position, like:

defer window.setTimeout(~, 100)

I still think we should still have the "defer" keyword, as the tilda is not obvious. Having the tilda only appear when the callback argument is not last will keep the flexibility, but avoid the awkwardness of many trailing tildas.

Looks like this thread is getting too long and winding for most mortals. matehat: perhaps we should write up a wiki page (or two) with the current state of each of our proposals / examples? Is that okay with you jashkenas?

Owner

jashkenas commented Mar 25, 2010

gfxmonk: hey, this ticket belongs to you and matehat now. If you'd like a wiki page, go right ahead. Otherwise you could close this one and start another, with your current proposals/branches at the top. Also, feel free to pop into #coffeescript if you'd like to bounce ideas off folks.

Contributor

timbertson commented Mar 25, 2010

weepy: my version would look like:

sleep: (x, fn) -> setTimeout(fn, x)

getAfterWait: (url) -> 
  defer sleep(1000)
  posts: defer $.get "/posts"
  return posts

[e, data] = defer getAfterWait "/posts"
($ @).html data

matehat commented Mar 26, 2010

Too many comments, as gfxmonk pointed out. I started a new discussion, http://github.com/jashkenas/coffee-script/issues/287, so we can close this one.

devongovett referenced this issue Dec 18, 2011

Closed

Tame #1942

This issue was closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment