New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Defer, for transforming async callbacks #241
Comments
It's a really neat idea. I took a brief scroll through the source, but didn't quite catch how he's doing the transformation. Doing the easy case would be easy (with a hypothetical
Into:
But doing it in the middle of something like this:
I don't understand how you could transform the loop into a callback that knows how to resume the loop at the proper iteration. Variables may have changed out from under you while the async request was running, as well. Any ideas for how this could be compiled, in general? If it was really so easy to implement, then I would imagine that a lot more languages would offer it. |
well the JS it makes sure ain't purty : function sleep(millis) { var notifier = new EventNotifier(); setTimeout(notifier, millis); notifier.wait->(); } ===>>> function sleep(millis){ var njf1=njen(this,arguments,"millis"); nj:while(1) { njf1.noex=1; switch(njf1.cp) { case 0: njf1._notifier=new EventNotifier(); setTimeout(njf1._notifier,njf1._millis); njf1.pc(1, njf1._notifier,"wait",[]); case 1: with(njf1) if((rv1=f.apply(c,a))==NJSUS){ return fh; } break nj; } } } |
To quote from the developer, for perspective:
|
Good -- technically infeasible. Closing the ticket. |
Reopening the ticket because gfxmonk is taking a shot at something very similar, with promising early results. Take a look at his branch here: |
Hiding the nested asynchronous callbacks with CS syntax is a delightful goal. |
I'm intrigued to know how it might be achieved. I cannot think of a neat mechanism ... |
I feel I should pipe up here ;) So what I'm planning is essentially an automatic transformation of procedural-style code into continuation-passing style. The code that comes out should look (more or less) like a human would write javascript to deal with an asynchronous API, but the input cofeescript will me much easier to read. For example, if you have a library function "func" that takes arguments (a,b,c, callback)
which should generate something like
Dealing with branches and loops will be tricky, but I think it can be solved with a judicious application of anonymous functions ;) |
From emails with gfxmonk: $.get "/posts", defer # do stuff with posts setTimeout defer, 10 # do something in 10ms The other syntax question that needs considering is how to handle function arguments. Perhaps something like: $.get "/posts", (a,b) ---> # use the variable a and b here after get call is complete Here I'm using a long arrow to indicate a deferred call. |
I think doing some detection of a splat / destructured array assignment would be good here. i.e to have multiple args, you would just do: a,b = defer get("/posts") But I have no answer for what to do if the callback argument is not the last. I think it's extremely rare, and writing a wrapper is simple enough for each instance you will encounter. There's also the issue of error-callbacks. I have no current plan to deal with them, but it's probably worth considering. |
I am pretty sure I've seen most of the possibilities, but don't actually remember running jwacs. I think I tried and it didn't work straight off, but I should probably give it another try. I think I decided that strands was a better option ( http://www.xucia.com/strands-doc/compile.html ). An array of bugs soon killed that idea... jwacs has a lot of good features (like the error handling), but I think that makes it somewhat incompatible with everything-that-doesn't-use-jwacs. I'm hoping to maintain a reasonable level of compatibility with library code you might actually write in javascript ;) |
This is an interesting read: http://hyperstruct.net/2008/5/17/synchronous-invocation-in-javascript-part-1-problem-and-basic-solution Is dependent on the yield operator however. |
Yep, async.js ( http://eligrey.com/blog/post/pausing-javascript-with-async-js ) makes a proper library out of this approach. I used it for a little while, but:
I made some longer-winded musings about trying to make it simpler to use here: http://gfxmonk.net/2010/02/12/making-sense-of-async-js.html |
gfxmonk: A question. Because you'll never know where a If that's the case, then I think that the code this generates will be a few stops past our tolerance for nastiness... |
Python deals with that by actually treating functions using the yield keyword as a different type at compile time (they essentially end up wrapping it and using syntactic sugar to hide the fact that it's wrapped). |
noonat: But it can occur through multiple levels of function calls. (I think). For example.
So, really, all of the code following the So, I think you end up having to pass continuations into every function call. If you have more insight into how Python has magically sidestepped it, I'd love to hear it. |
Jashkenas: You should always know when a defer is going to happen, because it's a keyword. I wasn't planning to deduce when a defer-using function is being called and automatically make that call a deferred one - rather, if you call a function that internally uses defer, its function signature will require a callback to be passed - and the best way to do that is to use the "defer" keyword where you call it. So nothing here is transitive or dynamic, it's all resolveable at compile-time. If you call a defer-using function without the defer keyword, things won't work (hopefully we can make a nice error message there, because the callback argument will be undefined). It's a convenience for not having to write callbacks - it sadly can't prevent you from needing to know where they are necessary (but hopefully error messages can help). to rewrite your example as it would need to be (in order to work):
that is, every call that uses a defer somewhere in its stack will have to be a "defer" call. This is still a fairly small inconvenience compared to the current state of javascript (writing callbacks explicitly, all the way down the call stack). Especially since it can hopefully deal with the awkwardness of branching and looping for you. noonat: changing the function at compile time is one part, but you still have to do something at runtime to treat a deferred function differently. That's outside the scope for coffeescript (and my plans ;)) |
how about |
"sync" seems a little too common of a word to reserve. I may be biased, as I'm currently writing a RSS sync application ;) also, i'm not sure that sync fits. What it is is an asynchronous call, that will automatically resume when it is done. It is not actually synchronous, this is simply shorthand for saying "resume this function when this asynchronous call is complete". I personally think "defer" captures that nicely, as it indicates background (async) work that the current function will depend on. But that could just be me. |
yes i think you're right. |
@jashkenas: oh, I misunderstood. Yeah, I don't think you could map calling scope to the continuation automatically... Continuation style development in Python can't even do this, and it has language support for yield. Twisted Python's deferreds (e.g. NodeJS promises) have a helper for continuation style stuff. It allows you to yield deferreds from within the function, and resume where you left off. So, this:
Becomes this:
This ends up creating a wrapper function which basically keeps continuing the foo() method until it stops yielding deferreds, then triggers the callback passed to the wrapper function (or the errback, if an exception occurs in the continuations). The wrapper adds itself as a callback/errback to the yielded deferred, and then passes the results off to the continuation when it gets a result. |
gfxmonk: can you convert your example into what the JS might look like - so we can see how you imagine it to work ? I mean this async: -> defer ... inner: -> defer async() ... outer: -> defer inner() ... |
sure, it would look something like: function async(_cb) { function inner(_cb) { function outer(_cb) { although the code might be a bit more verbose, and the callback variables would end up as _a, _b, etc.. Here's an actual example from my test cases, for a slightly more complex depiction of what's possible: input code:
(return_arg is a test helper that just returns the given argument via a callback) compiled code:
it ain't pretty, but that's kind of the point ;) |
gfxmonk: Thanks for the explanation. It's a very exciting approach to taming Node's asynchrony. Here's a modified snippet from the current
When your branch lands on master, I'll be able to write it like this, if I understand it correctly:
Although that ignores the error-as-first-argument bit of the Node API. Have you thought about how you're going to handle callback arguments, and expose them to the function using the |
Am I right in thinking that, as an unintended consequence of the
...since the callback/continuation would be invoked multiple times by |
gfxmonk: I don't know if your code currently handles it, but to be consistent with current coffeescript, I think a proper handling of callbacks with multiple arguments could be, referring to jashkenas example :
|
Also, if it's going to look like we're continuing execution in the same context, each passed callbacks would need to be bound to that context, so we don't get weird errors (would it make sense for |
Hey guys, I just stumbled across this issue while looking to add similar functionality to coffeescript. My interest is in adding some kind of generic support to make "functional reactivity" much much easier, e.g. Not sure if you guys have seen them before, but definitely worth taking a look! Do let me know if you think it might be worth supporting such things natively in coffeescript... It'd also be nice if the mechanism added to support Thoughts? |
Hey tav - can you explain what you mean by "functional reactivity" ? It's not clear from the links you gave (for me at least) |
Here's an piece on Ruby Fibers and trying to improve readability of async code: http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/ It's not directly applicable, but it's an interesting read. |
interesting indeed. That looks like a pretty cool library, I'm keen to investigate the magc behind it. It sounds (on the surface) a lot like async.js, but with much less awkwardness because it doesn't have to shoehorn it in to javascript. Ruby's fibers themselves sound a lot like python's generators. Is that accurate, or have I missed an important difference? |
After some deep thoughts about the whole synchrony question, I've come to think there's way too many ambiguities and uneasy assumptions, at least the way it is currently presented. I think, in particular, that some key concepts are mistakenly assumed as being some aspects of the same thing. So I'm going for some brainstorming : First, I don't think we should change the Implicit callback as assumed to be at the end, both for deferring and defining, implies comptatibility problems. A more flexible syntax could be based on a character, and I'm thinking
That way, deferring calls are not tightly bound to implicit callback. They are really seperate stuff and can be used in situations where they're not both needed :
So .. Ideas, anyone? |
I like the |
Sure, here's the above example, together with the equivalent vanilla coffeescript and its compiled javascript code : http://gist.github.com/342570 Also, I was just reading the past ticket about the |
Cool, it's good to have more people thinking about how this should work :) I personally have seen very few examples of callback as anything but the last argument, so I am keen to avoid the "~" magic variable. "defer" is much more explanatory and since it comes at the start of a call, you're less likely to miss its use (it does change control flow significantly, so I think it should be obvious). The "~>" operator is an interesting idea. One problem that I see is that an async function is very frequently (not in your examples, but I believe most of the time in actual usage) entirely determined by whether or not you use "defer". I'm quite worried about making the use of this mechanism too easy to get wrong. For example, consider the following function:
This seems like entirely reasonable code, but the return will actually have no effect (because you cannot usefully return from callback-based async code, you can only yield). Having the rule "always yield results instead of returning them after you use the "~" variable" is a pretty awkward thing to have to remember. And the penalties for returning instead of yielding are high - your program simply ceases execution, and you don't really know where things stopped working. Having said that, your final example there with the yield inside the anonymous function (forEach's iteration) is a good one. I don't think it ever could work with the current return-replacing behaviour, because it would be ambiguous what you meant. ~> could be useful for that, since I'm assuming that yield would look for the closest implicit-callback function in scope, rather than just the function it appears in. Although to be honest, i don't think a continuation should ever be used for anything other than a callback-based return. Using it for a looping construct is likely to be very confusing, as per sethaurus' example. |
I see what you mean, though I still think it could be useful to have a more general approach towards callback-based control flow (going back and forth between execution points, though not limited to async functions). We should make it so that it makes sense and is consistent, then build the async sugar on top of that. The whole point around That said, I think it is more confusing for people to see a Here, yielding to callback does not yet refer to asynchronous-oriented programming style, such as continuation. I think the latter can be implemented in coffeescript more easily if we have a readable callback semantic, such as I've tried to uncouple callback-based (read |
Also note that in your second example, you use:
this will return an error into whatever called your continuation. Since the async function thinks of the passed-in callback (your continuation) as a place to put return values, there is nothing meaningful it could do if your continuation were to actually return anything (every async function I've seen will completely ignore that error value). What you would probably want to do in practice is something along the lines of yield(err, result). Since it's unlikely we can make exceptions via callbacks actually work, the next best thing is to use node's approach where the first return value is reserved for any errors. If you were to do that, then you would have no use for both return and yield (in that second case). |
There, the I agree that, in that case, it would be better to yield the error, but I don't know about every situation and I'm not in favor of hard-coding what we think is better practice into language constructs. |
Just to pipe in about the error handling... I find Node's current error-handling convention completely unworkable. Errors are swallowed silently by default if you don't watch for them, and the code you need just to get a sane level of logging is absurd. Since the problem, fundamentally, is the difficulty of reconnecting an asynchronous error with the call stack(s) where it originated ... I think it would be worth considering ditching standard synchronous exceptions, and just defining top-level handlers for all of the exception types you want to cover. Instead of passing them to the callback, you'd attach the callback to the error before triggering the top-level handler. The handler can determine if it should be called or not. The default handler would error out with the message, the same as if an exception bubbles up to the top level. Just my two cents -- this isn't in scope for regular CoffeeScript, but it might be for your deferred branch, if you're trying to solve the async error problem. Potential code might look vaguely like:
|
jashkenas: yes, I'd love to fix this as well. I think it's fundamentally difficult though. Consider the simple case of:
your exception will bubble up through setTimeout, but I'm pretty sure there's absolutely nothing we can do to catch it. There is a chance that we could decorate all our continuations with error-handling code (assuming the [err, result] convention) but I'm not sure how we could connect the stacks. The global handler may be possible, but I'm not too sure what use it would be in terms of actually recovering from exceptions (and if you don't recover, is there any point catching it?). matehat: cool, I'm keen to see your branch :) a small note:
I'm planning to prevent this, actually ;) we already have a solution for multiple callbacks (a-la ruby's yield), and that is anonymous functions - and coffeescript makes them just as easy to write as a ruby block. I'm not sure you really gain much by allowing the following:
to be written as
It's a little nicer perhaps, but also more magic (and just another way to do something we already can). The magic I'm proposing is for a very specific use case (continuation-based returning), because I think the awkwardness in doing it any other way warrants such magic. |
gfxmonk: that's the idea. Because there is no way to correctly re-connect the stacks without resorting to twisty error-argument passing through every callback, instead you're responsible for attaching the information needed to recover from the error to the error object itself. Your handler should know what to do with it. A Still, not saying that the idea is any good, but just throwing it out there. I don't think it would work because it would have to be a convention at the Node.js level, and it simply isn't. |
ahh, yes fair enough - I was assuming you're getting errors from code you didn't write, which makes it difficult to attach the necessary metadata. But yeah I think it's an interesting idea, and it'd be good to get some more if others have thoughts :) |
Some thoughts
|
If we remove As for |
matehat: for all the cases where you might want to continue execution after a yield, are these also the exact same cases where you'd give it an anonymous function explicitly, rather than using Because if you pass someone a continuation, and they yield to it more than once, I'm pretty sure all manner of confusion will break loose. |
re weepy's point 1: we don't need anything like ~> for my proposed solution, but that's simply because we wouldn't deal with nested callbacks (you'd have to point 2:, if you were to return without calling the callback, then you're effectively stopping execution silently - from the perspective of your caller, the function never returns - not even with an error! Imagine if this were normal procedural code - you would either return (implicitly or explicitly), or you would raise an error. You can't simply stop the program from continuing, and even if you could then I doubt you would want to ;) |
Check out how Go is planning to add exceptions-via-defers ... seems relevant: http://groups.google.com/group/golang-nuts/browse_thread/thread/1ce5cd050bb973e4# |
Here's a simpler example from http://www.go-program.com/go-defer : # The defer statement executes a function (or method) when the # enclosing function returns. The arguments are evaluated at the # point of the defer; the function call happens upon return. func data(name string) string { f := os.Open(name, os.O_RDONLY, 0); defer f.Close(); contents := io.ReadAll(f); return contents; } # Useful for closing fds, unlocking mutexes, etc. |
forgot to say : So as you can see - it's only really relevant for clearing up at the end of the function (i.e. not synchronicity) |
Re In general, I think using character magic is a dangerous path. Succinct operators like these can be really nice, when you know the language. But if you go too far, as I think this does, the language can be incredibly hard to read for newcomers (Perl and Go are both examples of this... I think Ruby strikes a good balance). |
I'm getting a bit confused about all these proposed yields and defers etc etc. sleep: (x, fn) -> yield setTimeout(fn, x) getAfterWait: (url) -> sleep(1000) posts: defer $.get "/posts" yield posts [e, data] = defer getAfter1Sec "/posts" ($ @).html data cheers |
In my proposed paradigm, that's
It currently compiles to 42 LOC :) |
Ahh... all the |
How about swapping sleep: (x) -> setTimeout x, -> yield getAfterWait: (url) -> sleep 1000, defer posts: $.get url, defer yield posts [e, data] = getAfterWait "/posts", defer ($ @).html data Could that work ? |
I was thinking about it last night, and I'm happy to adopt
would become syntactic sugar for
but if you need to, you can specify the tilda position, like:
I still think we should still have the "defer" keyword, as the tilda is not obvious. Having the tilda only appear when the callback argument is not last will keep the flexibility, but avoid the awkwardness of many trailing tildas. Looks like this thread is getting too long and winding for most mortals. matehat: perhaps we should write up a wiki page (or two) with the current state of each of our proposals / examples? Is that okay with you jashkenas? |
gfxmonk: hey, this ticket belongs to you and matehat now. If you'd like a wiki page, go right ahead. Otherwise you could close this one and start another, with your current proposals/branches at the top. Also, feel free to pop into #coffeescript if you'd like to bounce ideas off folks. |
weepy: my version would look like:
|
Too many comments, as gfxmonk pointed out. I started a new discussion, http://github.com/jashkenas/coffee-script/issues/287, so we can close this one. |
Was just reading about the yield operator from NarrativeJS. It has some interesting ideas about how to make async JS more readable.
"Narrative JavaScript is a small extension to the JavaScript language that enables blocking capabilities for asynchronous event callbacks. This makes asynchronous code refreshingly readable and comprehensible."
More here: http://www.neilmix.com/narrativejs/doc/index.html
Could be something for CS, though it might be a bit wacky?
The text was updated successfully, but these errors were encountered: