backcalls: let's add them #2762

Open
michaelficarra opened this Issue Mar 4, 2013 · 166 comments

Projects

None yet
@michaelficarra
Collaborator

Copied proposal from #2662 (comment) below


Taken pretty much from the coffee-of-my-dreams list, the best way to explain my preferred backcall syntax is through example:

  • The continuation is captured and tacked on as the final arg by default

    <- fn a
    ...
    b

    compiles to

    fn(a, function(){
      ...;
      return b;
    });
  • The left side of the arrow specifies the argument list for the continuation. When only one is given, parens can be omitted.

    a <- fn b
    ...
    c

    compiles to

    fn(b, function(a){
      ...;
      return c;
    });
  • And for the rare occasion where the continuation is not the final argument of the function, we need to be able to specify where to put it with a marker like the <&> I use below. I'm open to suggestions for a better marker we can simply use an anonymous function to force the position of the callback.

    (a, b) <- (_) -> fn c, _, d
    ...
    e

    compiles to

    (function(_){ return fn(c, _, d); })(function(a, b){
      ...;
      return e;
    })

    or, if we detect this case, the simpler:

    fn(c, function(a, b){
      ...;
      return e;
    }, d);

this must be preserved in the continuation by rewriting references as we currently do in bound functions. I don't think we should worry about arguments rewriting, but that's your call.

Here's a real-world example @paulmillr gave in #1942:

compile = (files, callback) ->
  async.sortBy files, @sort, (error, sorted) =>
    return @logError error if error?
    async.map sorted, @map, (error, mapped) =>
      return @logError error if error?
      async.reduce mapped, null, @reduce, (error, reduced) =>
        return @logError error if error?
        @write reduced, (error) =>
          return @logError error if error?
          do @log
          callback? @getClassName()

and here it is using backcalls:

compile = (files, callback) ->
  (error, sorted) <- async.sortBy files, @sort
  return @logError error if error?
  (error, mapped) <- async.map sorted, @map
  return @logError error if error?
  (error, reduced) <- async.reduce mapped, null, @reduce
  return @logError error if error?
  error <- @write reduced
  return @logError error if error?
  do @log
  callback? @getClassName()

edit: dropped the callback position indicator, opting for "hugs" style

@00dani
00dani commented Mar 4, 2013

As observed back on #2662, backcall syntax works well with RequireJS:

($, someLib) <- require ["jquery", "someLib"]
codeThatUses $, someLib

Another potential use is amb:

x <- amb [1,2,3]
y <- amb [4,5,6]
amb() if x + y < 10
x * y
@michaelficarra
Collaborator

Also, for any Haskell fans, yes, this is just do notation under a different name. But don't let the procedural programmers know, or they will reject it as "elitist".

@summivox
summivox commented Mar 4, 2013

And for the rare occasion where the continuation is not the final argument of the function, we need to be able to specify where to put it with a marker like the <&> I use below. I'm open to suggestions for a better marker.

From the previous post I proposed ^ (caret) as non-final callback placeholder. Justification: visual hint of control flow.

(a, b) <- fn c, ^, d
# use (a, b)
e
@satyr
Collaborator
satyr commented Mar 4, 2013

JFYI, it originates from #1032.

@michaelficarra
Collaborator

Also, we could always drop the concept of a placeholder and force people to force the callback as the final argument. There's @00Davo's approach from #2662 of defining higher-order functions such as flip:

flip = (f) -> (x, y, rest...) -> f y, x, rest...
<- (flip setTimeout) 250
do takeAction

And there's also just manual partial application:

<- (_) -> setTimeout _, 250
do takeAction
@paulmillr

i’d vote for dropping placeholder +1 it brings more complexity, still

@epidemian
Contributor

And there's also just manual partial application:

<- (_) -> setTimeout _, 250
do takeAction

Using the right variable name you can have a person giving you a hug there:

<- (u_u) ->

I'd also vote against having the placeholder syntax; unless that concept of placeholder could be extended and used elsewhere too. In general i'm not much of a fan of this proposal, but i must admit that the use cases @00Davo mentions are quite really 😃

@shesek
shesek commented Mar 4, 2013

Is there any reason to automatically return inside the callback? its most likely that nothing would consume the return value of the callback anyway.

@ghost
ghost commented Mar 4, 2013

I'll throw in my hat in favor of the placeholder syntax. Just reading the code, this:

(a, b) <- fn c, ^, d
# ...
e

is infinitely more clear than this:

compose = (f) -> (c, d, callback) -> f c, callback, d
<- (compose fn) c, d
# ...
e

The latter also requires a separate "helper" function for each non-"standard" argument style that needs to be adapted for, harming interoperability with existing code.

@vendethiel
Collaborator

A compose function is unneeded here. I'm in favor of the hug approach

@ghost
ghost commented Mar 4, 2013

@Nami-Doc Ah, missed that. Yep, that looks good to me.

@epidemian
Contributor

Is there any reason to automatically return inside the callback? its most likely that nothing would consume the return value of the callback anyway.

Why not? In one of the use-cases mentioned in this thread, defining AMD modules, returning a value from the backcall would be really important:

($, _) <- define 'thing', ['jquery', 'underscore']
# Very typical use-case for AMD modules: returning a constructor function.
class Thing
  foo: -> # ...
@ghost
ghost commented Mar 4, 2013

What if intermediate callbacks need to have a value returned? Would backcalls not be usable in this situation?

@vendethiel
Collaborator

Why not?

Because we can't no-op them if chained (for example).

<- epi
<- dem i for i in [0..2]
an

I'd probably vote +1 tho.

@ghost
ghost commented Mar 4, 2013

@Nami-Doc I don't understand. Couldn't you tack a return, undefined, or null to the end of that snippet if you don't want it doing an implicit return?

@vendethiel
Collaborator

but that would no-op the inner backcall, not the one with the loop ;-).

@michaelficarra
Collaborator

@Nami-Doc: That's why you would never write it like that.

<- epi
for i in [0..2]
  <- dem i 
  an
return

or my preferred

<- epi
for i in [0..2]
  dem i, -> an
return

I hate bad code strawmen. You can write bad code in any language, we get it.

@ghost
ghost commented Mar 4, 2013

@Nami-Doc Ah, so

<- dem i for i in [0..2]

translates to

for i in [0..2]
  <- dem i
    # ...

and not

<- dem (i for in in [0..2])
# ...

?

@vendethiel
Collaborator

Considering
dem i for i in [0..2]
is
(dem i) for i in [0..2]
probably - coco treats it as an invalid callee (array)

@ghost
ghost commented Mar 4, 2013

Thinking about this a bit more, conditionals and loops complicate things quite a bit. Consider:

if a
  b = <- fn c
  d = b
else
  d = e()

console.log d
b =
  if a
    d <- fn c
  else
    d <- fn c

console.log b
console.log
  for item in array
    element <- transform item
result = null

console.log
  until result?
    result <- generate

Lots of messy edge cases to handle.

@vendethiel
Collaborator

Why are they complicated in your examples?

@michaelficarra
Collaborator

@mintplant: They don't complicate anything at all. The captured continuation is the rest of the block. Maybe you're thinking it's the rest of the function/program?

@ghost
ghost commented Mar 4, 2013

Ah, I see what you mean. I suppose I misunderstood the scope of this syntax addition. So, these only help with non-branching code paths, then?

@epidemian
Contributor
dem i, -> an

WTF happened to my name? xD

Anyway, between all those messages i got lost. The consensus was that it does make sense to return values from backcalls, wasn't it?

@michaelficarra
Collaborator

@mintplant: They work perfectly fine with branching code paths. It is a very simple transformation, you're overcomplicating it.

@epidemian: Yeah, it makes sense to auto-return from backcalls. That's why satyr/coco does it.

@vendethiel
Collaborator

WTF happened to my name? xD

Sorry, no idea why I came up with that :p . And yeah, that'd make sense.

@ghost
ghost commented Mar 4, 2013

@michaelficarra Right, I get that now. I never said they didn't work with branching code paths, just that they don't provide any special functionality to help account for them, which I had hoped they would, but now see is outside the scope of this change. Sorry for the misunderstanding.

@00dani
00dani commented Mar 4, 2013

The "hug-style" <- (u_u) -> is amazing.

As for the higher-order function approaches, this seems problematic:

compose = (f) -> (c, d, callback) -> f c, callback, d

Because it looks nothing like a standard definition of compose and has totally different semantics.

I think very simple higher-order manipulations, like flip, are reasonable to use, though, unless they have a big performance impact at runtime (although I doubt they'd really have a big impact?).

@michaelficarra
Collaborator

Hugs it is! I will edit the original proposal to omit the callback position indicator.

edit: done. Copied below:


And for the rare occasion where the continuation is not the final argument of the function, we need to be able to specify where to put it with a marker like the <&> I use below. I'm open to suggestions for a better marker we can simply use an anonymous function to force the position of the callback.

(a, b) <- (_) -> fn c, _, d
...
e

compiles to

(function(_){ return fn(c, _, d); })(function(a, b){
  ...;
  return e;
})

or, if we detect this case, the simpler:

fn(c, function(a, b){
  ...;
  return e;
}, d);
@vendethiel
Collaborator

This would make @tenderlove happy

@summivox
summivox commented Mar 4, 2013

Although the "hug" admittedly does the job, it's more or less desugared,
adds syntactic noise, as well as introducing a layer of unnecessary concept
hurdle to learners. By inventing a new syntax, we're already trying to
contract an idiom. Makes no sense to me to introduce a new one in the
process.

Is there a convincing reason to hate placeholder?

@shesek
shesek commented Mar 4, 2013

As a more generic solution for passing the callback in an arbitrary position, one could use something along the lines of:

marker = {}
midcb = (fn, args..., cb) -> fn.apply this, args.map (arg) -> if arg is marker then cb else arg
<- midcb bar, 1, 2, marker, 3, 4 
@vendethiel
Collaborator

"But duude, we could get a hug operator!"
wasn't partial application refused already?

@michaelficarra
Collaborator

@smilekzs: We're trying to make the language "simpler", or in other words, have fewer concepts. If we can combine backcalls and lambdas to support the rare case where the callback is not the final parameter, it allows us to have a simpler language than if we were to add a special placeholder syntax for that extraordinary case. That's why everyone was so in favour of using existing constructs.

@summivox
summivox commented Mar 4, 2013

I totally understand the call for simplicity.

I instead see the placeholder as an integral part of the backcall syntax,
not an added burden. It's intuitive to look at the "caret" as an extension
of the arrow symbol. Readability clearly IS simplicity.

I would argue that, removing the placeholder makes the unfortunate
transition to the "rare case" much more "hacky", which in turn
backfires on the original intention of simplifying. In the nominal
cases it's not needed anyway so no harm.

@shesek
shesek commented Mar 4, 2013

Any chance support for decorating the callback function can be added somehow? I regularly use some higher-order functions with callbacks to abstract away some common operations, like handling error delegation with an iferr [1] function. I would really like to be able to do that with backcalls.

Perhaps something like this: ?

iferr cb, (user) <- load_user id
# compiles as
load_user id, iferr cb, (user) ->

[1] https://gist.github.com/shesek/eff0c0abd31ad8457de8

edit: changed to a better example
edit 2: this obviously wouldn't work if backcalls works as an expression. do they?

@00dani
00dani commented Mar 5, 2013

@shesek You can already get that compiled output with the existing backcall design, like this:

contents <- baz qux, foo bar
# actually does not compile as
baz qux, foo bar, (contents) ->
# because it compiles as
baz qux, (foo bar), (contents) ->

Unless you meant something different? (Or unless I've misread the nesting of parentheses-less functions to do something different to what it actually does, which is also possible.)

Edit: Yes, I misinterpreted what baz qux, foo bar, (contents) -> actually means in CoffeeScript. Currently the only way to do that with backcalls is by using some sort of placeholder syntax (or the hug operator, if we go with that). Personally, I think using x <- anywhere that's not the start of the line is a confusing and bad idea, so I don't really think your proposal works so well.

The use of decorators like iferr is a good use case for supporting placeholders, I think:

contents <- baz qux, foo bar, ^
# compiles to
baz qux, foo bar, (contents) ->

It would be better if somehow we could apply decorators without needing an explicit placeholder like that, though.

@shesek
shesek commented Mar 5, 2013

@00Davo that should compile as baz qux, (foo bar), (contents) ->, which is quite different. For what I meant, the callback should be an argument to foo, an higher order function, which returns a new function that would be used as the callback argument.

edit: (in response to @00Davo's edit) yeah, my proposed syntax didn't feel quite right to me either, and I would prefer to have an placeholder-free syntax for that, too. any ideas for a different syntax, anyone?

@00dani
00dani commented Mar 5, 2013

It occurs to me that a combination of partial application and composition achieve this:

contents <- compose (baz qux, ...), (foo bar, ...)
# equivalent to
baz qux, (foo bar, (contents) -> *body here*)

That's still pretty verbose and rather unclear, though. Perhaps some sort of concise syntax for partial-application+composition is needed? contents <- baz qux <+> foo bar or something along those lines, maybe.

That idea's getting a bit off-track of backcalls, however, and it's special syntax for a very specific case at present. Thoughts?

@satyr
Collaborator
satyr commented Mar 5, 2013

What about fat version? <= obviously doesn't work.

@summivox
summivox commented Mar 5, 2013

If backcalls are not fat by default, +1 for <~ and ~>

@vendethiel
Collaborator

'Tis a bit too late to change that now, I think ;).

@00dani
00dani commented Mar 5, 2013

The plan was for backcalls always to produce fat-arrow callbacks, I believe. This seems a reasonable rule, since there aren't any particularly obvious use-cases that require backcalls not to preserve this.

@vendethiel
Collaborator

any particularly obvious use-cases that require backcalls not to preserve this.

not sure what you mean. That seems to be a fine use case to me :

<- $('img').click
alert @src
@00dani
00dani commented Mar 5, 2013

I personally don't think a backcall makes sense in that case, @Nami-Doc , mostly because the <- can't be interpreted as "magic =".

In practice that does seem like it might be used anyway, though, so we may need to consider this-preservation in backcalls a little more deeply.

@vendethiel
Collaborator

Considering we have async everywhere, I don't see why we'd need fat arrows everywhere. I don't need that when I fs.stat; ie ;).

@michaelficarra
Collaborator

In the proposal, backcalls were assumed to be all "fat" by default. In those uncommon cases where the callback has a meaningful context, either the value is also given as an argument (as in @Nami-Doc's case) or the callback is more of an "event handler" than a continuation (also as in @Nami-Doc's case). In the former case, use a parameter, and in the latter case, use a callback.

@hden
Contributor
hden commented Mar 5, 2013

Consider one use empty lines to improves readability, then what does the following code compiles to?

a <- fn b
c

do d

this

fn(b, function(a){
  c;
  return d();
});

or

fn(b, function(a){
  return c;
});

d();
@michaelficarra
Collaborator

@hden: The former. Empty lines do not delimit blocks, indentation changes do.

@hden
Contributor
hden commented Mar 5, 2013

@michaelficarra Thanks for clarifying.

So is it discouraged to use backcalls in the out most block, or I'll never be able to outdent back?

@vendethiel
Collaborator

You can outdent back.

createBinder = ->
  <- foobar
  @add 5

foo = createBinder()
@michaelficarra
Collaborator

@hden: Feel free to use them at program level. That's actually the only place where it's truly capturing the entire program's continuation.

@hden
Contributor
hden commented Mar 5, 2013

@Nami-Doc @michaelficarra You're right.

<- $
do makeSense
@00dani
00dani commented Mar 5, 2013

It's fine to use program-level backcalls. Without being allowed to do that, these don't work particularly well:

# load dependencies for this file
($, _) <- require ["jquery", "underscore"]
anyCodeCanGoHere()
evenCodeUsing $ and _
# or define a module with dependencies
($, _) <- define 'myModule', ["jquery", "underscore"]
@00dani
00dani commented Mar 6, 2013

Here's a question: Do we want a backcall syntax to work for assigning the function to some variable, instead of passing it into a function? The only place I see this as obviously useful is when assigning to module.exports, but that does seem a rather useful place for it. For example, writing a Gruntfile.coffee:

grunt <-= module.exports # syntax meaning module.exports = (grunt) ->
grunt.initConfig blah blah

However, it's possible just to do this:

export = (m) -> module.exports = m
grunt <- export
grunt.initConfig blah blah

So special syntactic support mightn't be necessary.

Edit: If we allow placeholders, this sort of structure might arise as possible:

grunt <- module.exports = ^
@00dani
00dani commented Mar 6, 2013

To throw another spanner into the very simple works of regular backcalls, what if they had special treatment when they can be interpreted as function arguments? Specifically, what if they worked like this?

(err, res) <- async.parallel
  users <- getTable "users"
  posts <- getTable "posts"
doSomethingWith res.users, res.posts
# translates as equivalent to
async.parallel {
  users: (cb) -> getTable "users", cb
  posts: (cb) -> getTable "posts", cb
}, (err, res) ->
  doSomethingWith res.users, res.posts

Might be a bit much, but currently backcalls don't particularly help much with async.parallel-type operations, and we don't have general partial application.

@shesek
shesek commented Mar 6, 2013

@00Davo I think it might be too confusing too have the <- operator mean partial application in some places, and backcalls in others.

You could still do that pretty easily with a partial application function (assuming passing arguments with indentation works?) with something like:

(err, res) <- async.parallel
  users: partial getTable, "users"
  posts: partial getTable, "posts"
doSomethingWith res.users, res.posts

And if a partial application operator is ever added, it should fit well with that syntax too.

@00dani
00dani commented Mar 7, 2013

@shesek Good point. That's still pretty readable, and it's less async-specific than my proposal, which relies pretty heavily on exactly how async.parallel's arguments work. (Passing arguments by indentation works in current CoffeeScript, and I think we shouldn't let backcalls interfere with that feature.)

What does <- mean in the context of a function argument, however? Will it just be treated as a syntax error, or will it work kind of like the "decorator" syntax you proposed earlier, or should there be some other interpretation entirely?

@shesek
shesek commented Mar 7, 2013

What does <- mean in the context of a function argument, however? Will it just be treated as a syntax error, or will it work kind of like the "decorator" syntax you proposed earlier, or should there be some other interpretation entirely?

It depends - are backcalls meant to be used as expressions or only as pure statements? If its a pure statement, than it should be a syntax error. If they work as expressions (and just return the synchronous return value of the asynchronous function), than backcalls as arguments should probably just be regular backcalls, and pass the return value of the backcall as the argument.

But I'm not sure how they'll work as expressions in some contexts... for example, with something like:

 foo
  bar: <- baz
  qux: <- corge

What would be the body of the function passed to baz? if it consumes the rest of the block, it should be the qux: <- corge, but that doesn't make much sense :-\ Backcalls effect the rest of the block, so its kinda odd to use them in the middle of another expression. Perhaps they should be statements?

edit also, any thoughts on http://irclogger.com/.coffeescript/2013-03-06#1362609333 ?

@00dani
00dani commented Mar 7, 2013

I don't particularly like putting (args) <- on the end of the line, as suggested in that IRC log. Making backcalls closer to other expressions is a good idea, but by requiring them to be located like normal callbacks you lose the visual similarity of x <- operation and x = operation.

It does afford the ability to decorate backcall functions much more simply, and it extends to variable assignments, without complicating the setup with placeholders, though. I'm not sure those advantages are worth giving up the original "magic =" arrangement, however.

@michaelficarra
Collaborator

@jashkenas: Can we get an official go-ahead on this feature? I find myself wanting it nearly every day I write CoffeeScript. I want to make sure it will be accepted before putting in the time to implement it.

@g0t4
g0t4 commented Mar 21, 2013

Just FYI https://github.com/Sage/streamlinejs has some interesting implications for this, particularly in how it also adds capacity to get back to try/catch/finally blocks, IMO error handling is as much of a problem with the verbosity of callbacks as other things mentioned above.

@bjmiller

+1 for backcalls usable as expressions, if possible. One of the guiding principles is that everything should be an expression that returns something if it can, right?

@jashkenas
Owner

@michaelficarra -- I'm afraid I can't give you the go ahead (for this particular version of the compiler, at least). I've skimmed/read through this ticket several times, but need to sit down and more thoroughly tinker with some of the examples, and think through it a bit more.

At the moment, I'm fairly strongly leaning towards leaving them out ... but that isn't final.

@mehcode
mehcode commented Mar 21, 2013

Something I'd like to add is that if this ever gets added It'll be strange if we can't have unbound back calls (from what I can see <- in its in current form is always bound). <= is a bit too ambiguous unfortunately to allow <- to not be bound. I'd vote for switching to <~ and ~> for bound calls if it's decided to include back calls.

@ashtuchkin

For the record: we extensively use a error handling helper along the lines of:

linesInFile = (name, cb) ->
  fs.readFile name, "utf8", errTo cb, (text) ->  # errTo sends error straight to cb.
    cb null, text.split("\n").length

With backcalls, it would be nice to keep this convenience, something like this (rewriting real-world example above):

compile = (files, cb) ->
  sorted <- async.sortBy files, @sort, errTo cb
  mapped <- async.map sorted, @map, errTo cb
  reduced <- async.reduce mapped, null, @reduce, errTo cb
  <- @write reduced, errTo cb
  do @log
  cb? null, @getClassName()

To keep it, we need to be sure that the backcalls are applied to the last function of the line (errTo in our case).

@vendethiel
Collaborator

To keep it, we need to be sure that the backcalls are applied to the last function of the line (errTo in our case).

I'm afraid this is not possible :(.

<- a b would then mean a b -> which is confusing and makes them pretty unusable.

@00dani
00dani commented Mar 22, 2013

It can be done, slightly more verbosely, given a placeholder. For example, with ^ as placeholder, that example would be written like this:

compile = (files, cb) ->
  sorted <- async.sortBy files, @sort, errTo cb, ^
  mapped <- async.map sorted, @map, errTo cb, ^
  reduced <- async.reduce mapped, null, @reduce, errTo cb, ^
  <- @write reduced, errTo cb, ^
  do @log
  cb? null, @getClassName()
@shesek
shesek commented Mar 22, 2013

@ashtuchkin heh, I'm using a function [1] exactly like your errTo, only with a different name. I asked about using that with backcalls a few comments ago [2]. I really want to be able to use it with backcalls, too :)

As @Nami-Doc said, the syntax you proposed already has other valid meaning in the current proposal, so it wouldn't work. I suggested an alternative syntax at [2], and its also possible to do that with placeholders, if they're added.

Also, you can do that with an higher-order function, like that:

# I should probably find a shorter name for that
bound_error_handler = (fn, errcb) -> (a..., succcb) -> fn a..., iferr errcb, succcb

sorted <- (bound_error_handler async.sortBy, cb) files, @sort

edit: another way to do that is with the "hug operator": (sorted) <- (_) -> async.sortBy files, @sort, iferr cb, _

[1] https://gist.github.com/shesek/eff0c0abd31ad8457de8
[2] github.com/jashkenas/coffee-script/issues/2762#issuecomment-14414831

@ashtuchkin

Thanks for the explanation. I then vote for placeholder support, because the majority of callbacks in my code use errTo (i.e. all interactions with node, express request handlers, etc.) and this feature has no value for me if I cant use it or it makes the code uglier.

@michaelficarra
Collaborator

Wow, have none of you bothered reading through the thread? We dropped placeholder support because it can always be replaced with a function literal.

edit: Just saw @shesek's edit above where he mentions it.

@ashtuchkin

You dropped it because non-last callback was seen as a rare case, which it is. Here we have a different case - error handling helper, which is arguably more common and writing the hug operator on almost every callback is definitely a show stopper for me.
It could be just my pain, I don't know. And I definitely want to make the language as simple as possible. But without placeholders, maybe optional ones, this feature has very limited usage in my case.

@michaelficarra
Collaborator

@ashtuchkin: Ah, okay. I agree, that is a more common case, but also a case where a higher order function like @shesek's would be more meaningful. I would opt to use that function since it works as an annotation, showing intent better than either the inline function or the placeholders.

@g0t4
g0t4 commented Mar 22, 2013

Perhaps we should just use https://github.com/Sage/streamlinejs, it already
takes care of this problem.

On Fri, Mar 22, 2013 at 10:59 AM, Michael Ficarra
notifications@github.comwrote:

@ashtuchkin https://github.com/ashtuchkin: Ah, okay. I agree, that is a
more common case, but also a case where a higher order function like
@shesek https://github.com/shesek's would be more meaningful. I would
opt to use that function since it works as an annotation, showing intent
better than either the inline function or the placeholders.


Reply to this email directly or view it on GitHubhttps://github.com/jashkenas/coffee-script/issues/2762#issuecomment-15304930
.

@ashtuchkin
sorted <- (bound_error_handler async.sortBy, cb) files, @sort

In my view it hides whats really happening (async.sortBy), placing too much attention to error handling, especially with those additional parentheses :(
With placeholder (or the way I suggested initially) it's just a suffix that you pay little attention to when reviewing the code, but still notice when its absent and easily understand what happens when error arises.

@jamesonquinn

I like this syntax, and think it could fit with mainline coffeescript.

And I like iced, and don't think that could fit with mainline coffeescript.

But as far as I can see this gives you only about half of what iced-cs gives you. That is, even with this, there is no clean, DRY way to build iced-cs as a library, with "await" and "defer" as functions instead of keywords.

But I think that the gap is small. The only thing missing is a way for "defer" to sneak variables from an inner to an outer context. I believe that a small bit of extra syntax could do this.

This is not easy to think out, and I'm sure my first proposal will be lacking. But I'm thinking along the lines of syntax that would let you write a library to make the iced-cs:

await
  for item, index in someList
    item.getValue defer values[index]
...

as:

<- await@@ (defer) ->
  for item, index in someList
    item.getValue defer (@@values[index]) -> 
...

The idea is that the @@ in await@@ sets up an outer context which all inner @@variables will be declared. So:

  1. "await" would just be a library function such that "await f1, f2" calls "f1 defer", where defer is a function such that "defer f3" returns a "satisfied" function which simply passes its arguments to f3. When f1 is done, "await" waits until all the "satisfied" functions which were created have been called, then calls f2.
  2. The syntax (@@x) -> ... is shorthand for (x) -> @@x = x; .... Thus in the code above, the "satisfied" function returned by defer, sets values[index] in the context of the outer await.

I'm not satisfied with the above syntax (for instance, the @@ as it stands doesn't allow nested await statements; you could patch that with "await@@x" and then "x@@varName" but that's getting even uglier), but to me it does serve as a proof of concept that with backcalls plus a bit of syntax for putting variables into outer contexts, you could make it possible to write iced-cs as a library without violating coffeescript's "it's just javascript" philosophy in the way that the current iced-cs does.

@vendethiel
Collaborator

I don't like your syntax (especially since we don't have @@)

(also edited your message to add coffee highlighting)

@jamesonquinn

@Nami-Doc: Neither do I; it was intended as a proof-of-concept. My point was that:

  1. I was +1 on adding backcalls to cs.
  2. Even with backcalls, I thought that you couldn't write iced-cs as a library, so I thought that it was premature to close the iced-cs issue on this one.

But on second thought, 2 isn't true. With just backcalls, you could write a library (with a single function "await") so that the following would work:

awaitResult <- await (defer, innerAwaitResult) =>
  innerAwaitResult.values = []
  for item, index in someList
    item.getValue defer (@values[index]) -> 
...
console.log awaitResult.values[2]
...

That is, instead of inventing ugly new syntax to put variables into outer contexts, you can simply put them as properties onto an object which you then return.

So now that I realize that this would truly make iced-cs unnecessary without breaking the "it's just javascript" philosophy, I am going from +1 on this issue to +2.

@jamesonquinn

So the more I think about this, the more I am enamored of its awesome power. @jashkenas : what would it take to help convince you? A working patch? A draft version of iced-coffeescript-as-a-library (which would work with a smoothed-out version of the syntax I proposed just above)? More votes for this issue? Time to think without us pressuring you?

@jamesonquinn

Trivial question: what should the indentation be like in the compiled javascript?

@vendethiel
Collaborator

same as a function call.

@michaelficarra
Collaborator

@jamesonquinn: I love the enthusiasm. I think we can pretty much call this feature request accepted at this point. We've gotten a lot of support, shown its usefulness, and discovered some useful idioms (such as the one you mentioned or the anonymous function for functions that take the callback in non-final positions). I could almost guarantee a good pull request would be accepted, but it would be great if @jashkenas confirmed that. I'll open an issue in CoffeeScriptRedux to implement them.

@michaelficarra michaelficarra referenced this issue in michaelficarra/CoffeeScriptRedux Jun 22, 2013
Open

implement backcalls #217

@jashkenas
Owner

Time, I'm afraid, is not on my side ... but. In general, for any feature, what we need is a clean, well-written pull request, with tests, and a demonstration of how the feature works and is useful in real-world code in the ticket. Those PRs are easy to simply merge quickly. (Perhaps too easy ;)

That said, I'm still not sold on the parallelism between back-calls and the usual straightline syntax of programming without the continuation. In particular, this bit:

simply use an anonymous function to force the position of the callback.

(a, b) <- (_) -> fn c, _, d

... seems far, far too cryptic to add to CoffeeScript, even if it's a rare case that the callback happens to not be in the tail position. Is there a better way, or is Michael's summary at the top still state of the art?

@vendethiel
Collaborator

Things have been proposed for this case, not sure what you think about them.

@shesek
shesek commented Jun 24, 2013

@jashkenas what you think about #issuecomment-14413104?

@jashkenas
Owner

@shesek: A pretty nasty hack, no? You wouldn't want to tell or teach a beginner to do that ... would you?

@michaelficarra
Collaborator

@jashkenas: There were suggestions for a placeholder syntax, but I rejected them in favour of keeping the language simpler. I don't think that anonymous function looks so bad. Is the _ making it look cryptic to you?

@shesek
shesek commented Jun 24, 2013

@jashkenas Well, I wouldn't wanna teach a beginner how to write that higher-order function... but just learning to use that function shouldn't be that problematic. It could feel very "syntaxy" if its given proper names, like <- callpos load_user, id, _, true (underscore being the "marker", callpos being still not a very good name, but I couldn't think of anything better)

@bjmiller

Let's take that example. How would you read that off to someone in English (or, your own local spoken tongue)? If you can break it down in one sentence to a beginner with CS, then I think maybe that meet's JA's criteria. (As always, I could be wrong.)

@jashkenas
Owner

Perhaps I just don't understand the example well enough. Can someone write @paulmillr's compile example, from the bottom of Michael's post, using this syntax ... as if the callback was the first argument to each function, instead of the last one?

@vendethiel
Collaborator
compile = (files, callback) ->
  (error, sorted) <- async.sortBy _, files, @sort
  return @logError error if error?
  (error, mapped) <- async.map _, sorted, @map
  return @logError error if error?
  (error, reduced) <- async.reduce _, mapped, null, @reduce
  return @logError error if error?
  error <- @write _, reduced
  return @logError error if error?
  do @log
  callback? @getClassName()
@shesek
shesek commented Jun 24, 2013

@Nami-Doc that relies on the placeholder that was canceled (tho it was ^, not _, which is already a valid identifier). I think he meant with the "hug operator", or possibly with the higher-order function.

Edit: wording and attempted mind reading

@vendethiel
Collaborator

Yes, that's what the code is about. The symbol doesn't matter.

@jashkenas
Owner

@Nami-Doc -- that doesn't seem like what Michael is suggesting with the anonymous function trick. That looks like a placeholder.

@vendethiel
Collaborator

AH, I misread. Ok :

compile = (files, callback) ->
  (error, sorted) <- (_) -> async.sortBy _, files, @sort
  return @logError error if error?
  (error, mapped) <- (_) -> async.map _, sorted, @map
  return @logError error if error?
  (error, reduced) <- (_) -> async.reduce _, mapped, null, @reduce
  return @logError error if error?
  error <- (_) -> @write _, reduced
  return @logError error if error?
  do @log
  callback? @getClassName()
@shesek
shesek commented Jun 24, 2013

Edit: @Nami-Doc beat me to it.

I don't find it very readable myself, too.

Edit 2: Not really related to the callback position, but I really dislike all the error handling you'd have to use. I really think we should consider a better way to handle that (see #issuecomment-14414831, #issuecomment-15292734 and this).

Edit 3: We can use a much nicer higher-order function than what I wrote above. I think that looks much better and is an acceptable solution for error handling:

iferr = (errcb, fn, a..., cb) -> fn a..., (err, b...) ->
  if err? then errcb err else cb b...

compile = (files, callback) ->
  sorted <- iferr @logError, async.sortBy, files, @sort
  mapped <- iferr @logError, async.map, sorted, @map
  reduced <- iferr @logError, async.reduce, mapped, null, @reduce
  <- iferr @logError, @write, reduced
@jashkenas
Owner

One last wrinkle ... bound-function-back-calls? Do we have an acceptable solution? They're pretty common.

I see in this ticket that backcalls are assumed to be bound by default ... but that doesn't seem quite right, as the parallel between <- and -> is broken.

@00dani
00dani commented Jun 24, 2013

We could use Coco's solution of adding ~> for bound functions and thus an analogous <~, perhaps?

@vendethiel
Collaborator

I see in this ticket that backcalls are assumed to be bound by default ... but that doesn't seem quite right, as the parallel between <- and -> is broken.

agreed that we can't assume bound, <= is not possible (and we're not using ~>)

@michaelficarra
Collaborator

Backcalls were assumed bound because you don't want this changing out from under you without a change in indentation. They are supposed to be implicit continuations with explicit scope enrichments. It's unfortunate that there won't be parity between => and <-, but I don't think that's so bad.

@jashkenas
Owner

It's unfortunate that there won't be parity between => and <-, but I don't think that's so bad.

I think it's fairly bad. Again, imagine trying to explain to a beginner, ->, <- and =>. All different syntaxes for writing slightly different versions of a "function". If <- is semantically kin to =>, but syntactically an inversion of a normal function callback, it gets pretty confusing.

As a small anecdote -- I tried to explain this proposed feature (in all it's glory) to some JS developers last night on the back of a cocktail napkin, and it didn't really translate well, especially with the subtleties.

@00dani
00dani commented Jun 25, 2013

Just to throw another spanner into the works, how do backcalls work in conjunction with promises? Since promises are monadic and backcalls are do notation, they really should work well together, but as far as I can tell a promise do block has to be written "explicitly" like this:

y <- (f x).then
z <- (g y).then
h z

Is there any chance of cleaning that up without crippling backcalls' other applications? (Obviously we could make backcalls y <- x desugar to x.then (y) ->, rather than x (y) ->, but that would render the syntax pretty much useless in non-promises situations.)

@jamesonquinn

@00Davo : I think that you'd probably write a library function to make it look better:

after <- After f, x
after <- after g
after.final h

... or something of the sort. But in general, I think it's OK if promises and backcalls are largely separate solutions to similar (but not identical) problems.

Afterthought: a library like this could do a similar thing as promises do in terms of passing error handling to the next available handler.

@jamesonquinn

@jashkenas : how do you feel about > and <?

Also, how would the thread in general feel about using the same symbol as a placeholder? I mean:

(a, b) <- fn c, <-, d

or maybe

(a, b) <~ fn c, ~, d

I'm sympathetic with the pro-hugs argument (simple syntax) but if jashkenas favors the pro-placeholder side (easier to read and learn) then that's fine too.

@jamesonquinn

New idea: what about =< ? It's not as pretty as <= would be, and "hugs" aren't as cute looking, but it does clearly signal that this is bound syntax.

@michaelficarra
Collaborator

@jamesonquinn: I like =< a lot. I still think we should do without a placeholder syntax for now. It can always be added in later.

@jashkenas
Owner

So, in order to make some progress on this ... personally, I'm still not quite entirely sold, but am very much sitting on the fence. If someone want's to cook one up, I'd love to see two linked pull requests.

  1. Implement Backcalls along the lines of Michael's description, sans placeholders, with some sort of rationale and solution for "bound backcalls".
  2. A separate pull request that attempts to implement all of the compiler's asynchronous bits in the most idiomatic way, using the new feature.

Either way, with a merge or without, I think that would be instructive.

@epidemian
Contributor

I'm really not sold on the idea of backcalls yet. I think it complicates the syntax and adds new corner-cases for not much gain.

I think the problem with the (in)famous JS Callback Hell is not so much the indentation per se, but the complexity of having all those different scopes and possibly delayed executions intermixed. I think the proposed backcall syntax does not solve the problem of the complexity of callbacks, it just makes them look like sequential code (when its execution is not necessarily sequential), but the complexity is still there: lots of different function scopes mixed-up, returns and exceptions not working as in sequential code, etc.

The most compelling use-case for backcalls i've seen is removing some of the tedious boilerplate and gratuitous indentation AMD modules or similar constructs introduce (e.g. jQuery.ready). Maybe for those cases we could introduce a simpler construct, like a different kind of function token, like a long arrow -->, that encloses everything after it without needing an extra level of indentation.

If backcalls get implemented, i hope they ends up being a very simple transformation as @michaelficarra proposes (no placeholder syntax or anything... the "hug" idea was a joke; i wouldn't like seeing it become an indiom xD). But i fear it'll still be unclear when they should be used and where normal callbacks should be used instead.

@jashkenas
Owner

You make fine points. I more or less agree -- one little clarification though, in backcalls' defense...

it just makes them look like sequential code (when its execution is not necessarily sequential)

... the point of the backcalls is that they are the serial transformation of the continuations, not any parallel version. If you write a series of backcalls in a single block of code (no indentations) ... then every line within that block will be executed sequentially ... just not necessarily right away. That's part of the reason why they make some sense.

But your points about return and exceptions are quite right.

we could introduce a simpler construct, like a different kind of function token, like a long arrow

... This is that different kind of function token. That's all that this is.

@jamesonquinn

I think that the simplest use cases are nice, but not compellingly necessary. The reason I think this language feature is important is that it would enable organic growth of powerful tools. My iced-cs-as-a-library and backcall-style-promises-library ideas, mentioned above, are only two of the many possibilities. Iced-cs features are powerful (read the iced-coffeescript docs for why: http://maxtaco.github.io/coffee-script/ ) and to me, backcalls are by far the cleanest way to enable that style of programming in coffeescript. (In fact, that's what led me to this issue in the first place.)

I am confident that if this were enabled, we'd soon have libraries to enable tricks we don't currently imagine. Some of these would turn out to be dead ends, but some could catch on.

Let a thousand flowers bloom.

@jashkenas
Owner

Let a thousand flowers bloom.

Indeed, and absolutely. A thousand flowers and a thousand forks (of which we're currently at nearly 900). That said...

I am confident that if [backcalls] were enabled, we'd soon have libraries to enable tricks we don't currently imagine.

I don't think so. Backcalls are actually a somewhat restrictive way to add a flavor of syntax that favors a particular asynchronous style -- e.g. serial-continuation-as-the-final-callback-to-the-function. They actually make it less easy and less likely for other styles (icey await and defer, promises of various stripes) to become successful within CoffeeScript.

@jamesonquinn

As to "tricks we don't currently imagine" I'm thinking in terms of

subtrick? =< trick fun, arg1, arg2

where "trick" is some kind of "meta-function" or "decorator" for fun arg1, arg2. Three ideas I can already imagine for "trick":

  1. iced-like parallel callback bundling: like "await", this would be something that could easily refactor to a serial version by just switching places between it and "for".
  2. promise-like error-handling: some way to declare a single error handler for a whole chain of callbacks.
  3. placeholders to ensure the callback comes last.

But I suspect if I can come up with 3 such ideas off the top of my head, that further possibilities exist. I also suspect it would take a bit of experimentation get an implementation for any of the above ideas (or the combination of them) really "right".

@paulmillr

I don’t think backcalls are worth it.

Node will switch soon to yield, it is already in v8 and actually works (people started creating modules for it).

Browser async code is less a mess.

We have promises until yield which solve all callback problems and provide returns and throws.

Yield code:

var sorted = yield sort(files, sorter);
var mapped = yield async.map(sorted, this.map);
var reduced = yield async.reduce(mapped, null, reduce);
yield thiswrite(reduced);

Promises code:

async.sortBy(files, sort)
  .then (sorted) =>
    async.map sorted, @map
  .then (mapped) =>
    async.reduce mapped, null, reduce
  .then (reduced) =>
    @write reduced

backcalls code (looks shitty):

compile = (files, callback) ->
  (error, sorted) <- async.sortBy _, files, @sort
  return @logError error if error?
  (error, mapped) <- async.map _, sorted, @map
  return @logError error if error?
  (error, reduced) <- async.reduce _, mapped, null, @reduce
  return @logError error if error?
  error <- @write _, reduced
  return @logError error if error?
  do @log
  callback? @getClassName()
@jamesonquinn

@paulmillr : You've left the error handling out of the promise and yield code, which isn't very fair, even if you're right that it's cleaner than the naive backcall code you gave. But my point is that libraries would develop so that the backcalls code would be prettier. Something roughly like:

compile = (files, callback) ->
  seq = new Sequence
    errorHandler: (err) -> @logError err
  (seq, sorted) =< seq async.sortBy, seq, files, @sort
  (seq, mapped) =< seq async.map, seq, sorted, @map
  (seq, reduced) =< seq async.reduce, seq, mapped, null, @reduce
  seq =< seq, @write, seq, reduced
  do @log
  callback? @getClassName()

The above is a very rough indication of the kind of thing you could do. For instance, "Sequence" is the first name that occurred to me; I'm pretty sure you could find better words for that class and object. The Sequence class should be written so that the above works exactly like the code you wrote. Notice that in the above, a Sequence instance is assumed to have both placeholder and error-passing functionality.

@paulmillr

@jamesonquinn it is absolutely fair -- 98% of the time you don’t want to specify error handlers for every single error. Still, with yield it is super-simple.

Sequence is just another package for solving a problem that is more effectively solved with yield or promises.

Yield:

try {
  yield ...
  yield ...
} catch(err) {this.logError(err)}

Promises:

// just last step
.then(@write, @logError)
@jamesonquinn

My point is that

  1. yield doesn't exist today, and won't reliably exist in the browser for years.
  2. Promises give one way to solve this problem, but it's possible to imagine that backcalls would give another. If the hypothetical "Sequence" package, whose functionality I dreamed up in under 5 minutes, has warts, that doesn't mean that any possible such package would have them.

... added later:

Here's a quick demo of a bunch of different things backcalls could enable:

katch =< trie (yeeld) ->
  arg3 =< yeeld someAsync, arg1, arg2 #yeeld soaks up err and passes it to katch
  arg5 =< yeeld annoyingArgumentOrderAsync, arg3, yeeld.cb, arg4 #look, placeholders!
  someList =< yeeld.noErr errorFreeAsync, arg5 #when cb isn't `(err, result) ->`
  hurryUp =< yeeld.either (yeeld2) ->  #two sources, we'll go with the fastest
    onePossibleDataSource yeeld2
    anotherDataSource yeeld2
  results =< yeeld.several (defer) ->  #look, iced-style parallelization!
    setTimeout defer.finish, 10000 #global timeout, unlike iced-cs, 
    defer.results = []
    for item, index in someList
      item.asyncGetVal defer (@[index]) ->
        #you could put a conditional defer.finish() or yeeld.throwe() here
  display results
katch (err) -> logError err #errors get sent skipping to here
katch.finaly -> cleanup()
moreStuff()

All of the above is possible with a smart enough "trie" package. As always, the above names are first-pass placeholders, and I'm sure a finished package could improve them.

That's a kitchen-sink vision of what could be; it includes more possible tricks than are available today with promises, yield, and iced combined. And thus it would require roughly 6 different kinds of sub-entities plus 6 special methods on those entities. I'm sure that many would consider that overkill. So perhaps eventually most of the community would settle on a package that did only half of that or less. And even the ultra-purists would be fine; they wouldn't have to use any package at all. They'd still get the power of =< require.

@jamesonquinn

@jashkenas : "A separate pull request that attempts to implement all of the compiler's asynchronous bits in the most idiomatic way, using the new feature." Can you suggest a good place to start with that? (I'll see if I can make time to do 1 and start on 2 next week.)

@vendethiel
Collaborator

We don't have a lot of async, so I'd say mainly command.

@jamesonquinn

Ok, looking at command, you get things like:

watch = (source, base) ->

  compileTimeout = null

  ...

  compile = ->
    clearTimeout compileTimeout
    compileTimeout = wait 25, ->
      ...

If translated into a backcall, would that last line be:

    compileTimeout = =< wait 25
    ...

That is,

  1. Should the value of the whole backcall expression be the return value of the async function call?
  2. Even if it is, is the above = =< idiomatic?
  3. Unrelated but inspired by this problem: what if you had compileTimeout =< wait 25 in the above? The simple backcall transform wouldn't work as expected, because compileTimeout already exists in outer scope. Should the compiler throw an error for cases like this; or should it add code to do what you'd expect, similar to the (@x) -> idiom?
@vendethiel
Collaborator

= =< ?

@jamesonquinn

If =< fn a turns into fn a, =>, then b = =< fn a would turn into b = fn a, =>, right? But yes, even if the compiler worked like that, I think that = =< is irredeemably ugly. So while I'm still very much in favor of backcalls, I'm not sure that there's any good place to actually use them in the codebase of coffeescript itself.

@epidemian
Contributor

Besides, =< alone looks like a sad smiley =<

@jashkenas
Owner

irredeemably ugly

...

not sure that there's any good place to actually use them in the codebase of coffeescript itself

... if that turns out to the the case, that's a pretty excellent bit of evidence against adding them.

@michaelficarra
Collaborator

@jashkenas: This compiler has no async interface. It exposes an entirely sync interface and doesn't interact with slow/network/blocking resources. This just isn't a good project to show off a feature like backcalls. I think the minimum requirement would be a library that finds value currently in depending upon caolan/async. At least that library would be passing around continuations.

@jamesonquinn

@jashkenas : I agree with @michaelficarra that one wouldn't expect a compiler to be the best place to show off this feature. Given that, would either of you like to suggest how I should look for a good demo ground; that is, an existing project, in coffeescript and with a slow/network/blocking component?

@ricardobeat
Contributor

yield 😿

yield doesn't exist today, and won't reliably exist in the browser for years.

It's already in node, it has existed in FF for a couple years (though slightly different), and should be coming to webkit/blink in the near future. I'd bet on > 80% support about a year from now.

@zhaizhai
zhaizhai commented Aug 8, 2013

In case anybody is interested, I started implementing some of the suggested features in the backcall branch here:

https://github.com/zhaizhai/coffee-script

Some simple examples are in test/backcall.coffee. I figure it could be a useful starting point for others interested in exploring this feature.

@tashemi
tashemi commented Aug 8, 2013

I like CoffeeScript because of its minimalism simplicity and clearance.
With this construction I have too much questions:
a <- fn b
fn(b, function (a){})
Why does first argument b of fn go after second? Why does body of callback go after first argument of fn? Why is a an argument of callback? It confuses me a lot. I can not understand this construction after first time reading.

@00dani
00dani commented Aug 8, 2013

@tashemi:

Why does first argument b of fn go after second?

The arguments go in the order they do because of an existing convention: Most all asynchronous calls in Node.js take the form fn(arg1, arg2, …, argN, callback), so backcalls by default assume that the callback belongs at the end. The rest of the arguments arg1 to argN are not reordered by the backcall transformation.

res <- fn arg1, arg2, …, argN
# equivalent to
fn arg1, arg2, …, argN, (res) ->

Why does body of callback go after first argument of fn?

Well, it would anyway, even in vanilla JavaScript. It's probably more useful to view the body of the callback as going after the entire line a <- fn b. As for the reason it does that…

Why is a an argument of callback?

To make it look more like synchronous code. In fact, that's the entire purpose of backcall transformation, really. The rest of the function (well, the current indent block) is wrapped up in the callback body because that'll work more like sync code, and a is made the callback's argument because that'll work more like sync code, and so on.

Specifically, the arguments to the callback are pushed to the left side by analogy to the standard = operator: <- is meant to be viewed as a magical, asynchronous version of =. For instance:

# synchronous code
a = fn b
console.log a
# asynchronous code
a <- fn b
console.log a
@jamesonquinn

I agree with 00Davo's response. But wasn't the =< syntax currently preferred over the <- one?

@tashemi
tashemi commented Aug 9, 2013

@00Davo thanks for response. Now it looks more obviously.

@00dani
00dani commented Aug 9, 2013

I'm not totally sure where we got =< as the symbol for all backcalls. I believe it was suggested as an alternative specifically for bound backcalls, by analogy to =>, which would suggest that the corresponding unbound backcall syntax should be -<.

Both of those (-< and =<) look off to me, however, compared to <-. I'd really prefer <~ for bound backcalls and an analogous ~> for bound callbacks (which seems to be what Coco has already), although I doubt adding new syntax for bound callbacks is an option at this stage.

(It's perhaps worth nothing that the bound backcall =< is now pretty close to Haskell's inverse bind =<<. This… may not be particularly relevant for anything, but I thought it was interesting.)

@vendethiel
Collaborator

Well, <- exists in haskell as well, doesn't it? :)

@jamesonquinn

All backcalls are bound. A non-bound backcall would only be an evil gotcha; a sudden and drastic change in context without a corresponding change in indentation.

It would be great if we could use something perfectly symmetric. But <- for backcalls and -> for bound function declarations isn't going to happen because coffeescript is NOT going to radically change the function declaration syntax just for a feature that some might never use. And <= is simply not going to happen for similarly obvious reasons.

So we're choosing between various second-best options, principally the following:

  • <-:
    • Pros: Looks like an arrow, similar to function declaration. Leads to cute-looking "hugs" idiom.
    • Cons: Resembles the unbound function declaration, but actually corresponds to a bound function. Yuck.
  • =<:
    • Pros: Closer resemblance to bound function declaration than to unbound. Resembles an assignment operator, which helps develop intuition about what's happening here. Suggests the name "fat arrowtail", which helps remind developers that the assumption is that the callback is the tail argument
    • Cons: Leads to an less-pretty-looking "hugs" idiom: =< (_) => setTimeout _, 250 (Sad person, pac man, happy person? The "I need pac man" idiom?)
  • <~:
    • Pros: Does not look like <-
    • Cons: Would work best if bound functions were ~>, but that's either a major backwards-incompatible change in removing => (not going to happen), or a confusing and unnecessary TMTOWTDI (there's more than one way...).

My vote is that =< gets a B, <~ a D, and <- an F.¹

¹ Off-topic: Voting works best by giving grades and taking the median grade; it solves most vote-splitting and is relatively simple and non-strategic for voters. Tied medians are resolved by some arbitrary rule such as most votes above median. This voting system is called Majority Approval Voting.

@vendethiel
Collaborator

Well, that's what I call a detailed answer.
I'd like to see "~>", maybe coffee2 ;).

@epidemian
Contributor

Well, one benefit of => over ~> is that it aligns pretty well with the new JS arrow function syntax.

I think @tashemi brings a valid concern in that backcalls would involve a not very obvious CS -> JS conversion compared to other constructs, which may be at odds with the "it's just JS" spirit. That being said, the JS generated by the class construct is also quite contrived when involving inheritance and other stuff; and we're all fine with that (probably because the benefits outweigh the drawbacks... IDK).

For the moment, i'd prefer to see Coffee supporting yield generators; and using normal calbacks when coding for platforms that don't support it.

@00dani
00dani commented Aug 9, 2013

@epidemian

I think the backcall transformation is no more complex than what the do keyword does, and class is much more complicated than both of them.

Speaking of yield, though, what if we did something crazy and made:

func = ->
  y <- f x
  z <- g q
  y + z

compile to:

func = function*() {
  var y, z;
  y = yield f(x);
  z = yield g(q);
  return y + z;
}

This is almost certainly a bad idea, since it makes backcall use with RequireJS and Gruntfiles and so on totally impossible (along with amb, but honestly people probably weren't going to use amb in CoffeeScript enough to want backcalls for it) and has an inverted function arrow <- that doesn't even produce a function in the output. So it's a possibility but not one really worth considering too much.

Still, CS does really need to support yield and generators at some point, and I think using some sort of operator in place of the actual yield keyword would suit the language.

@jamesonquinn

From the discussion in another bug, here's how I'd expect a library to replace iced-coffeescript with backcalls to work:

Here's an example from the iced-coffeescript docs:

parallelSearch = (keywords, cb) ->
  out = []
  await 
    for k,i in keywords
      search k, defer out[i]
  cb out

I'd expect to write the "await" library so that the following would work as above:

parallelSearch = (keywords, cb) ->
  out =< await (defer) ->
    for k,i in keywords
      search k, defer i #defer effectively does: (i) -> return ((x) -> out[i] = x)
      #but it also keeps track and await only returns after all defer callbacks have fired.
  cb out #out is now an array of the same length as keywords

You could easily add extra options like defer.setGlobalTimeout(ms) which are impossible in iced.

@00dani
00dani commented Aug 9, 2013

@jamesonquinn

So your await library would work like the this.group() option provided by creationix/step, effectively? Reasonable, but:

  1. Why does each defer require an index? That isn't needed in Step's design. Why not just search k, defer()?
  2. There are other kinds of control flow you'd want to support which "real" await/defer can, and I don't think that your library as presented exposes functionality other than effectively that offered by async.parallel or Step's this.group(). You could have different basic functions in the vein of your await function for different kinds of control flow, but then you don't really have an await library, since such a library wouldn't have a single keyword await for generally any kind of "pausing" an asynchronous function.

Personally, I think encoding asynchronous operations in terms of their control flow at all is kind of a huge mess, compared to using value dependencies, i.e., promises. Note that for example the kriskowal/q version of parallelSearch is a mere:

parallelSearch = (keywords) -> Q.all keywords.map Q.nfbind search
# or if you need to maintain nodeback compat
parallelSearch = (keywords, cb) -> (Q.all keywords.map Q.nfbind search).nodeify cb

And that of course compiles to almost identical JS:

var parallelSearch;
parallelSearch = function(keywords, cb) {
  return Q.all(keywords.map(Q.nfbind(search))).nodeify(cb);
}

Considering how simple that is, I honestly can't really see the appeal of ICS-style await/defer, especially considering the shape of the compiled output. Still, people seem to like ICS's input, if not its output, so… (Also, await/defer doesn't handle or propagate errors, as far as I can tell?)

@jamesonquinn
  1. You're right, it could of course infer an index from order of creation of the callbacks. The above syntax is just an idea and would probably need refinement and/or flexibility.
  2. Right, you could have a larger library, and then call the bits of it lib.awaite, lib.yielde, lib.catche... and other async equivalents. Again, the above was just a sketch to show how libraries could be written to make backcalls even more useful.
@00dani
00dani commented Aug 10, 2013

Hmm, true, there're lots of possibilities for varied workflows and such.

On the subject of libraries, I think something like this is all that's really needed for promise backcalls to work pretty nicely:

p = (p, cb) -> p.then cb
# for example
readShoutyJsonFile = (path) ->
  data <-p Q.nfcall fs.readFile, path, 'utf8'
  JSON.parse data.toUpperCase()

In general, I can imagine a library of single-letter functions that implement various different "bind" operations, kind of like this.

module.exports = m = monad = {}
m.l = monad.list    = (lst, cb) -> [].concat lst.map(cb)...
m.p = monad.promise = (p, cb)   -> p.then cb
m.r = monad.reader  = (e) -> (f, cb) -> cb f e

Some of those are likely more useful than others. :P

@danschumann

What if we care about the scope of our callback? maintaining the same arrow syntax makes sense

user =
  username: 'franky beano callienmano'
  save: (done) ->
    console.log 'saving ', @username

    setTimeout (=> done.call this, 'taco argument'), 1000

app.get '/', (req, res, next) ->
  <= user.save
  @username == undefined

  <- user.save
  @username == 'franky beano callienmano'

  (save_success) <- user.save
  console.log 'save was tacos?', save_success

This syntax is beautiful and should be implemented in 2 ways:

1: some async library that we can use with existing coffeescript ( our compiled javascript will have tabs, oh well )
2: with yields and such.

+1

@danschumann

Also, what about other types of deferred? Are we expecting the callback to be the last argument in the function?

# consider
$.post('/url').sucess(callback1).error(callback2)

# OPTIONAL ( ERROR OR SUCCESS )
post_deferred = $.post('/url')
(success_args...) <~ post_deferred.success
(error_args...) <~ post_deferred.error
if success_args
  return 'cool'
else
   return 'doh'

Hey look at that, a pretty decent reason to use ~, optional callbacks. What if we have multiple callbacks like this? How do we break them up? We could use <~~ to start a chain of optional callbacks and <~ for each subsequent one.

  • 1 for options
@danschumann

Also, what about parallel?

I don't see the need for async anymore, the whole point is to make coffeescript good enough to do it on it's own, right?

(err{users}, users) <== users_collection.fetch()
(err{pages}, pages) <== pages_collection.fetch()
# now a single arrow to execute the parallel functions
(err{documents}, documents) <= documents_collection.fetch()
# Err could be {users: 'user error', pages: 'pages error', documents: 'documents error'}

There is no err{users} right now, because there is no parallel / separate declaration of a single value. == two arrows, means you're doing two things(at once), in parallel.

In the past my suggestions have been shutdown, probably due to the difficulty that would be in implementing them, so it could end up being that a new compiled js language emerges, inspired by the beauty that is coffeescript, while fulfilling my wildest dreams.

@00dani
00dani commented Oct 15, 2013

@danschumann

maintaining the same arrow syntax makes sense

We can't have <= mean a bound backcall, because <= already means "less than or equal to". This is why Coco switched to ~> and <~ for bound functions, since that introduces no such ambiguity.

Are we expecting the callback to be the last argument in the function?

Yes. In earlier iterations support for putting a "placeholder" elsewhere (to relocate the callback) was suggested, but eventually it was decided that simply wrapping the expression with a function would suffice. Look for the "hug operator".

Hey look at that, a pretty decent reason to use <~, optional callbacks. What if we have multiple callbacks like this?

By far the most common pattern for callbacks is to use a single callback with an (err, res) signature: a nodeback. Backcalls are designed to work with a sequential chain of async calls following that pattern. There's really no particular support for a call that requires multiple callbacks, because such calls don't correspond to the concept behind a backcall, that being that <- is nothing more than a magical async =:

x =  someSyncCall  arg, arg
y <- someAsyncCall arg, arg
f x, y

Most calls that require more than one callback just don't make sense if you view <- as magic =. There's one common case that takes multiple callbacks and also makes perfect sense when used with backcalls, however: promises, which can take a normal callback and an error-case callback. Fortunately, however, you can still use them with backcalls easily enough, since the backcalls simply provide the success codepath:

p = (pr) -> (f) -> pr.then f
backcallCode = ->
  x <-p somePromiseCall arg, arg
  y <-p someOtherPromiseCall arg
  x + y
directPromisesCode = ->
  somePromiseCall(arg, arg).then (x) ->
    someOtherPromiseCall(arg).then (y) ->
      x + y

Error is simply propagated past the backcall chain, invoking none of the success-path calls. This essentially corresponds to the way the promise monad's error-handling basis, the Either monad, works.

I don't see the need for async anymore, the whole point is to make coffeescript good enough to do it on it's own, right?

Actually, no. If we were trying to make CoffeeScript's asynchronous support complex/powerful enough to model all async patterns by itself, we would probably have jumped straight to IcedCoffeeScript. The problem is that nearly all asynchronous patterns other than simple serial will compile to JavaScript code that's a lot less pleasant: Running calls in parallel requires some kind of reference-counter to be declared and tracked, for example, and at the deep end we end up with the monstrosity that is CPS-transformed JavaScript.

Backcalls are a simple enough syntactic transformation that the compiled JavaScript is not particularly horrible. All they do is flatten callback pyramids. They do not and will not model all async patterns, because that need is much better served by a library such as caolan/async. Such libraries would be used in conjunction with backcalls:

(err, [one, two]) <- async.parallel [oneF, twoF]
codeUsingAsyncResults one and two
@xixixao
Contributor
xixixao commented Jan 22, 2014

-1 Let's not. Promises or generators are the way out of callback hell, otherwise I (did) would use IcedCoffeeScript.

@00dani
00dani commented Jan 22, 2014

@xixixao Absolutely they are, but backcalls a) work quite well with promises in conjunction with a helper like the p I defined earlier and b) also work for things other than the callback hell promises solve. They're superior to using IcedCoffeeScript specifically because the syntactic transformation is so trivial: Your compiled JS isn't going to become CPS-conversion hell when you use backcalls.

Generators are definitely a more flexible way to make promise code look synchronous, but they're still not available in every browser, nor do they exist in CoffeeScript yet either. Meanwhile, generators are not going to help you at all with cases like:

export = (x) -> module.exports = x
grunt <- export
grunt.loadNpmTasks 'whatever'
grunt.allYourGruntStuffHere withNoIndent

Or:

$, _ <- define ['jquery', 'underscore']
requireJS.module goesHere withNoIndent

Or even:

result = do ->
  x <- amb [1..10]
  y <- amb [1..30]
  fail() if x * y < 10
  x*2 + y*2
@kibin
kibin commented Apr 15, 2014

Is this still valid? Does somebody investigate/develop them?

@zhaizhai

A while ago I prototyped a basic implementation:

https://github.com/zhaizhai/coffee-script/tree/backcall

In particular, look at

https://github.com/zhaizhai/coffee-script/blob/backcall/test/backcall.coffee

In my opinion it ended up being weird to not be able to write if statements and for loops normally with "backcalls" inside, so you would really need to compile the appropriate CPS transformations on those constructs to make things feel natural. At that point maybe you should just use IcedCoffeeScript.

If you're interested the code linked above should still work, but presumably it's a ways behind the main branch by now. It seems like people are not that interested in this feature anymore. It's an interesting discussion topic and attempts to address a real problem, but nobody (including me) actually tries seriously to get it done 😛 .

@kibin
kibin commented Apr 15, 2014

I see, thanks!

@jashkenas
Owner

In my opinion it ended up being weird to not be able to write if statements and for loops normally with "backcalls" inside, so you would really need to compile the appropriate CPS transformations on those constructs to make things feel natural. At that point maybe you should just use IcedCoffeeScript.

That's really useful feedback on the feature. Thanks for sharing it!

@jashkenas jashkenas added the wontfix label Apr 15, 2014
@jashkenas jashkenas closed this Apr 15, 2014
@vendethiel
Collaborator

I don't get how you can't use if and stuff in a backcall ? You're saying the backcall doesn't work like await. We're aware of that, that's not what we're proposing here.

EDIT : We're not proposing an alternative to iced's async, but something much, much more general.

@jashkenas jashkenas reopened this Apr 15, 2014
@jamesonquinn

I think the thing with using if statements is talking about a situation like the following (using =< for the backcall operator, as discussed above):

  if is_async get_x_using
    x =< get_x_using y
  else
    x = get_x_using y
  ....

It's true that this makes things hairy for the compiler. Of course, the whole if/else block above could be refactored into a backcallable function; but zhaizhai is saying that, upon playing with it, that seems artificial (if the programmer does it) or ugly (if the compiler does).

I'm not sure what the right answer is, but it's probably worth leaving this one open for a little longer to see if somebody comes up with a good solution for this issue. For instance, I suspect you could make a "hugs"-like idiom for a workaround....

@vendethiel
Collaborator

But -- as I said -- we don't want <- (or =<) to be async-like. Just a callback that fits better.

@danschumann

Looking back on this again it just looks confusing. Won't this be sort of moot when generators and yield become the norm?

@vendethiel
Collaborator

co rather than generators

@mmotorny
Contributor

I skimmed through the thread but didn't find a convincing example why monadic notation is useful, so I decided to compile my own. Please review: https://docs.google.com/document/d/1pmvd6Gd-Scj06dB6uLunU52eNpMCjAqM9t6HobOJ758/edit#

@QxQ
QxQ commented Jan 24, 2015

Just a thought an alternate syntax for this, maybe this would be easier for people to understand?

What if we did "->..." meaning "make a function, but the indentation for it is on the same line", so...

callme param, (arg) ->...
codeHere()

is equivalent to

callme param, (arg) ->
    codeHere()

Also, for the problem with the function parameter in a funny place, it could be solved like this:

setTimeout ->..., 100
codeHere()

is equivalent to

setTimeout ->
    codeHere()
,100
@taoeffect

I just saw that ES7 will have async and await keywords. So what does that imply for coffeescript?

Should I open another issue for async and await support in CS or is this issue sufficient? cc @jashkenas

@jashkenas
Owner

Feel free to open another issue that specifically talks about what adding the ES(7) keywords would look like — and how we might compile them.

I don't think it's something that's possible without large-scale runtime support, no?

@aurium
aurium commented Mar 4, 2015

+1 to @QxQ syntax ->...

@Baudin999

I appologize for the long comment, please forgive me; but; I really think that a backcall operator obfuscates the code, imagine this:

result1 <- foo a
result2 <- foo b

Now, does this traslate into:

function foo(a, function (result1) { /* do something */ }
function bar(b, function (result2) { /* do something */ }

or:

function foo(a, function(result1) { 
    function bar(b, function(result2) { /* do something */ })
})

with the proposed syntax you do not know whether or not the second function should be a part of the inner-function of the first or if it should run in parallel.

I would rather wrap my code in a yield statement like so:

result1 = yield foo a
result2 = yield foo b

this would resolve into the first option. For the second option you would get:

await result1 = yield foo a
result2 = yield foo b

This results into much cleaner code. Now resolving a promise becomes almost trivial:

promise = do -> foo a
await result = yield promise.done
# do something with the result
console.log result

Now for the problem of the order of the parameters and the callback being somewhere in the middle, combined with mutiple parameters within the callback:

r1, r2 = do -> foo a, yield, b
# you could rewrite the first example to:
result1 = foo a, yield
# but what would be the point?

Here you sacrifice readability for distance to code.

@00dani
00dani commented Mar 11, 2015

@Baudin999 Backcalls always desugar into the second of your proposed syntaxes, i.e.,

a <- thingOne
b <- thingTwo
moreCode
// *always* means
thingOne(function(a) {
  thingTwo(function(b) {
    moreCode;
  });
)};

This is consistent with the behaviour of backcalls in Coco and LiveScript, Haskell's do notation, and the intuitive sense that a backcall is a "magical async = operator". If you want parallel operations, you'll still need a "normal" async helper library. For example:

{items, otherStuff} <- async.parallel items: getItems, otherStuff: getOtherStuff
# or with promises and a 'p' helper as discussed earlier
results <-p Q.all [firstThing(), secondThing(), thirdThing()]

Allowing for distinct yield and await keywords would be more powerful but also vastly complicates the compiled JavaScript - take a look at IcedCoffeeScript, which provides those exact features, for instance.

@maxtaco
Contributor
maxtaco commented Jun 12, 2015

FWIW, I'm contemplating switching IcedCoffeeScript over to a generator-based transpilation, which is of course much simpler. Here's a hand-compiled example of how it can work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment