do blocks, anonymous functions, and control flow #1288

Closed
StefanKarpinski opened this Issue Sep 18, 2012 · 26 comments

Comments

Projects
None yet
7 participants
@StefanKarpinski
Owner

StefanKarpinski commented Sep 18, 2012

Now that we have the do-block syntax, one is tempted to write things like this:

function f(dir,file)
  cd(dir) do
    return readall(file)
  end
  error("didn't expect to get here")
end

Of course, since the do-block syntax is just shorthand for passing an anonymous function to cd, this throws an error, rather than returning the contents of "$dir/$file". I would like to keep do-blocks as just a syntactic sugar over anonymous functions. Is there some way we can make these things work more conveniently, or is that just going to cause complications? It's also convenient to use do-blocks for loop-like constructs, at which point one often wants features like break and continue, which raises similar issues.

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Sep 18, 2012

Owner

Another more subtle issue that I just encountered is this:

reqs = parse_requires("REQUIRES")
Git.each_submodule(false) do pkg, path, sha1
  reqs = [reqs,parse_requires("$path/REQUIRES")]
end

There's more context, of course, and it occurs in a function body, but basically, this doesn't work because the inner assignment to reqs causes it to be a local variable which then isn't defined on the first pass through.

Owner

StefanKarpinski commented Sep 18, 2012

Another more subtle issue that I just encountered is this:

reqs = parse_requires("REQUIRES")
Git.each_submodule(false) do pkg, path, sha1
  reqs = [reqs,parse_requires("$path/REQUIRES")]
end

There's more context, of course, and it occurs in a function body, but basically, this doesn't work because the inner assignment to reqs causes it to be a local variable which then isn't defined on the first pass through.

@JeffBezanson

This comment has been minimized.

Show comment Hide comment
@JeffBezanson

JeffBezanson Sep 18, 2012

Owner

As for the first problem: ha! have fun implementing that.

The second problem I don't believe, because variable bindings are inherited by default. I get the following:

bar(f) = f(0)

function qux()
        x = 1
       bar() do y
        x = [x,y]
       end
       x
     end

qux()
2-element Int64 Array:
 1
 0
Owner

JeffBezanson commented Sep 18, 2012

As for the first problem: ha! have fun implementing that.

The second problem I don't believe, because variable bindings are inherited by default. I get the following:

bar(f) = f(0)

function qux()
        x = 1
       bar() do y
        x = [x,y]
       end
       x
     end

qux()
2-element Int64 Array:
 1
 0
@JeffBezanson

This comment has been minimized.

Show comment Hide comment
@JeffBezanson

JeffBezanson Sep 18, 2012

Owner

I honestly don't understand why we suddenly need these do blocks all over the place. Now you have the option of handling control flow in either the caller or the callee, and it's not clear which one to use. If we want to change control flow to be closure-based, we need to take a good 3 months to redesign a lot of things.

Owner

JeffBezanson commented Sep 18, 2012

I honestly don't understand why we suddenly need these do blocks all over the place. Now you have the option of handling control flow in either the caller or the callee, and it's not clear which one to use. If we want to change control flow to be closure-based, we need to take a good 3 months to redesign a lot of things.

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Sep 18, 2012

Owner

Have fun implementing what? I wasn't proposing a particular behavior, just pointing out that this is a potential point of confusion that we should consider. The second one I observed, but apparently it only happens in the repl, not inside a function as I had claimed (the repl is where I observed it, but the code example was in a function body):

julia> x = 1
1

julia> bar() do y
         x = [x,y]
       end
in anonymous: x not defined
 in anonymous at no file:2
 in bar at none:1
Owner

StefanKarpinski commented Sep 18, 2012

Have fun implementing what? I wasn't proposing a particular behavior, just pointing out that this is a potential point of confusion that we should consider. The second one I observed, but apparently it only happens in the repl, not inside a function as I had claimed (the repl is where I observed it, but the code example was in a function body):

julia> x = 1
1

julia> bar() do y
         x = [x,y]
       end
in anonymous: x not defined
 in anonymous at no file:2
 in bar at none:1
@JeffBezanson

This comment has been minimized.

Show comment Hide comment
@JeffBezanson

JeffBezanson Sep 18, 2012

Owner

This is the same behavior we've had forever, where globals aren't overwritten inside functions by default. So perhaps do blocks are not just syntactic sugar for functions. In hindsight I can see I should not have added them so hastily. This is a good example of everything starting to fall apart once you start throwing out new features too quickly.

Owner

JeffBezanson commented Sep 18, 2012

This is the same behavior we've had forever, where globals aren't overwritten inside functions by default. So perhaps do blocks are not just syntactic sugar for functions. In hindsight I can see I should not have added them so hastily. This is a good example of everything starting to fall apart once you start throwing out new features too quickly.

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Sep 18, 2012

Owner

The point of opening the issue was to discuss problems raised by do-block syntax. Honestly, I'm not entirely in love with it. Maybe it's a failed experiment, but if that's the case then we need another way to handle things like opening files with a guarantee that they'll get closed again. The fact that do-block syntax raises these issues — which it does in Ruby as well — may be a design smell. If so, let's figure out a better way to do it.

There was the idea for regex matching of using a for loop, as in:

for m = match(r"^(\w+)=(.*)$", line)
  # handle match, doesn't execute if no match
end

If there were else blocks on for loops, then you can also handle non-matching smoothly:

for m = match(r"^(\w+)=(.*)$", line)
  # handle match, doesn't execute if no match
else
  # handle not matching
end

I like this approach a lot for regex matching. It doesn't rely on a closure, inlines very nicely, and the else version relies on a generally useful feature that we wouldn't mind adding anyway. This approach, however, doesn't obviously help in the case of things like open or cd where you want to have guaranteed exit behavior in the case of errors.

For what it's worth, I experimented in the package code with both using do-blocks and using coroutines (Task objects) to do complex iterations and I much prefer using task objects with a for loop. Sure, there's some overhead in switching tasks, but then again, there's overhead in using closures. The coroutines have the clear advantage of looking like control flow and actually just being control flow (so return, break, and continue are non-issues and work as expected).

Owner

StefanKarpinski commented Sep 18, 2012

The point of opening the issue was to discuss problems raised by do-block syntax. Honestly, I'm not entirely in love with it. Maybe it's a failed experiment, but if that's the case then we need another way to handle things like opening files with a guarantee that they'll get closed again. The fact that do-block syntax raises these issues — which it does in Ruby as well — may be a design smell. If so, let's figure out a better way to do it.

There was the idea for regex matching of using a for loop, as in:

for m = match(r"^(\w+)=(.*)$", line)
  # handle match, doesn't execute if no match
end

If there were else blocks on for loops, then you can also handle non-matching smoothly:

for m = match(r"^(\w+)=(.*)$", line)
  # handle match, doesn't execute if no match
else
  # handle not matching
end

I like this approach a lot for regex matching. It doesn't rely on a closure, inlines very nicely, and the else version relies on a generally useful feature that we wouldn't mind adding anyway. This approach, however, doesn't obviously help in the case of things like open or cd where you want to have guaranteed exit behavior in the case of errors.

For what it's worth, I experimented in the package code with both using do-blocks and using coroutines (Task objects) to do complex iterations and I much prefer using task objects with a for loop. Sure, there's some overhead in switching tasks, but then again, there's overhead in using closures. The coroutines have the clear advantage of looking like control flow and actually just being control flow (so return, break, and continue are non-issues and work as expected).

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Sep 18, 2012

Owner

A couple of ideas...

Have something like ensure as in: f = open("file") ensure close(f).

Add onexit(itr) to the iteration protocol and ensure that it is called no matter how a loop terminates, which would allow this as a file open or cd idiom:

for f = open("file")
  # use open file handle f
end

for d = cd("dir")
  # do something in directory
end

This kind of thing would arguably make Julia a for-loop-oriented language (which might not be a bad thing).

Owner

StefanKarpinski commented Sep 18, 2012

A couple of ideas...

Have something like ensure as in: f = open("file") ensure close(f).

Add onexit(itr) to the iteration protocol and ensure that it is called no matter how a loop terminates, which would allow this as a file open or cd idiom:

for f = open("file")
  # use open file handle f
end

for d = cd("dir")
  # do something in directory
end

This kind of thing would arguably make Julia a for-loop-oriented language (which might not be a bad thing).

@JeffBezanson

This comment has been minimized.

Show comment Hide comment
@JeffBezanson

JeffBezanson Sep 18, 2012

Owner

These are good points. I do like for/else and while/else. They also play nice with henchmen unrolling, since both require copying the condition. You can simply make the first copy of the branch point to the else block instead of the end of the loop.

Owner

JeffBezanson commented Sep 18, 2012

These are good points. I do like for/else and while/else. They also play nice with henchmen unrolling, since both require copying the condition. You can simply make the first copy of the branch point to the else block instead of the end of the loop.

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Sep 18, 2012

Owner

So one obvious takeaway is that we ought to have for-else and while-else: #1289.

Owner

StefanKarpinski commented Sep 18, 2012

So one obvious takeaway is that we ought to have for-else and while-else: #1289.

@nolta

This comment has been minimized.

Show comment Hide comment
@nolta

nolta Sep 18, 2012

Member

Have something like ensure as in: f = open("file") ensure close(f).

Sort of like go's defer statement:

http://blog.golang.org/2010/08/defer-panic-and-recover.html

Add onexit(itr) to the iteration protocol and ensure that it is called no matter how a loop terminates, ...

Sort of like python's with statement:

with open(...) as f:
    f.write(...)

which translates roughly as:

f = open(...)
f.__enter__()
try:
    f.write(...)
except e:
    f.__exit__(e)
Member

nolta commented Sep 18, 2012

Have something like ensure as in: f = open("file") ensure close(f).

Sort of like go's defer statement:

http://blog.golang.org/2010/08/defer-panic-and-recover.html

Add onexit(itr) to the iteration protocol and ensure that it is called no matter how a loop terminates, ...

Sort of like python's with statement:

with open(...) as f:
    f.write(...)

which translates roughly as:

f = open(...)
f.__enter__()
try:
    f.write(...)
except e:
    f.__exit__(e)
@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Sep 18, 2012

Owner

Yes, both are very apt comparisons. More familiar names would, of course, be welcomed (although I'm not sure Go's choice of defer can be considered standard). One argument for having an onexit function as part of the iteration protocol is things like each_line which, when applied to some objects really ought to ensure that a file handle gets closed whenever the loop exits.

Owner

StefanKarpinski commented Sep 18, 2012

Yes, both are very apt comparisons. More familiar names would, of course, be welcomed (although I'm not sure Go's choice of defer can be considered standard). One argument for having an onexit function as part of the iteration protocol is things like each_line which, when applied to some objects really ought to ensure that a file handle gets closed whenever the loop exits.

@JeffBezanson

This comment has been minimized.

Show comment Hide comment
@JeffBezanson

JeffBezanson Sep 18, 2012

Owner

I realize that I like the do syntax when it only needs to be sugar for passing a function to a higher-order function, and doesn't need any magic control flow behavior. The only real problem is that its syntax obscures what is going on, so that somebody might reasonably expect return to return from the outer function. Explicit parentheses and lambdas do have advantages you know :) Maybe a more revealing syntax would be something like

f(x) with function (y,z)
  ...
end

We could also consider adding named blocks so you can return from any block at any point, including nested loops etc.

Owner

JeffBezanson commented Sep 18, 2012

I realize that I like the do syntax when it only needs to be sugar for passing a function to a higher-order function, and doesn't need any magic control flow behavior. The only real problem is that its syntax obscures what is going on, so that somebody might reasonably expect return to return from the outer function. Explicit parentheses and lambdas do have advantages you know :) Maybe a more revealing syntax would be something like

f(x) with function (y,z)
  ...
end

We could also consider adding named blocks so you can return from any block at any point, including nested loops etc.

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Sep 18, 2012

Owner

The trouble with that is that the whole point of the do block syntax is that it obscures what's really going on. It almost seems like we want a here doc syntax for function arguments...

Owner

StefanKarpinski commented Sep 18, 2012

The trouble with that is that the whole point of the do block syntax is that it obscures what's really going on. It almost seems like we want a here doc syntax for function arguments...

@JeffBezanson

This comment has been minimized.

Show comment Hide comment
@JeffBezanson

JeffBezanson Sep 18, 2012

Owner

I'm not sure what that syntax proposal means without an example.

Obscuring what's going on doesn't seem to work well, since it makes it hard to predict what happens in edge cases like return, break, and continue without running to the manual.

Owner

JeffBezanson commented Sep 18, 2012

I'm not sure what that syntax proposal means without an example.

Obscuring what's going on doesn't seem to work well, since it makes it hard to predict what happens in edge cases like return, break, and continue without running to the manual.

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Sep 19, 2012

Owner

It wasn't even a proposal, just saying that what you're describing with named blocks sounds like here documents.

Owner

StefanKarpinski commented Sep 19, 2012

It wasn't even a proposal, just saying that what you're describing with named blocks sounds like here documents.

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Sep 19, 2012

Owner

I'm into the idea of a with-like syntax. Of course, in Julia, there would be something like withok and witherr generic functions that you would define methods for and then writing something like

with f = open("file")
  # do stuff with file handle f
end

would get expanded into something like this:

let f = open("file")
  try
    # do stuff with file handle f
  catch err
    witherr(f,err)
  end
  withok(f)
end

The default witherr would call withok and then rethrow the error. My main objection to this whole construct is that it almost wants to be combined with something else — possibly the for loop as I suggested above.

It would also just be generally useful to have something like Go's defer mechanism or D's scope guards. For open/close, those are easy enough:

f = open("file")
defer close(f)
# do stuff with file handle f

But what about changing directories? The code for that is significantly more complicated since the robust implementation opens the current directory, keeps a file handle to it, and later uses fchdir to get back to it by file handle rather than by name (which could have changed or disappeared in the mean time). That's the kind of stuff you really want to hide away in a with-like mechanism, rather than letting the programmer try to write it themselves.

Owner

StefanKarpinski commented Sep 19, 2012

I'm into the idea of a with-like syntax. Of course, in Julia, there would be something like withok and witherr generic functions that you would define methods for and then writing something like

with f = open("file")
  # do stuff with file handle f
end

would get expanded into something like this:

let f = open("file")
  try
    # do stuff with file handle f
  catch err
    witherr(f,err)
  end
  withok(f)
end

The default witherr would call withok and then rethrow the error. My main objection to this whole construct is that it almost wants to be combined with something else — possibly the for loop as I suggested above.

It would also just be generally useful to have something like Go's defer mechanism or D's scope guards. For open/close, those are easy enough:

f = open("file")
defer close(f)
# do stuff with file handle f

But what about changing directories? The code for that is significantly more complicated since the robust implementation opens the current directory, keeps a file handle to it, and later uses fchdir to get back to it by file handle rather than by name (which could have changed or disappeared in the mean time). That's the kind of stuff you really want to hide away in a with-like mechanism, rather than letting the programmer try to write it themselves.

@jsarnoff

This comment has been minimized.

Show comment Hide comment
@jsarnoff

jsarnoff Sep 20, 2012

From a "logical construction of software" perspective, having guards ... for/else while/else until/else are guarded block expressions of a kind ... makes designing bad path avoidance and direction good program flow easier. The more capable, concise, consistent and complete guard is enactable as gatekeeper and as guide. Software development is more efficient where tangle avoidance is given consistent expression.

And it is just fine to enfold the capability, as occurs some with be-iteratively/else and do-currently/later.

From a "logical construction of software" perspective, having guards ... for/else while/else until/else are guarded block expressions of a kind ... makes designing bad path avoidance and direction good program flow easier. The more capable, concise, consistent and complete guard is enactable as gatekeeper and as guide. Software development is more efficient where tangle avoidance is given consistent expression.

And it is just fine to enfold the capability, as occurs some with be-iteratively/else and do-currently/later.

@jsarnoff

This comment has been minimized.

Show comment Hide comment
@jsarnoff

jsarnoff Sep 20, 2012

FYI the clear sense from your discussion is that Julia should ditch the do block and better solve a simpler (not less powerful, less complex and more enabling) intent.

FYI the clear sense from your discussion is that Julia should ditch the do block and better solve a simpler (not less powerful, less complex and more enabling) intent.

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Sep 20, 2012

Owner

Yes, the basic issues with do blocks are:

  1. They look like control flow but aren't;
  2. They are anonymous functions but don't look like it.

Both of these mismatches cause problems.

Owner

StefanKarpinski commented Sep 20, 2012

Yes, the basic issues with do blocks are:

  1. They look like control flow but aren't;
  2. They are anonymous functions but don't look like it.

Both of these mismatches cause problems.

@o-jasper

This comment has been minimized.

Show comment Hide comment
@o-jasper

o-jasper Oct 4, 2012

Member

I use this:

macro with(setting, body)
  w_var = isa(setting,Expr) && is(setting.head,symbol("="))
 #Make a variable if none given.
  setting = w_var ? setting : :($(gensym()) = $setting) 
  assert(length(setting.args)==2, "Cannot set more than one thing at a time.
 (incorrect number of arguments in Expr of setting; $(length(setting.args)))")
  ret= gensym()
  return esc(quote 
              local $setting
              $ret = $body
              no_longer_with($(setting.args[1]), $ret)
             end)
end
#If ret not input argument, defaults to return it.
function no_longer_with{T,With}(with::With, ret::T) 
  no_longer_with(with)
  return ret
end

no_longer_with(stream::IOStream) = close(stream) #!

It is in this file but i haven't pushed it yet.. I have also changed @with_primitive for openGL to use it, you can basically make anything work with it by defining no_longer_with.

readall(file::String) = @with stream = open(file,"r") readall(stream)

Note this has an abstraction leak, it won't call no_longer_with if you return or throw(or exit somehow) in the middle of it. In common lisp i would use unwind-protect to help against that, but not sure how that is implemented.

@with could do cd too, if cd returns something no_longer_with can work with. Alternatively symbol-pairs can be kept that link a function with their @with version.(It does set a global(nonspecial) var though?)

Maybe everything that can be done with a macro, should be? The dropped @ and (implicit)use of else, elseif and implicit begins could signify that it is a 'standard' one, maybe. Looking at it that way, maybe do has a little arbitary reversal of where the macro name is.

Member

o-jasper commented Oct 4, 2012

I use this:

macro with(setting, body)
  w_var = isa(setting,Expr) && is(setting.head,symbol("="))
 #Make a variable if none given.
  setting = w_var ? setting : :($(gensym()) = $setting) 
  assert(length(setting.args)==2, "Cannot set more than one thing at a time.
 (incorrect number of arguments in Expr of setting; $(length(setting.args)))")
  ret= gensym()
  return esc(quote 
              local $setting
              $ret = $body
              no_longer_with($(setting.args[1]), $ret)
             end)
end
#If ret not input argument, defaults to return it.
function no_longer_with{T,With}(with::With, ret::T) 
  no_longer_with(with)
  return ret
end

no_longer_with(stream::IOStream) = close(stream) #!

It is in this file but i haven't pushed it yet.. I have also changed @with_primitive for openGL to use it, you can basically make anything work with it by defining no_longer_with.

readall(file::String) = @with stream = open(file,"r") readall(stream)

Note this has an abstraction leak, it won't call no_longer_with if you return or throw(or exit somehow) in the middle of it. In common lisp i would use unwind-protect to help against that, but not sure how that is implemented.

@with could do cd too, if cd returns something no_longer_with can work with. Alternatively symbol-pairs can be kept that link a function with their @with version.(It does set a global(nonspecial) var though?)

Maybe everything that can be done with a macro, should be? The dropped @ and (implicit)use of else, elseif and implicit begins could signify that it is a 'standard' one, maybe. Looking at it that way, maybe do has a little arbitary reversal of where the macro name is.

@timholy

This comment has been minimized.

Show comment Hide comment
@timholy

timholy Jan 9, 2013

Owner

If we're re-entering a chaos period, can I nominate this issue as particularly in need of a decision?

Owner

timholy commented Jan 9, 2013

If we're re-entering a chaos period, can I nominate this issue as particularly in need of a decision?

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Jan 9, 2013

Owner

Agreed.

Owner

StefanKarpinski commented Jan 9, 2013

Agreed.

@vtjnash

This comment has been minimized.

Show comment Hide comment
@vtjnash

vtjnash Feb 1, 2013

Member

related: #441

Member

vtjnash commented Feb 1, 2013

related: #441

@JeffBezanson

This comment has been minimized.

Show comment Hide comment
@JeffBezanson

JeffBezanson Feb 5, 2013

Owner

Decision: we will keep it as-is. The reason is that often you want return to return from the inner block, for example in a callback that hangs around for a long time. In such a case, the enclosing function has returned already and cannot be returned from anyway. In the future, I'd like to add the ability to return from any specified block, at which point you'd be able to get the other behavior, so we might as well keep things as they are for now.

Owner

JeffBezanson commented Feb 5, 2013

Decision: we will keep it as-is. The reason is that often you want return to return from the inner block, for example in a callback that hangs around for a long time. In such a case, the enclosing function has returned already and cannot be returned from anyway. In the future, I'd like to add the ability to return from any specified block, at which point you'd be able to get the other behavior, so we might as well keep things as they are for now.

@StefanKarpinski

This comment has been minimized.

Show comment Hide comment
@StefanKarpinski

StefanKarpinski Feb 5, 2013

Owner

The other part of this is that we recommend against using do blocks for iteration constructs, in favor of expressing all iteration using for loops. If you don't care too much about performance, you can use a task and get complex iteration behavior; if you do care about performance, you need to introduce an object that supports our start/done/next iteration protocol.

Another thought was that it might be more suggestive of the fact that the do block body is an anonymous function if we required parens around the block arguments:

listen("localhost", 2000) do (sock,status)
  # do something with the socket
end
Owner

StefanKarpinski commented Feb 5, 2013

The other part of this is that we recommend against using do blocks for iteration constructs, in favor of expressing all iteration using for loops. If you don't care too much about performance, you can use a task and get complex iteration behavior; if you do care about performance, you need to introduce an object that supports our start/done/next iteration protocol.

Another thought was that it might be more suggestive of the fact that the do block body is an anonymous function if we required parens around the block arguments:

listen("localhost", 2000) do (sock,status)
  # do something with the socket
end
@vtjnash

This comment has been minimized.

Show comment Hide comment
@vtjnash

vtjnash Aug 4, 2016

Member

while playing around with some other changes, I came across this syntax as a pretty close approximation to the do-else form (using match as an example):

if match(r"egex", "egex") do m
    println(m)
    return :something
end === nothing
    println(nothing)
end
function Base.match(f, re, str)
     m = match(re, str)
     m !== nothing && return f(m)
     nothing
end
Member

vtjnash commented Aug 4, 2016

while playing around with some other changes, I came across this syntax as a pretty close approximation to the do-else form (using match as an example):

if match(r"egex", "egex") do m
    println(m)
    return :something
end === nothing
    println(nothing)
end
function Base.match(f, re, str)
     m = match(re, str)
     m !== nothing && return f(m)
     nothing
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment