proposal: Go 2: allow defer to have its own scope

[vincent-163]

EDIT: Added a solution to the problem. Removed the arguments taken from other issues since the deferblock keyword doesn't change these code.

I know this has been proposed many times but I really do find it extremely inconvenient to use defer in the current way. Even if we can't change the current functionality of defer I would suggest that another keyword be introduced with similar functionality.

The nature of defer where all functions are forced to run just before the function returns seems counter-intuitive to me since there are some cases where I would expect the deferred function to run when the resource goes out of to scope instead of when the function returns.

Here are some cases where I find it hard to write code:

1. Using multiple resources one by one.
   Pseudo code:

lock1.Lock()
if err := operation1(); err != nil {
  lock1.Unlock()
  return err
}
if err := operation2(); err != nil {
  lock1.Unlock()
  return err
}
lock1.Unlock()

lock2.Lock()
if err := operation3(); err != nil {
  lock2.Unlock()
  return err
}
if err := operation4(); err != nil {
  lock2.Unlock()
  return err
}
lock2.Unlock()

The lock1 gets used first, then lock2. If I used defer in this case, I would have to lock lock2 without unlocking lock1, causing potential deadlock and performance issues. In some cases, I want to perform some expensive operations after obtaining the results from operation2(), but if the operation is performed with the lock held it can hurt performance.

2. Using defer in a for block.
   This is similar to the first issue, except that the resources are wrapped in a for loop. Pseudocode:

for i := 0; i < 10; i++ {
  locks[i].Lock()
  performOperation(i)
  locks[i].Unlock()
}

Without defer Unlock() will not be called in case of a panic(), causing deadlocks. Also all the locks from 0 to 8 would be held while performing operation 9, which will certainly cause deadlocks.

A workaround for the issue is to wrap the defer statement in a closure, like this:

for i := 0; i < 10; i++ {
  func() {
    locks[i].Lock()
    defer locks[i].Unlock()
    performOperation(i)
  }()
}

The problem with this approach is that the closure creates a completely different scope. If we want to break out of the for loop from the closure or return from the function, it would not be possible without some extra code that checks for breaks or returns in the closure. This hurts simplicity.

Intuitively the defer is just like any other blocks, such as for and switch. The deferred function gets called as soon as the program leaves the scope.

I propose a special keyword, called deferblock or something else, that expects to be followed by a block of code, like:

deferblock {
  lock.Lock()
  defer lock.Unlock()
  // perform operations with lock
}

defer always appends a deferred function to the last deferblock rather than an arbitrarily chosen deferblock, since defer functions have to be arranged like a stack and it's unnatural to "insert" to the stack.

deferblock can be break-ed out, just like for, switch and select. When program leaves the deferblock in any way, by normal control flow, break or panic, the deferred functions associated with the scope gets called, and if there's a panic and one of the deferred functions recovers from the panic, the program continues from the point after deferblock.

Such deferblock is consistent with what Go already has, so programmers don't have to rethink about how to use defer.

The new functionality missing from the language is to break out of a deferblock to an outer label, which cannot be easily implemented by using a closure.

[as]

The lock1 gets used first, then lock2. If I used defer in this case, I would have to lock lock2 without unlocking lock1, causing potential deadlock and performance issues.

Why does lock1 and lock2 have dependencies on each other? The problem here seems to be a result of misusing sync.

For the other case, how does breaking the function into smaller ones not solve the problem? Theoretical performance impacts are inapplicable for solving problems on a scale measured in nanoseconds.

[vincent-163]

Why does lock1 and lock2 have dependencies on each other? The problem here seems to be a result of misusing sync.

The idea is that the two locks don't have dependencies, but since defer defers the function call until return, it's not possible to defer Unlock without wrapping it in a closure, which suffers from the other problem (cannot break out of the closure).

For the other case, how does breaking the function into smaller ones not solve the problem? Theoretical performance impacts are inapplicable for solving problems on a scale measured in nanoseconds.

The impact is not in terms of performance but in terms of writing clean code. Wrapping it in a closure forces me to write additional code to handle breaks when break should have done it.

[networkimprov]

I would use a scope-limited defer.

Combining another keyword or symbol with defer avoids a backwards-incompatible new keyword:

defer{}
defer case
defer range
defer for

You don't need a new block definition method; you can write:

func f() {
   ...
   {
      door.Lock()
      defer{} door.Unlock()
      ...
   }
}

[DeedleFake]

Or just allow defer to be given a label much like break and continue can be:

lock:
  for {
    m.Lock()
    defer lock, m.Unlock()
  }

Not too keen on the syntax, but it allows you to select a block that way. The question in this case becomes when exactly the defer is run. Does it run when the loop loops or when the loop exits? I can see arguments in either direction.

The more I think about this, the more I think that it's best to just keep the simplicity of the current version, even if it can lead to some occasional awkwardness.

[vincent-163]

Combining another keyword or symbol with defer avoids a backwards-incompatible new keyword:

Not too keen on the syntax, but it allows you to select a block that way. The question in this case becomes when exactly the defer is run. Does it run when the loop loops or when the loop exits? I can see arguments in either direction.

Thanks for your interest in this proposal.

A highlight in this proposal is that there is no change to the defer syntax; it always puts the deferred function to the direct parent block, and the deferred functions are always called when the code leaves the block in any way. It's just that it's no longer limited to function scope, and people can define their own blocks. The reason, as I've already covered in the first post, is that defer is always a stack and it's unnatural to "insert" to a stack.

I believe this would keep simplicity while encouraging good coding practices. Without a block to contain the defer-ed functions, people would be encouraged to defer everything until function return. Such coding practice causes subtle problems since defer is usually used to free unused resources, so the functions can be called arbitrarily late. People would tend to write:

lock1.Lock()
defer lock1.Unlock()
// operation 1 with lock 1 held

lock2.Lock()
defer lock2.Unlock()
// operation 2 with lock 2 held and lock 1 accidentally held

instead of:

func() {
  lock1.Lock()
  defer lock1.Unlock()
  // operation 1 with lock 1 held
}()

func() {
  lock2.Lock()
  defer lock2.Unlock()
  // operation 2 with lock 2 held
}()

The first piece of code doesn't cause problems until people run into race conditions that are much harder to locate.

By introducing a dedicated keyword for defer blocks, it encourages people to clean up resources as soon as they are out of scope and additionally solves the problem with break-ing out of the closure.

[ianlancetaylor]

I note that you don't actually need a new keyword. Since defer can not be followed by a block, it would suffice to write

defer {
    lock.Lock()
    defer lock.Unlock()
    ...
}

That is, defer followed by a block would mean that any defer statements within the block would execute when the code leaves the block. I don't know if I like this idea, but it doesn't seem to require a new keyword.

[DeedleFake]

I don't think that would work out well in practice. There would likely be too many times that you would need to do a normal function-scoped defer inside of a defer block. It could also complicate things because you'd have to know if you were inside a defer block or not in order to know what a defer was actually going to do, which could complicate reading other people's code quite a bit. It makes it sound a bit like try {} finally {}, which is something that I, at least, would certainly like to avoid.

In general, I'm not a fan of special blocks of code that change the way syntactically unrelated lines inside the blocks work. It tends towards some annoying to read and maintain code.

[deanveloper]

The new functionality missing from the language is to break out of a deferblock to an outer label, which cannot be easily implemented by using a closure.

What would this even mean? This seems to be the only advantage of deferblock { ... } over defer func() { ... }().

defer functions are called before a function is exiting, so would breaking out of a defer mean that the function just "stops exiting"? It seems to be a confusing concept. An example where breaking out of a deferblock to an outer label, resulting in clean code, would be very useful.

Until then, this proposal really just seems like syntactic sugar over defer func() { ... }()

[vincent-163]

What would this even mean? This seems to be the only advantage of deferblock { ... } over defer func() { ... }().

deferblock is not meant to replace defer func(). It surrounds a block of code to call defer methods earlier, so they are called when the program leaves deferblock instead of before function exit.

An example is:

func f() {
  for i := 0; i < 1000; i++ {
    deferblock {
      file := open(i)
      defer file.Close()
      // work with the file
    }
  }
}

The Close happens before the next iteration. This would be useful when the file is only used shortly, so the file doesn't have to wait until function returns before it is closed. With a deferblock it is possible to return directly or break out of the for loop from within the block, which you cannot do with a closure.

It makes it sound a bit like try {} finally {}, which is something that I, at least, would certainly like to avoid.

I personally believe that defer is more like finalizers. People write defer to keep the code that opens a resource and closes a resource together, and to forget about the fact that the resource needs to be closed while writing function body.

In general, I'm not a fan of special blocks of code that change the way syntactically unrelated lines inside the blocks work. It tends towards some annoying to read and maintain code.

This makes sense but I find the defer case special. Think of it like a scope, where variables inside the scope cannot be used outside the scope. A deferblock makes sure that defer functions inside the block are called before the program leaves the scope. It is a bit like how finalizer in C++ works, but IMHO defers are mostly used as finalizers.

There would likely be too many times that you would need to do a normal function-scoped defer inside of a defer block.

When you would need to do this you might not want to use deferblock. Since defer is usually used to execute something arbitrarily late, it's often OK to remove deferblock without serious consequences. I acknowledge that I would seldom use a deferblock since it is often OK to scope defer calls to the function. But there are some rare occasions when I do.

A case that I could imagine you would need to do function-scoped defer is when you want to use an object outside the block, like:

var files [1000]file
for i := 0; i < 1000; i++ {
  deferblock {
    f := openfile(i)
    defer f.Close()
    files[i] := openfile(f.Read())
    defer files[i].Close() // needs to be function-scoped
  }
}
// use files

You need to move the defer outside the deferblock:

var files [1000]file
for i := 0; i < 1000; i++ {
  deferblock {
    f := openfile(i)
    defer f.Close()
    files[i] := openfile(f.Read())
  }
  defer files[i].Close()
}
// use files

Think of this as calling defer at the same scope where you need the files.

[deanveloper]

deferblock is not meant to replace defer func(). It surrounds a block of code to call defer methods earlier, so they are called when the program leaves deferblock instead of before function exit.

My bad. I made a mistake when I was originally typing. What I meant was func() { ... defer foo() ... }(). I can rewrite all of those with the notation I had originally meant.

func f() {
  for i := 0; i < 1000; i++ {
    func() {
      file := open(i)
      defer file.Close()
      // work with the file
    }()
  }
}

etc etc. I understand the problem, but it seems like a closure would do the same thing, and the only advantage at the higher level is to "break to an outer label," which still doesn't make sense to me as to why you would ever want to do this.

So to revise, it would be good if you could provide an example where a closure would not suffice. Again, my mistake for not typing the original statement correctly.