proposal: spec: redefine range loop variables in each iteration

As an alternative to #20725 (which was originally about go vet), let the variables of range loops be implicitly redefined in each iteration like in Dart's for loops. That is,

for k, v := range vals {
  // ...
}

should be equivalent to

for k, v := range vals {
  k := k
  v := v
  // ...
}

This will make it "safe" to take the loop variable's addresses as well as capturing the loop variables in nested functions (see #16520).

The proposal could be expanded to vanilla for loops, although that would make it diverge semantically from other languages.

We considered this prior to Go 1, but considered it too quickly. We'll definitely reconsider it for Go 2. I actually think there's a duplicate tracking bug for this.

I think this would be an important improvement to a situation that catches every go developer at least once in their efforts to learn Go.

…

On Tue, 20 Jun 2017, 06:10 Brad Fitzpatrick ***@***.***> wrote: We considered this prior to Go 1, but considered it too quickly. We'll definitely reconsider it for Go 2. I actually think there's a duplicate tracking bug for this. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#20733 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA80GqcY1F4zKSo-optHwPwFOb1Jpks5sFtXGgaJpZM4N-uiT> .

And I would argue even catches seasoned Go programmers occasionally. See https://golang.org/cl/40937. It took me a while staring at that logic trying to figure out why the code wasn't doing what it naturally read as.

Adopting this proposal can make performance of perfectly valid code worse, sometimes dramatically. For example, when escape analysis cannot determine it's safe to keep the {k,v} value on stack while it's address is taken. Then we get O(n) heap allocations instead of the current O(1).

@cznic

@cznic can you give some examples of this. I would expect that most of these cases would actually be bugs because taking the address of the same variable over and over again was the problem in the first place.

@davecheney

@davecheney Consider #20725 (comment) and imagine escape analysis is not able to prove foo only reads via the passed pointer.

@cznic

@cznic

Why would you want to take the address of a copy of the current range value? If i'm not mistaken, any modification to it would be lost.

@davecheney

@davecheney

Why would you want to take the address of a copy of the current range value?

To avoid copying a big value when passing it as an argument.
To call a pointer receiver method on the value.

If i'm not mistaken, any modification to it would be lost.

Taking the address of a value does not imply intent to modify the pointee.

To avoid copying a big value when passing it as an argument.

But the value is already copied as part of the range iteration. Having a separate value escape on every iteration is sub optimal, but hopefully you'll agree it's reasonable to trade a potential performance problem for a very common correctness problem.

To call a pointer receiver method on the value.

True, that is asking a lot of escape analysis and I really don't know the ways that escape analysis and the implicit taking of address interact. But I will note that your example explicitly took the address of the copy of the current index value, so if the intention was to pass a pointer to the value down to a function, you'd want to make sure you were passing a pointer to the original slice element, not a copy. In the code that I have read that is more commonly written as

for i := range vals {
     foo(&vals[i])
}

Taking the address of a value does not imply intent to modify the pointee.

Respectfully, generally it does, as in your method example above.

foo(&vals[i])

Iff vals is a slice/array. There are also maps and channels.

Respectfully, generally it does, as in your method example above.

Even the innocent looking fmt.Println(value) always takes the address of 'value'. As does every function taking an interface (not only interface{}, any interface) argument. Every assignment to an interface variable takes the address of the RHS (if not already a pointer), etc.

Edit: Added '(if not already a pointer)'.

Iff vals is a slice/array. There are also maps and channels.

If it's an array, it's already copied during the evaluation of the range statement. If it's a channel, it's a copy of the value received. Again i'm falling back on arguments of correctness over performance.

Even the innocent looking fmt.Println(value) always takes the address of 'value'.

I believe a copy of value is passed to fmt.Println. That copy is almost always heap allocated due to a limitation introduced in the Go 1.5 garbage collector.

cmd/link: do not close over the loop variable in testDWARF

I'm very supportive of this change. For code that really needs to be optimised then the allocation can be avoided by declaring the variables outside the loop like this:

var (
	i int
	v string
)
for i, v = range []string{"foo", "bar"} {
	println(i, v)
}

Last year there was high profile example of this class of bug:
https://community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591
https://www.theregister.com/2020/03/03/lets_encrypt_cert_revocation/
Detailed analysis: https://bugzilla.mozilla.org/show_bug.cgi?id=1619047
Fix: https://github.com/letsencrypt/boulder/pull/4690/files#diff-d02067a9f9a2bed1110fd4e98641c2effcf5d1d5f18461e35d6ac1535f6e2c21L1411-R1414

I think this can be done now by keeping it behind a go.mod version flag. When you run go mod edit version -go=1.XX it should scan for existing closure captures and convert them to safe equivalents as part of the upgrade. I think it's doable. E.g. it would change this:

for i := 0; i < n; i++ {
  use(&i)
}

to this as part of the upgrade:

{ 
  i := 0
  for ; i < n; i++ {
    use(&i)
  }
}

@carlmjohnson

Hi @carlmjohnson, FWIW, the Go language changes design document uses this issue here as an example:

I think the only feasible safe approach is to not permit language redefinitions.

We are stuck with our current semantics. This doesn't mean we can't improve them. For example, for issue 20733, the range issue, we could change range loops so that taking the address of a range parameter, or referring to it from a function literal, is forbidden. This would not be a redefinition; it would be a removal. That approach might eliminate the bugs without the potential of breaking code unexpectedly.

That document also describes using the go directive in go.mod to control the version of the language used by each module (with some additional discussion & refinement in #28221).

So I suspect it is at least possible to remove this as a feature. One minor comment is I suspect go mod edit would be not used to change source code; it would more likely be go fix from what I understand.

FYI, there is a discussion covering how to redefine "for" loop variable semantics on #56010.

Merge #90221

Closing as duplicate of #60078.

Navigation Menu

Repository navigation

proposal: spec: redefine range loop variables in each iteration #20733

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

proposal: spec: redefine range loop variables in each iteration #20733

Description

Activity

bradfitz commented on Jun 19, 2017

davecheney commented on Jun 19, 2017

dsnet commented on Jun 19, 2017

cznic commented on Jun 20, 2017

davecheney commented on Jun 20, 2017

cznic commented on Jun 20, 2017

davecheney commented on Jun 20, 2017

cznic commented on Jun 20, 2017

davecheney commented on Jun 20, 2017

cznic commented on Jun 20, 2017

davecheney commented on Jun 20, 2017

15 remaining items

bronze1man commented on Oct 22, 2019

danielchatfield commented on Jul 27, 2020

candlerb commented on Jan 2, 2021

earthboundkid commented on Mar 16, 2022

thepudds commented on Mar 16, 2022

mpx commented on Oct 4, 2022

rsc commented on May 9, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions