go/ast: no way to distinguish some For statements, like for ;; {} and for {}

[quasilyte]

What version of Go are you using (go version)?

$ go version

go version devel +5c7748dc9d Mon Aug 10 23:44:58 2020 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE=""

GOENV=""

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOINSECURE=""

GOMODCACHE=""

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH=""

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT=""

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR=""

GCCGO="gccgo"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build608119567=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Parse a and b: they produce the same AST. There is no way to distinguish them.

// a
for ;; {}
// b
for {}

Same for the cond-only for loops, we can't distinguish them:

// a
for cond {}
// b
for ; cond; {}

With SliceExpr, we have a useful Slice3 flag that helps to distinguish ambiguous cases.

Maybe we can have something similar for the ForStmt? It can also be solved by adding semicolon positions to the ForStmt.

I know that Go AST is (ahem) AST, not a parse tree, so some information can be lost there, but it seems like this would be useful for tools that analyze the Go source code style.

In particular, gogrep tool could benefit from that.

isharipov@lispbook:go$ cat example.go
package example

func foo(cond bool) {
	for ;; {}
	for {}

	for cond {}
	for ; cond; {}
}
isharipov@lispbook:go$ gogrep -x 'for ;; {}' example.go
example.go:4:2: for { }
example.go:5:2: for { }
isharipov@lispbook:go$ gogrep -x 'for cond {}' example.go
example.go:7:2: for cond { }
example.go:8:2: for cond { }

(It also looks like go/printer has the same difficulties, this is why we get identical printed forms above.)

I'm also aware that after gofmt both a and b become identical; it's not always desirable to reformat the code being processed.

Another gogrep-related thing. for $*_ {} could read as "any for loop" while for ; $*_; {} should probably match only non-range loops. Right now it's hard to produce a different pattern for these as both of them generate identical AST.

What did you expect to see?

A way to distinguish between 2 syntax forms.

What did you see instead?

No way to distinguish between 2 syntax forms.

[networkimprov]

cc @griesemer

[griesemer]

Marked for Go 1.17 but only so it's on the radar if we decide to address this.

[findleyr]

Moving this to 1.18 as it's not going to be addressed before the freeze.

Note that adding information to the AST is not always a net win. For example, a significant fraction of gopls' memory use is devoted to AST, and adding fields can have a real (albeit incremental) impact on performance. While this change might have negligible impact, we have to draw the line somewhere, and for this there are some heuristics:

• Does the information affect the correctness of the code. If so, we must capture it. This is the case for Slice3 (s[1:2:] is disallowed by the spec), but is not the case here.
• Is the information required for gofmt, i.e. canonical formatting of the code. Again, this is not the case here.

So with these considerations, I'm inclined to say that we shouldn't do this. However, I am very sympathetic to the underlying motivation. In gopls we've had various struggles writing refactoring tools, related to the fact that we can't compute the exact input from the AST. There is probably room for a higher-level library on top of go/{scanner,ast} that provides a better API for working with raw Go syntax.