x/tools/gopls: crash due to fixed ast.Node.End beyond EOF

[nobelsmith]

gopls version: v0.15.2/go1.22.1
gopls flags:
update flags: proxy
extension version: 0.41.2
environment: Visual Studio Code darwin
initialization error: undefined
issue timestamp: Fri, 05 Apr 2024 20:02:00 GMT
restart history:
Thu, 04 Apr 2024 19:08:50 GMT: activation (enabled: true)
Fri, 05 Apr 2024 17:57:26 GMT: manual (enabled: true)

ATTENTION: PLEASE PROVIDE THE DETAILS REQUESTED BELOW.

Describe what you observed.

panic: file not found for pos = 1 (-)

goroutine 21096 [running]:
go/types.(*Checker).handleBailout(0x14005c32000, 0x140073eb878)
	  check.go:367  0x9c
panic({0x1014ecbe0%3F, 0x140072e0b40%3F})
	  panic.go:770  0x124
go/types.(*Checker).fileFor(0x14005c32000, 0x1)
	  version.go:131  0x154
go/types.(*Checker).allowVersion(0x14005c32000, 0x101566700%3F, {0x101669480%3F, 0x140062a46a0%3F}, {0x1011a8a18, 0x6})
	  version.go:97  0x64
go/types.(*Checker).genericExprList(0x14005c32000, {0x140062a4720, 0x2, 0x140062a4640%3F})
	  call.go:383  0xb64
go/types.(*Checker).callExpr(0x14005c32000, 0x140028bb480, 0x14001816600)
	  call.go:303  0x560
go/types.(*Checker).exprInternal(0x14005c32000, 0x0, 0x140028bb480, {0x10166eda8, 0x14001816600}, {0x0, 0x0})
	  expr.go:1374  0xd0
go/types.(*Checker).rawExpr(0x14005c32000, 0x0, 0x140028bb480, {0x10166eda8%3F, 0x14001816600%3F}, {0x0%3F, 0x0%3F}, 0x0)
	  expr.go:979  0x12c
go/types.(*Checker).suspendedCall(0x14005c32000, {0x1011a5052, 0x2}, 0x14001816600)
	  stmt.go:173  0x64
go/types.(*Checker).stmt(0x14005c32000, 0x7, {0x10166f528, 0x140042f74c0})
	  stmt.go:499  0xe38
go/types.(*Checker).stmtList(0x14005c32000, 0x7, {0x140062a4740%3F, 0x1011a5ff9%3F, 0x4%3F})
	  stmt.go:121  0x88
go/types.(*Checker).stmt(0x14005c32000, 0x3, {0x10166f0a8, 0x140067a73b0})
	  stmt.go:628  0x249c
go/types.(*Checker).stmtList(0x14005c32000, 0x3, {0x140060b2b80%3F, 0x1011a7620%3F, 0x5%3F})
	  stmt.go:121  0x88
go/types.(*Checker).stmt(0x14005c32000, 0x3, {0x10166ee08, 0x140067a73e0})
	  stmt.go:562  0x1974
go/types.(*Checker).stmt(0x14005c32000, 0x3, {0x10166edd8, 0x140018167c0})
	  stmt.go:574  0x258c
go/types.(*Checker).stmtList(0x14005c32000, 0x3, {0x140042f7510%3F, 0x1011a7620%3F, 0x5%3F})
	  stmt.go:121  0x88
go/types.(*Checker).stmt(0x14005c32000, 0x3, {0x10166ee08, 0x140067a7410})
	  stmt.go:562  0x1974
go/types.(*Checker).rangeStmt(0x14005c32000, 0x3, 0x140042dbbc0)
	  stmt.go:970  0x580
go/types.(*Checker).stmt(0x14005c32000, 0x3, {0x10166f138, 0x140042dbbc0})
	  stmt.go:827  0x874
go/types.(*Checker).stmtList(0x14005c32000, 0x3, {0x140042f7520%3F, 0x1011a7620%3F, 0x5%3F})
	  stmt.go:121  0x88
go/types.(*Checker).stmt(0x14005c32000, 0x3, {0x10166ee08, 0x140067a7440})
	  stmt.go:562  0x1974
go/types.(*Checker).stmt(0x14005c32000, 0x3, {0x10166edd8, 0x14001816800})
	  stmt.go:574  0x258c
go/types.(*Checker).stmtList(0x14005c32000, 0x3, {0x140060b2c80%3F, 0x1011a7620%3F, 0x5%3F})
	  stmt.go:121  0x88
go/types.(*Checker).stmt(0x14005c32000, 0x3, {0x10166ee08, 0x140067a7650})
	  stmt.go:562  0x1974
go/types.(*Checker).stmt(0x14005c32000, 0x0, {0x10166f108, 0x14001816bc0})
	  stmt.go:823  0x2034
go/types.(*Checker).stmtList(0x14005c32000, 0x0, {0x14000871900%3F, 0x0%3F, 0x140039f3eec%3F})
	  stmt.go:121  0x88
go/types.(*Checker).funcBody(0x14005c32000, 0x10166ba18%3F, {0x140039f3fb0%3F, 0x101c810c0%3F}, 0x14002abc140, 0x140067a7680, {0x0%3F, 0x0%3F})
	  stmt.go:41  0x21c
go/types.(*Checker).funcDecl.func1()
	  decl.go:852  0x44
go/types.(*Checker).processDelayed(0x14005c32000, 0x0)
	  check.go:467  0x12c
go/types.(*Checker).checkFiles(0x14005c32000, {0x14002885100, 0x7, 0x8})
	  check.go:411  0x188
go/types.(*Checker).Files(...)
	  check.go:372
golang.org/x/tools/gopls/internal/cache.(*typeCheckBatch).checkPackage(0x140064fe660, {0x1016706e8, 0x140070c02a0}, 0x140038486c0)
	  check.go:1532  0x764
golang.org/x/tools/gopls/internal/cache.(*typeCheckBatch).handleSyntaxPackage(0x140064fe660, {0x1016706e8, 0x140070c02a0}, 0x0, {0x140008dc120, 0x24})
	  check.go:563  0x534
golang.org/x/tools/gopls/internal/cache.(*Snapshot).forEachPackageInternal.func2()
	  check.go:414  0x34
golang.org/x/sync/errgroup.(*Group).Go.func1()
	  errgroup.go:78  0x58
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 20851
	  errgroup.go:75  0x98

gopls stats -anon

{ "DirStats": { "Files": 805, "TestdataFiles": 0, "GoFiles": 41, "ModFiles": 1, "Dirs": 322 }, "GOARCH": "arm64", "GOOS": "darwin", "GOPACKAGESDRIVER": "", "GOPLSCACHE": "", "GoVersion": "go1.22.1", "GoplsVersion": "v0.15.2", "InitialWorkspaceLoadDuration": "708.152041ms", "MemStats": { "HeapAlloc": 33689896, "HeapInUse": 51576832, "TotalAlloc": 551701312 }, "WorkspaceStats": { "Files": { "Total": 1850, "Largest": 935931, "Errs": 0 }, "Views": [ { "GoCommandVersion": "go1.22.1", "AllPackages": { "Packages": 307, "LargestPackage": 155, "CompiledGoFiles": 1846, "Modules": 42 }, "WorkspacePackages": { "Packages": 16, "LargestPackage": 12, "CompiledGoFiles": 41, "Modules": 1 }, "Diagnostics": 1 } ] } }

OPTIONAL: If you would like to share more information, you can attach your complete gopls logs.

NOTE: THESE MAY CONTAIN SENSITIVE INFORMATION ABOUT YOUR CODEBASE.
DO NOT SHARE LOGS IF YOU ARE WORKING IN A PRIVATE REPOSITORY.

<OPTIONAL: ATTACH LOGS HERE>

[findleyr]

CC @adonovan @griesemer

A couple things going on here: (1) we should repair the AST in gopls if it records any positions outside of the file, but also (2) I'm not sure go/types should panic here. Simply using the default version seems acceptable.

[findleyr]

Alan points out that this may be related to #66683.

@nobelsmith can you share anything about the structure of the code you were editing when this occurred?

[nobelsmith]

This error I was doing work on a fiber api, but it was the first time I was trying live server within vs code with another dev. Running live server is the only thing that was out of the ordinary for me otherwise just go installed through brew and working in vscode. @findleyr

[adonovan]

There are two problems here:

1. Checker.fileFor panics if given an non-zero ("IsValid") but actually invalid pos. It should probably just return nil.
2. The AST contains a CallExpr whose first Args expression has a pos of 1. This is a bug in the parser, no doubt related to recovery in the face of errors. I tried to find an input that would reproduce this specific syntax, but without success; I did find a whole lot of other inputs that create ASTs with invalid positions though.

For example:

1. An empty input file yields a File with a non-nil Name Ident that is empty, with Pos=0 End=0.
2. if cond at EOF (sans braces) yields a synthetic BlockStmt with Pos=EOF End=EOF+1; all enclosing nodes have an End of at least EOF+1.
   3 package p; func _() { if cond { if cond { if cond yields three BlockStmts all at EOF.
3. package p; func _() { if at EOF yields an ExprStmt of a BadExpr, both with Pos=EOF End=EOF, and zero width.
4. package p; var yields a fake Ident _ whose End is greater than the End of the parent File (EOF).

I was hoping we can repair the tree to achieve these invariants:
(a) each node's (pos,end) range is a (non-strict subrange) of its parent range, or of (0,size) if the parent is File.
(b) all token.Pos fields are within their enclosing Node's (pos,end) range (with an exception for FuncType.Func).

But I'm not so sure it's feasible. A consequence of (a) is that all computed End positions must be more careful not to report the suffix of fake tokens. For example, given if cond, struct or var at EOF, the parse tree reflects these fake tokens: if cond{}, struct{} and var _. In the first two cases, both fake brace tokens are placed at (EOF, EOF), so Pos is strictly out of range; in the third case the blank Ident has range (EOF, EOF+1), so its end is out of range. (It's tempting to change the fake BlockStmt to allow its Lbrace token to be optional, just like Rparen, but then then whole node would have no position at all, so that's a nonstarter.) So I don't see us getting away from the need for exactly one byte beyond EOF (as handled by logic in the safetoken package).