go/types: SIGSEGV in (*Checker).exprInternal

[bcmills]

#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`

2020-03-01T21:04:30-a4f7b08/darwin-amd64-nocgo

    go_test.go:2345: # strings_test
        panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        	panic: runtime error: invalid memory address or nil pointer dereference
        [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1215e20]
        
        goroutine 1 [running]:
        go/types.(*Checker).handleBailout(0xc000131680, 0xc00067f9f0)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/check.go:243 +0x98
        panic(0x12fe680, 0x15c1cf0)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/runtime/panic.go:967 +0x15d
        go/types.(*Checker).exprInternal(0xc000131680, 0xc0001c3d80, 0x0, 0x0, 0x0, 0x0, 0x30)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:1526 +0x1980
        go/types.(*Checker).rawExpr(0xc000131680, 0xc0001c3d80, 0x0, 0x0, 0x0, 0x0, 0x132b060)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:987 +0x81
        go/types.(*Checker).exprOrType(0xc000131680, 0xc0001c3d80, 0x0, 0x0)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:1644 +0x55
        go/types.(*Checker).call(0xc000131680, 0xc0001c3d80, 0xc00027f080, 0x4)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/call.go:15 +0x66
        go/types.(*Checker).exprInternal(0xc000131680, 0xc0001c3d80, 0x13ce0c0, 0xc00027f080, 0x0, 0x0, 0x11fbb58)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:1471 +0x1e61
        go/types.(*Checker).rawExpr(0xc000131680, 0xc0001c3d80, 0x13ce0c0, 0xc00027f080, 0x0, 0x0, 0x1)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:987 +0x81
        go/types.(*Checker).multiExpr(0xc000131680, 0xc0001c3d80, 0x13ce0c0, 0xc00027f080)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:1601 +0x58
        go/types.(*Checker).expr(0xc000131680, 0xc0001c3d80, 0x13ce0c0, 0xc00027f080)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:1595 +0x49
        go/types.(*Checker).exprInternal(0xc000131680, 0xc0001c3d80, 0x13ce1c0, 0xc00027f140, 0x13cb340, 0xc0000dce60, 0x100bb95)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:1145 +0x15be
        go/types.(*Checker).rawExpr(0xc000131680, 0xc0001c3d80, 0x13ce1c0, 0xc00027f140, 0x13cb340, 0xc0000dce60, 0x100c438)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:987 +0x81
        go/types.(*Checker).exprWithHint(0xc000131680, 0xc0001c3d80, 0x13ce1c0, 0xc00027f140, 0x13cb340, 0xc0000dce60)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:1623 +0x73
        go/types.(*Checker).indexedElts(0xc000131680, 0xc000246c00, 0xf, 0x10, 0x13cb340, 0xc0000dce60, 0xffffffffffffffff, 0xd)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:957 +0x1e2
        go/types.(*Checker).exprInternal(0xc000131680, 0xc0001c3400, 0x13ce1c0, 0xc00027f440, 0x0, 0x0, 0x0)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:1200 +0x584
        go/types.(*Checker).rawExpr(0xc000131680, 0xc0001c3400, 0x13ce1c0, 0xc00027f440, 0x0, 0x0, 0x0)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:987 +0x81
        go/types.(*Checker).multiExpr(0xc000131680, 0xc0001c3400, 0x13ce1c0, 0xc00027f440)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:1601 +0x58
        go/types.(*Checker).expr(0xc000131680, 0xc0001c3400, 0x13ce1c0, 0xc00027f440)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/expr.go:1595 +0x49
        go/types.(*Checker).varDecl(0xc000131680, 0xc000071220, 0xc00000f148, 0x1, 0x1, 0x0, 0x0, 0x13ce1c0, 0xc00027f440)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/decl.go:445 +0x187
        go/types.(*Checker).objDecl(0xc000131680, 0x13d3ca0, 0xc000071220, 0x0)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/decl.go:189 +0x666
        go/types.(*Checker).packageObjects(0xc000131680)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/resolver.go:570 +0x390
        go/types.(*Checker).checkFiles(0xc000131680, 0xc00025fc00, 0x7, 0x8, 0x0, 0x0)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/check.go:257 +0xa3
        go/types.(*Checker).Files(...)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/check.go:248
        go/types.(*Config).Check(0xc000358ac0, 0xc00012abd0, 0xc, 0xc00019c880, 0xc00025fc00, 0x7, 0x8, 0xc00035c5a0, 0x0, 0x16, ...)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/go/types/api.go:348 +0x14e
        cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run(0xc00019c880, 0xc000134c60, 0xc000164600, 0x15, 0x20, 0x2e746c7573, 0x13d4660, 0x12fed40, 0xc00012a81c, 0xc000143c10)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:235 +0x42a
        cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.Run(0x7ffeefbff749, 0x83, 0xc000164600, 0x15, 0x20)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:131 +0x113
        cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.Main(0xc000164600, 0x15, 0x20)
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:118 +0x25f
        main.main()
        	/private/var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/cmd/vet/main.go:35 +0x2bd
        
    go_test.go:2345: go [test -cover -short math strings] failed unexpectedly in /var/folders/9w/4l2_g3kx01x199n37fbmv3s80000gn/T/workdir-host-darwin-10_14/go/src/cmd/go: exit status 2

This may be related to #36605, although the failures reported for that issue are supposed to be fixed in 1.14 and at tip (#36605 (comment)).

CC @matloob @griesemer

[bcmills]

Whatever the source of memory corruption here is, it may also account for some or all of the crashes observed in #35541. Several of those include an exprInternal stack frame:

2020-02-26T17:58:41-a1103dc/linux-mips64le-mengzhuo
2020-02-21T22:00:03-240498d/linux-mips64le-mengzhuo
2019-12-20T23:07:52-4d5bb9c/linux-mips64le-mengzhuo

CC @mengzhuo

[matloob]

I think this is unrelated to #36605, because, as you mentioned, the underlying issue for that was fixed in 1.14. But the go/types code is complex, so I wouldn't be surprised at another race lurking there.

[bcmills]

2020-04-24T21:12:21-943a0d0/linux-mips64le-mengzhuo
2020-04-17T22:12:02-4d9ecde/openbsd-arm-jsing
2020-04-14T19:01:47-334d410/linux-mips64le-mengzhuo
2020-04-08T20:57:58-b2790a2/linux-mips64le-mengzhuo
2020-04-07T18:19:55-7ee8467/linux-mips64le-mengzhuo
2020-04-07T17:18:30-64f19d7/linux-mips64le-mengzhuo
2020-04-02T22:10:27-2bed279/linux-mips64le-mengzhuo
2020-03-25T04:06:34-6b6414c/linux-mips64le-mengzhuo
2020-03-24T22:21:50-9f343b1/linux-mips64le-mengzhuo
2020-03-19T16:17:59-f9f57c4/linux-mips64le-mengzhuo
2020-03-04T19:06:44-68fea52/linux-mips64le-mengzhuo
2020-03-04T18:09:01-588ee79/linux-mips64le-mengzhuo
2020-03-03T11:36:47-bd6f4cd/linux-mips64le-mengzhuo
2020-03-01T21:04:30-a4f7b08/darwin-amd64-nocgo

[bcmills]

2020-06-10T21:39:03-6aadfcd/netbsd-amd64-9_0
2020-05-30T03:20:54-1ed57c1/openbsd-arm-jsing
2020-05-13T12:28:04-866d71a/openbsd-386-64
2020-05-07T19:23:25-6441d34/openbsd-386-64
2020-04-27T20:59:12-352a540/openbsd-386-64
2020-04-27T16:39:59-9ea0146/openbsd-386-64
2020-04-27T15:30:19-a90b730/openbsd-386-64
2020-04-27T14:36:10-bed5418/openbsd-386-64

[bcmills]

2021-01-12T21:39:59-ba76567/darwin-amd64-10_14
2020-12-03T19:32:01-9bc6a97/openbsd-386-64
2020-12-01T06:44:07-fd09bd9/openbsd-386-64
2020-12-01T01:28:06-bf23c54/openbsd-386-64
2020-11-25T22:00:14-2e53cd6/openbsd-386-64
2020-11-24T12:36:21-a966329/openbsd-386-64
2020-11-23T21:59:33-b5535b1/openbsd-386-64
2020-11-19T13:27:11-4783bc9/openbsd-386-64
2020-11-18T17:45:08-6ed8ff9/openbsd-386-64
2020-11-17T22:57:35-f544f6c/openbsd-386-64
2020-11-17T15:25:13-9036a0f/openbsd-386-64
2020-11-16T00:27:33-ac45abd/openbsd-386-64
2020-11-13T20:20:37-1643af1/openbsd-386-64
2020-11-12T18:51:08-eeaa07d/openbsd-386-64
2020-10-30T14:32:52-cf7a54d/openbsd-386-64
2020-10-28T02:59:01-8cd080b/openbsd-386-64
2020-10-27T23:31:11-0b86805/openbsd-386-64
2020-10-21T21:49:18-23787c0/darwin-amd64-nocgo
2020-10-21T00:02:07-d49c4ed/openbsd-386-64
2020-10-17T00:14:24-6003fad/darwin-amd64-10_15

[bcmills]

This one has a SIGSEGV in the collector, but exprInternal on one of the other call stacks:
https://build.golang.org/log/dc8b837f7f5de247bcd9f07144f6d18bec4fe2db

[bcmills]

Funky. Only seems to persist on openbsd-386-64 now.

2021-03-16T19:06:39-9e9211a/openbsd-386-64
2021-03-11T22:40:02-44abc2a/openbsd-386-64
2021-03-10T19:04:36-11e8f6b/openbsd-386-64
2021-03-03T21:54:20-376db57/openbsd-386-64
2021-03-03T20:04:07-dafbee5/openbsd-386-64
2021-03-02T22:01:38-2ac05c8/openbsd-386-64
2021-03-02T17:33:32-16b2c87/openbsd-386-64
2021-03-02T01:39:42-f9c628b/openbsd-386-64
2021-02-27T20:30:37-f5a4005/openbsd-386-64
2021-02-25T15:03:53-54dc8c5/openbsd-386-64
2021-02-25T05:05:20-24aca17/openbsd-386-64
2021-02-23T16:31:27-f48e60b/openbsd-386-64
2021-02-19T19:26:32-bdaa8bf/openbsd-386-64
2021-02-18T20:23:11-4b19790/openbsd-386-64
2021-02-18T03:45:40-67e49ef/openbsd-386-64
2021-02-17T20:31:47-4534fc3/openbsd-386-64
2021-02-17T17:17:01-35839b7/openbsd-386-64
2021-02-12T16:44:11-5bd3da9/openbsd-386-64
2021-02-11T05:13:29-706a59c/openbsd-386-64
2021-02-11T00:57:35-8316e56/openbsd-386-64
2021-02-04T18:06:13-842a928/openbsd-386-64
2021-02-02T23:37:55-e7dfe02/openbsd-386-64
2021-02-02T11:32:59-c3402e3/openbsd-386-64
2021-02-01T17:10:16-f1f686b/openbsd-386-64
2021-02-01T17:07:00-d8a2a07/openbsd-386-64
2021-01-26T13:52:46-c2bea79/openbsd-386-64
2021-01-25T18:56:18-68bf78a/openbsd-386-64
2021-01-22T19:37:57-e13398c/openbsd-386-64
2021-01-22T17:08:14-cf1022a/openbsd-386-64
2021-01-21T18:28:06-bec622c/openbsd-386-64
2021-01-19T22:25:03-fe37c9e/openbsd-386-64
2021-01-15T20:22:50-e0d2015/openbsd-386-64
2021-01-14T06:55:38-d78b04b/openbsd-386-64
2021-01-13T18:03:00-f964368/openbsd-386-64

[bcmills]

OTOH, the darwin failures were always pretty infrequent. Maybe those builders have just been luckier since January.