Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: internal compiler error instantiating generic type #48185

Closed
dsnet opened this issue Sep 3, 2021 · 9 comments
Closed

cmd/compile: internal compiler error instantiating generic type #48185

dsnet opened this issue Sep 3, 2021 · 9 comments
Labels
Milestone

Comments

@dsnet
Copy link
Member

@dsnet dsnet commented Sep 3, 2021

I apologize for all the bug reports, but I'm hitting another internal compiler error.

$ go.tip version
go version devel go1.18-d9244f8b64 Fri Sep 3 21:31:30 2021 +0000 linux/amd64

$ go.tip test
# github.com/go-json-experiment/json_test [github.com/go-json-experiment/json.test]
<autogenerated>:1: internal compiler error: Dictionary should have already been generated: github.com/go-json-experiment/json..dict.(*arshalers[json.UnmarshalOptions,json.Decoder]).lookup

goroutine 1 [running]:
runtime/debug.Stack()
	/usr/local/go.tip/src/runtime/debug/stack.go:24 +0x65
cmd/compile/internal/base.FatalfAt({0x9df580, 0xc0}, {0xd120ee, 0x34}, {0xc000589320, 0x2, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/base/print.go:227 +0x154
cmd/compile/internal/base.Fatalf(...)
	/usr/local/go.tip/src/cmd/compile/internal/base/print.go:196
cmd/compile/internal/reflectdata.getDictionary(0xc000732eb0, {0xc0006b1c00, 0x2, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:2043 +0x245
cmd/compile/internal/reflectdata.methodWrapper(0xc000730070, 0xc000732f00, 0xf5)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:1909 +0x10ea
cmd/compile/internal/reflectdata.methods(0xc000730070)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:347 +0x39f
cmd/compile/internal/reflectdata.uncommonSize(...)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:79
cmd/compile/internal/reflectdata.dcommontype(0xc0006dbf80, 0xc000730070)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:716 +0x1cc
cmd/compile/internal/reflectdata.writeType(0xc0006b29a0)
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:1140 +0x327
cmd/compile/internal/reflectdata.WriteRuntimeTypes()
	/usr/local/go.tip/src/cmd/compile/internal/reflectdata/reflect.go:1254 +0x3d8
cmd/compile/internal/gc.dumpdata()
	/usr/local/go.tip/src/cmd/compile/internal/gc/obj.go:118 +0x6e
cmd/compile/internal/gc.Main(0xd1dcb0)
	/usr/local/go.tip/src/cmd/compile/internal/gc/main.go:327 +0x10de
main.main()
	/usr/local/go.tip/src/cmd/compile/main.go:55 +0xdd

FAIL	github.com/go-json-experiment/json [build failed]

Reproduction:

$ git clone https://github.com/go-json-experiment/json.git
$ cd json
$ git checkout origin/generics-bug4
$ go test

Based on @mdempsky's comment in #48016 (comment), I tried building this with GOEXPERIMENT=unified and that also panics, but due to a different problem:

$ GOEXPERIMENT=unified go.tip test
# github.com/go-json-experiment/json_test [github.com/go-json-experiment/json.test]
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
sync.runtime_SemacquireMutex(0xc0000cf068, 0x98, 0xa2)
	/usr/local/go.tip/src/runtime/sema.go:71 +0x25
sync.(*Mutex).lockSlow(0xc0004ade4c)
	/usr/local/go.tip/src/sync/mutex.go:138 +0x165
sync.(*Mutex).Lock(...)
	/usr/local/go.tip/src/sync/mutex.go:81
sync.(*Once).doSlow(0xb70e27, 0xc0000cf0e0)
	/usr/local/go.tip/src/sync/once.go:64 +0x53
sync.(*Once).Do(...)
	/usr/local/go.tip/src/sync/once.go:59
cmd/compile/internal/types2.(*Named).load(0xc0004addd0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:56 +0xdb
cmd/compile/internal/types2.(*Named).TParams(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:124
cmd/compile/internal/types2.(*typeWriter).typ(0xc0000cf398, {0xe50750, 0xc0004addd0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:237 +0x3bb
cmd/compile/internal/types2.(*typeWriter).typeList(0xc0000cf398, {0xc0003daec0, 0x2, 0x138b1e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:272 +0xd2
cmd/compile/internal/types2.typeHash({0xe50750, 0xc0004ade60}, {0xc0003daec0, 0x10, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/types2/subst.go:269 +0xef
cmd/compile/internal/types2.(*Checker).instance(0xc0000f4640, {0x0, 0x0, 0x0}, {0xe50750, 0xc0004ade60}, {0xc0003daec0, 0xc0005a9e90, 0xc0005d7bc0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:128 +0x31f
cmd/compile/internal/types2.Instantiate(0xc0005d7bc0, {0xe50750, 0xc0004ade60}, {0xc0003daec0, 0x2, 0x2}, 0x0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:53 +0xb1
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7bc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:231 +0x150
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d7b60, 0xb}, 0xc0005cc4e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d7b60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7b60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:248 +0x7d
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d7680, 0x9}, 0xc0005cc4e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d7680)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).structType(0xc0005d7680)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:268 +0xf2
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7680)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:254 +0x39c
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xb0, 0xe0}, 0xc0005cc4e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005cc540)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*pkgReader2).objIdx.func1.1(0xd1dca0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:406 +0x3f
cmd/compile/internal/types2.(*Named).load.func1()
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:66 +0x2a
sync.(*Once).doSlow(0xb70e27, 0xc0000c80a0)
	/usr/local/go.tip/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
	/usr/local/go.tip/src/sync/once.go:59
cmd/compile/internal/types2.(*Named).load(0xc0004addd0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:56 +0xdb
cmd/compile/internal/types2.(*Named).TParams(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/named.go:124
cmd/compile/internal/types2.(*typeWriter).typ(0xc0000cff90, {0xe50750, 0xc0004addd0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:237 +0x3bb
cmd/compile/internal/types2.(*typeWriter).typeList(0xc0000cff90, {0xc0003dae80, 0x2, 0x138b1e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/typestring.go:272 +0xd2
cmd/compile/internal/types2.typeHash({0xe50750, 0xc0004ade60}, {0xc0003dae80, 0x1, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/types2/subst.go:269 +0xef
cmd/compile/internal/types2.(*Checker).instance(0xc0000f4640, {0x0, 0x0, 0x0}, {0xe50750, 0xc0004ade60}, {0xc0003dae80, 0xc0005a9e90, 0xc0005d70e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:128 +0x31f
cmd/compile/internal/types2.Instantiate(0xc0005d70e0, {0xe50750, 0xc0004ade60}, {0xc0003dae80, 0x2, 0x2}, 0x0)
	/usr/local/go.tip/src/cmd/compile/internal/types2/instantiate.go:53 +0xb1
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d70e0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:231 +0x150
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d7080, 0xb}, 0xc0005d6c60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d7080)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).doTyp(0xc0005d7080)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:248 +0x7d
cmd/compile/internal/noder.(*pkgReader2).typIdx(0xc0000e0900, {0xc0005d6cc0, 0x81}, 0xc0005d6c60)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:206 +0x175
cmd/compile/internal/noder.(*reader2).typ(0xc0005d6cc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:178 +0x38
cmd/compile/internal/noder.(*reader2).param(0xc0005d6cc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:333 +0x69
cmd/compile/internal/noder.(*reader2).params(0xc0005d6cc0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:323 +0x74
cmd/compile/internal/noder.(*reader2).signature(0xc0005d6cc0, 0x0)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:313 +0x47
cmd/compile/internal/noder.(*pkgReader2).objIdx.func1()
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:392 +0x4f5
cmd/compile/internal/types2.resolve.func1()
	/usr/local/go.tip/src/cmd/compile/internal/types2/scope.go:255 +0x3c
sync.(*Once).doSlow(0x6, 0x7f9623e772cb)
	/usr/local/go.tip/src/sync/once.go:68 +0xc2
sync.(*Once).Do(...)
	/usr/local/go.tip/src/sync/once.go:59
cmd/compile/internal/types2.resolve({0x7f9623e3d55e, 0xc0004fac00}, {0xe61cf8, 0xc0005d24e0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/scope.go:254 +0x85
cmd/compile/internal/types2.(*Scope).Lookup(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/scope.go:72
cmd/compile/internal/noder.(*reader2).obj(0xc0004ff620)
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:346 +0xd0
cmd/compile/internal/noder.readPackage2(0xc0000f4640, 0xc0003d7080, {{0xc0000b24b1, 0x22}, {0x5cc, 0x5cd, 0x607, 0x687, 0x79f, 0xc78, ...}, ...})
	/usr/local/go.tip/src/cmd/compile/internal/noder/reader2.go:46 +0x314
cmd/compile/internal/noder.unified.func1({0x7f9623e35ccd, 0x5ccd}, 0xc0005001e0, 0x22, 0x40a3bd)
	/usr/local/go.tip/src/cmd/compile/internal/noder/unified.go:86 +0x108
cmd/compile/internal/noder.readImportFile({0xc0000b24b1, 0xc7f9e0}, 0xc0000e0180, 0xc0003c8901, 0xc0003d7080)
	/usr/local/go.tip/src/cmd/compile/internal/noder/import.go:282 +0x493
cmd/compile/internal/noder.(*gcimports).ImportFrom(0xc77f60, {0xc0000b24b1, 0xc00009f4a0}, {0x0, 0xc0000f18f0}, 0x20)
	/usr/local/go.tip/src/cmd/compile/internal/noder/import.go:64 +0x3a
cmd/compile/internal/types2.(*Checker).importPackage(0xc0000f4640, {0xc000376ea0, 0x0, 0x0}, {0xc0000b24b1, 0x22}, {0x7ffed7a85ba1, 0x9})
	/usr/local/go.tip/src/cmd/compile/internal/types2/resolver.go:144 +0x19e
cmd/compile/internal/types2.(*Checker).collectObjects(0xc0000f4640)
	/usr/local/go.tip/src/cmd/compile/internal/types2/resolver.go:252 +0x10b0
cmd/compile/internal/types2.(*Checker).checkFiles(0xc0000f4640, {0xc0000b45e0, 0x0, 0x0})
	/usr/local/go.tip/src/cmd/compile/internal/types2/check.go:266 +0x1ec
cmd/compile/internal/types2.(*Checker).Files(...)
	/usr/local/go.tip/src/cmd/compile/internal/types2/check.go:245
cmd/compile/internal/noder.checkFiles({0xc0000b4450, 0x1, 0xc0000d1548})
	/usr/local/go.tip/src/cmd/compile/internal/noder/irgen.go:65 +0x485
cmd/compile/internal/noder.writePkgStub({0xc0000b4450, 0x1, 0x1})
	/usr/local/go.tip/src/cmd/compile/internal/noder/unified.go:163 +0x46
cmd/compile/internal/noder.unified({0xc0000b4450, 0x0, 0x2})
	/usr/local/go.tip/src/cmd/compile/internal/noder/unified.go:90 +0x12a
cmd/compile/internal/noder.LoadPackage({0xc0000ba0e0, 0x1, 0x0})
	/usr/local/go.tip/src/cmd/compile/internal/noder/noder.go:84 +0x35d
cmd/compile/internal/gc.Main(0xd1dcb0)
	/usr/local/go.tip/src/cmd/compile/internal/gc/main.go:190 +0xaf3
main.main()
	/usr/local/go.tip/src/cmd/compile/main.go:55 +0xdd
FAIL	github.com/go-json-experiment/json [build failed]

The relevant change that seems to cause an ICE is the code that instantiates a generic type:

Unmarshalers: jsonv2.UnmarshalFuncV2(func(opts jsonv2.UnmarshalOptions, dec *jsonv2.Decoder, val *interface{}) (err error) {
	if dec.PeekKind() == '0' {
		*val = jsonv2.RawValue(nil)
	}
	return jsonv2.SkipFunc
}),

\cc @danscales @mdempsky @griesemer @ianlancetaylor

@cherrymui cherrymui added this to the Go1.18 milestone Sep 3, 2021
@cuonglm

This comment has been hidden.

@dsnet

This comment has been hidden.

@cuonglm

This comment has been hidden.

@anacrolix
Copy link
Contributor

@anacrolix anacrolix commented Sep 4, 2021

I get the same error with anacrolix/stm@80e033a, running go test ./rate.

# github.com/anacrolix/stm/rate [github.com/anacrolix/stm/rate.test]
<autogenerated>:1: internal compiler error: Dictionary should have already been generated: github.com/anacrolix/stm..dict.(*Var[bool]).Set

goroutine 1 [running]:
runtime/debug.Stack()
	/Users/anacrolix/src/go.master/src/runtime/debug/stack.go:24 +0x65
cmd/compile/internal/base.FatalfAt({0x27faeca, 0x0}, {0x190f2b3, 0x34}, {0xc0004caec0, 0x2, 0x2})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/base/print.go:227 +0x154
cmd/compile/internal/base.Fatalf(...)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/base/print.go:196
cmd/compile/internal/reflectdata.getDictionary(0xc000961630, {0xc000a1a4b8, 0x1, 0x1})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:2043 +0x245
cmd/compile/internal/reflectdata.methodWrapper(0xc0008a79d0, 0xc000961680, 0x40)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:1909 +0x10ea
cmd/compile/internal/reflectdata.methods(0xc0008a79d0)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:347 +0x39f
cmd/compile/internal/reflectdata.writeITab(0xc000b27e00, 0xc0008a79d0, 0xc00086a070)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:1290 +0x26a
cmd/compile/internal/reflectdata.ITabLsym(0xc00086a070, 0xc0008776c0)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/reflectdata/reflect.go:847 +0x19c
cmd/compile/internal/noder.(*irgen).finalizeSyms(0xc0000bae00)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/stencil.go:1632 +0x7a5
cmd/compile/internal/noder.(*irgen).stencil(0xc0000bae00)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/stencil.go:234 +0x46d
cmd/compile/internal/noder.(*irgen).generate(0xc0000bae00, {0xc000096b20, 0x2, 0xc0})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/irgen.go:304 +0x359
cmd/compile/internal/noder.check2({0xc000096b20, 0x2, 0x2})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/irgen.go:92 +0x175
cmd/compile/internal/noder.LoadPackage({0xc0000ba0e0, 0x2, 0x0})
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/noder/noder.go:90 +0x335
cmd/compile/internal/gc.Main(0x191aec0)
	/Users/anacrolix/src/go.master/src/cmd/compile/internal/gc/main.go:190 +0xaf3
main.main()
	/Users/anacrolix/src/go.master/src/cmd/compile/main.go:55 +0xdd

FAIL	github.com/anacrolix/stm/rate [build failed]
FAIL

Loading

@cuonglm
Copy link
Member

@cuonglm cuonglm commented Sep 5, 2021

Simpler reproducer for unified IR:

$ cat p.go
package p

type MarshalOptions struct {
	Marshalers *Marshalers
}

type Encoder struct {}

type Marshalers = arshalers[MarshalOptions, Encoder]

type arshalers[Options, Coder any] struct{}

func MarshalFuncV1[T any](fn func(T) ([]byte, error)) *Marshalers {
	return &Marshalers{}
}
$ cat p_test.go
package p

I think this is problem of types2 lazy loading API. In (*Named).load, we call t.resolve(t) which may refer back to the same named type, so calling (*Named).load again, causing the deadlock. In above code, it's the arshalers type. I can't find a way to trigger the bug purely in types2, though.

cc @mdempsky @griesemer

Loading

@danscales
Copy link

@danscales danscales commented Sep 7, 2021

I have a fix for the -G=3 version of the issue. I will not close automatically, so the bug can remain open for the unified IR version of the issue. Or we can create separate issues.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 7, 2021

Change https://golang.org/cl/347909 mentions this issue: cmd/compile: make sure imported instantiated types have their methods created

Loading

@danscales
Copy link

@danscales danscales commented Sep 7, 2021

Thanks for all the issue reports, @dsnet and @anacrolix . They are all useful, so don't worry about filing them. Of course, somewhat minimized cases are always appreciated (so it is easier to add a test case).

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 7, 2021

Change https://golang.org/cl/347534 mentions this issue: cmd/compile: fix deadlock in (*Named).load

Loading

gopherbot pushed a commit that referenced this issue Sep 7, 2021
For lazy import resolution, there's reentrancy issue with (*Named).load
method, when "t.resolve(t)" can lead us to the same named type, thus
(*Named).load is called recursively, causing the deadlock.

The main problem is that when instantinate a type, we calculate the type
hashing, including TParams. Calling t.TParams().Len() triggers the
reentrancy call to "(*Named).load".

To fix this, just not checking TParams().Len() if we are hashing.

Updates #48185

Change-Id: Ie34842d7b10fad5d11fbcf75bb1c64a89deac6b8
Reviewed-on: https://go-review.googlesource.com/c/go/+/347534
Trust: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
@gopherbot gopherbot closed this in 8078355 Sep 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants