cmd/compile: compiler hang involving recursive generic type instantiation

[jake-ciolek]

Go version

go 1.26.1 + tip at cbc2d06

Output of go env in your module/workspace:

GOARCH=arm64

What did you do?

package p

type I[T any] interface {
	M() interface{ I[*T] }
}

ran go build on the above

What did you see happen?

go build command doesn't terminate

What did you expect to see?

go build command terminates

[thepudds]

This hang seems to bisect to the following (and looks like it was introduced circa Go 1.20):

840b346 cmd/compile: reject anonymous interface cycles

CC @golang/compiler @cuonglm (if interested)

[griesemer]

Looks like another issue with an incorrect cyclic data structure. From a stack trace:

goroutine 21 gp=0x179e80584b40 m=nil [runnable]:
runtime.interequal(0x179e84361a90?, 0x179e87547ae0?)
        /Users/gri/goroot/src/runtime/alg.go:302 +0x50 fp=0x179e84361770 sp=0x179e84361770 pc=0x1004ee380
internal/runtime/maps.(*table).Delete(0x179e82f99aa0, 0x100999fa0, 0x179e838c8180, 0x800000000000?, 0x179e84361a90)
        /Users/gri/goroot/src/internal/runtime/maps/table.go:439 +0x134 fp=0x179e843617f0 sp=0x179e84361770 pc=0x1004e5064
internal/runtime/maps.(*Map).Delete(0x179e838c8180, 0x100999fa0, 0x179e84361a90)
        /Users/gri/goroot/src/internal/runtime/maps/map.go:669 +0xb4 fp=0x179e84361830 sp=0x179e843617f0 pc=0x1004e3964
runtime.mapdelete(0x179e84361888?, 0x1006c344c?, 0x179e82f99aa0?)
        /Users/gri/goroot/src/runtime/map.go:149 +0x28 fp=0x179e84361860 sp=0x179e84361830 pc=0x10055a268
cmd/compile/internal/types2.(*typeWriter).typ.deferwrap1()
        /Users/gri/goroot/src/cmd/compile/internal/types2/typestring.go:118 +0x2c fp=0x179e84361890 sp=0x179e84361860 pc=0x1006c434c
cmd/compile/internal/types2.(*typeWriter).typ(0x179e838c82d0, {0x1009ebd80, 0x179e80800fc0})
        /Users/gri/goroot/src/cmd/compile/internal/types2/typestring.go:347 +0x900 fp=0x179e84361ae0 sp=0x179e84361890 pc=0x1006c3b20
cmd/compile/internal/types2.(*typeWriter).typ(0x179e838c82d0, {0x1009ebd80, 0x179e80800ff0})
        /Users/gri/goroot/src/cmd/compile/internal/types2/typestring.go:186 +0x23c fp=0x179e84361d30 sp=0x179e84361ae0 pc=0x1006c345c
cmd/compile/internal/types2.(*typeWriter).typ(0x179e838c82d0, {0x1009ebd80, 0x179e80801020})
        /Users/gri/goroot/src/cmd/compile/internal/types2/typestring.go:186 +0x23c fp=0x179e84361f80 sp=0x179e84361d30 pc=0x1006c345c
cmd/compile/internal/types2.(*typeWriter).typ(0x179e838c82d0, {0x1009ebd80, 0x179e80801040})
        /Users/gri/goroot/src/cmd/compile/internal/types2/typestring.go:186 +0x23c fp=0x179e843621d0 sp=0x179e84361f80 pc=0x1006c345c
cmd/compile/internal/types2.(*typeWriter).typ(0x179e838c82d0, {0x1009ebd80, 0x179e80801070})
        /Users/gri/goroot/src/cmd/compile/internal/types2/typestring.go:186 +0x23c fp=0x179e84362420 sp=0x179e843621d0 pc=0x1006c345c
cmd/compile/internal/types2.(*typeWriter).typ(0x179e838c82d0, {0x1009ebd80, 0x179e808010a0})
        /Users/gri/goroot/src/cmd/compile/internal/types2/typestring.go:186 +0x23c fp=0x179e84362670 sp=0x179e84362420 pc=0x1006c345c
...

[griesemer]

We should move the anonymous interface cycle check, currently in the compiler, to the type checkers, then this is likely going to be fixed (i.e., an error will be reported).

Note that the example can be rewritten as:

type I[T any] interface {
	M() I[*T]
}

which leads to an error.

Esoteric case with work-around. Move to 1.28.