Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime,cmd/compile: unexpected fault address in runtime.mapassign #56337

Open
gopherbot opened this issue Oct 19, 2022 · 10 comments
Open

runtime,cmd/compile: unexpected fault address in runtime.mapassign #56337

gopherbot opened this issue Oct 19, 2022 · 10 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@gopherbot
Copy link

gopherbot commented Oct 19, 2022

#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`

Issue created automatically to collect these failures.

Example (log):

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x21 pc=0xb91b7a]

goroutine 1 [running]:
cmd/link/internal/loader.(*Loader).OuterSym(...)
	C:/workdir/go/src/cmd/link/internal/loader/loader.go:1695
cmd/link/internal/ld.(*deadcodePass).flood(0xc00006aa00)
	C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:239 +0x265
cmd/link/internal/ld.deadcode(0xc0000c0000)
	C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:336 +0x89
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0x0, 0x0}, {0xe248e3, ...}, ...})
	C:/workdir/go/src/cmd/link/internal/ld/main.go:269 +0xef2
main.main()
	C:/workdir/go/src/cmd/link/main.go:72 +0xedb

watchflakes

@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 19, 2022
@gopherbot
Copy link
Author

gopherbot commented Oct 19, 2022

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "golang.org/x/text/internal/triegen.test" && test == ""
2022-10-14 17:33 windows-amd64-race text@1bdb400f go@bfcc3a75 x/text/internal/triegen.test [build] (log)
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x21 pc=0xb91b7a]

goroutine 1 [running]:
cmd/link/internal/loader.(*Loader).OuterSym(...)
	C:/workdir/go/src/cmd/link/internal/loader/loader.go:1695
cmd/link/internal/ld.(*deadcodePass).flood(0xc00006aa00)
	C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:239 +0x265
cmd/link/internal/ld.deadcode(0xc0000c0000)
	C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:336 +0x89
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0x0, 0x0}, {0xe248e3, ...}, ...})
	C:/workdir/go/src/cmd/link/internal/ld/main.go:269 +0xef2
main.main()
	C:/workdir/go/src/cmd/link/main.go:72 +0xedb

watchflakes

@gopherbot gopherbot added this to the Unreleased milestone Oct 19, 2022
@bcmills bcmills changed the title x/text/internal/triegen.test: unrecognized failures runtime,cmd/compile: unexpected fault address in cmd/compile/internal/ssagen.InitTables Oct 20, 2022
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Oct 20, 2022
@bcmills
Copy link
Member

bcmills commented Oct 20, 2022

(attn @golang/runtime; CC @golang/windows)

@bcmills bcmills changed the title runtime,cmd/compile: unexpected fault address in cmd/compile/internal/ssagen.InitTables runtime,cmd/compile: unexpected fault address in runtime.mapassign Oct 20, 2022
@alexbrainman
Copy link
Member

alexbrainman commented Oct 23, 2022

@bcmills

There are couple of different stack traces.

I looked at the first stack trace in the log:

go build golang.org/x/text/internal/number: # golang.org/x/text/internal/number [golang.org/x/text/internal/number.test]
unexpected fault address 0xffffffffffffffff
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x4fe86]

goroutine 1 [running]:
runtime.throw({0x9d37da?, 0xc00011c4a8?})
	C:/workdir/go/src/runtime/panic.go:1047 +0x65 fp=0xc00011c470 sp=0xc00011c440 pc=0x79e85
runtime.sigpanic()
	C:/workdir/go/src/runtime/signal_windows.go:270 +0xd0 fp=0xc00011c4b8 sp=0xc00011c470 pc=0x8ed50
runtime.mapassign(0x95cbe0, 0xc000113e00, 0x755a2d?)
	C:/workdir/go/src/runtime/map.go:621 +0x246 fp=0xc00011c540 sp=0xc00011c4b8 pc=0x4fe86
cmd/compile/internal/ssagen.InitTables.func3({0x9dcb21, 0xb}, {0x9dbcfe, 0xb}, {0x9e9cf8, 0x17}, {0x9d2ad8, 0x5}, {0xc000126580, 0xe, ...})
	C:/workdir/go/src/cmd/compile/internal/ssagen/ssa.go:3944 +0x1fa fp=0xc00011c608 sp=0xc00011c540 pc=0x74a25a
cmd/compile/internal/ssagen.InitTables()
	C:/workdir/go/src/cmd/compile/internal/ssagen/ssa.go:4755 +0xb054 fp=0xc00011da98 sp=0xc00011c608 pc=0x72e754
cmd/compile/internal/gc.Main(0xa0cfd8)
	C:/workdir/go/src/cmd/compile/internal/gc/main.go:173 +0xaaa fp=0xc00011df20 sp=0xc00011da98 pc=0x8e672a
main.main()
	C:/workdir/go/src/cmd/compile/main.go:57 +0xdd fp=0xc00011df80 sp=0xc00011df20 pc=0x90e93d
runtime.main()
	C:/workdir/go/src/runtime/proc.go:250 +0x1fe fp=0xc00011dfe0 sp=0xc00011df80 pc=0x7c5fe
runtime.goexit()
	C:/workdir/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0xaca81

The crash happens here

if b.tophash[i] != top {

Which looks like uninitialised map used in

intrinsics[intrinsicKey{a, pkg, fn}] = b

I have no idea how to fix that problem.

Alex

@randall77
Copy link
Contributor

randall77 commented Oct 23, 2022

This crash happens early on during compiler startup, before it has even read any of the code to be compiled.
In fact, I think the compiler is completely deterministic up to this point, so it is hard to imagine why it would crash on these runs but not every time.
So some unlikely but possible causes: something wrong with runtime/gc, randomness in map hash triggering some rare map bug, or our favorite boogeyman, bad machine memory.

@randall77
Copy link
Contributor

randall77 commented Oct 23, 2022

Both failures are with -race, which may implicate the race detector itself or our instrumentation that calls it (e.g. if it messed up live variable stack maps that in turn caused the GC to overcollect).

@prattmic
Copy link
Member

prattmic commented Oct 25, 2022

The crashes are in the compiler and linker. Just double checking, do we build the toolchain with the race detector enabled on race builders, or just the tests?

@randall77
Copy link
Contributor

randall77 commented Oct 25, 2022

Good point, I don't think the compiler/linker is race-enabled when on a race builder. Just the resulting tests.

@mknyszek
Copy link
Contributor

mknyszek commented Oct 26, 2022

In triage the consensus is to wait for more to happen. @prattmic notes that it's suspicious that 3 compiler/linker processes crashed at once. Maybe something wrong with the machine?

@mknyszek mknyszek modified the milestones: Unreleased, Go1.20 Oct 26, 2022
@gopherbot
Copy link
Author

gopherbot commented Nov 17, 2022

Found new dashboard test flakes for:

#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`
2022-09-29 20:41 windows-386-2008 sync@8fcdb60f go@b7662047 x/sync/syncmap.TestMapMatchesDeepCopy (log)
unexpected fault address 0xb4df0b0
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xb4df0b0 pc=0x40eaea]

runtime.throw({0x533c48, 0x5})
	C:/workdir/go/src/runtime/panic.go:1047 +0x4d fp=0xa8439c0 sp=0xa8439ac pc=0x43947d
runtime.sigpanic()
	C:/workdir/go/src/runtime/signal_windows.go:270 +0x105 fp=0xa8439e4 sp=0xa8439c0 pc=0x44d395
runtime.evacuated(...)
	C:/workdir/go/src/runtime/map.go:204
...
	C:/workdir/go/src/runtime/asm_386.s:541 +0x30 fp=0xa843c00 sp=0xa843bdc pc=0x467f40
reflect.Value.call({0x5198a0, 0x540e88, 0x13}, {0x5339d3, 0x4}, {0xa969390, 0x1, 0x1})
	C:/workdir/go/src/reflect/value.go:586 +0x894 fp=0xa843e9c sp=0xa843c00 pc=0x4a79a4
reflect.Value.Call({0x5198a0, 0x540e88, 0x13}, {0xa969390, 0x1, 0x1})
	C:/workdir/go/src/reflect/value.go:370 +0x7e fp=0xa843ed8 sp=0xa843e9c pc=0x4a707e
testing/quick.CheckEqual({0x5198a0, 0x540e8c}, {0x5198a0, 0x540e88}, 0x0)
	C:/workdir/go/src/testing/quick/quick.go:331 +0x314 fp=0xa843f74 sp=0xa843ed8 pc=0x505994
golang.org/x/sync/syncmap_test.TestMapMatchesDeepCopy(0xa8312c0)
	C:/workdir/gopath/src/golang.org/x/sync/syncmap/map_test.go:108 +0x47 fp=0xa843f9c sp=0xa843f74 pc=0x507b47
testing.tRunner(0xa8312c0, 0x540e80)

watchflakes

@randall77
Copy link
Contributor

randall77 commented Nov 17, 2022

That latest failure looks unrelated, maybe?
Nothing -race about it.
There do seem to be 2 simultaneous failures in the log (like was observed above, I think), the linker crashing on a bad map operation and a golang.org/x/sync/syncmap test also crashing similarly.
This failure was also 386, not amd64.

No obvious culprit here, unfortunately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Status: Todo
Status: No status
Development

No branches or pull requests

6 participants