Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: failure of sweet/fogleman-fauxgl on darwin/amd64 on go1.20 #58533

randall77 opened this issue Feb 14, 2023 · 9 comments

runtime: failure of sweet/fogleman-fauxgl on darwin/amd64 on go1.20 #58533

randall77 opened this issue Feb 14, 2023 · 9 comments
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.


Copy link

To reproduce:

git clone
cd benchmarks/sweet
go build ./cmd/sweet
echo "[[config]]" > config.toml
echo "  name = \"config120\"" >> config.toml
echo "  goroot = \"/Users/khr/go1.20\"" >> config.toml
./sweet run -run=fogleman-fauxgl -count=10 config.toml

Crashes in a bunch of random ways, all gc-adjacent. Zombie objects, 0xb01dfacedebac1e errors, driver segfaults, etc. (A few traces pasted below.)

Fails on both 1.20 and 1.20.1. Seems to work fine on both 1.19.4 and tip.
I'm running Ventura 13.0.1 (Darwin Kernel Version 22.1.0).
Seems to work fine on linux.

Possibly related to #54760 ? Seems somewhat different but the fix there turned this benchmark off on arm64. I wonder if darwin/arm64 fails also, but we need to reenable this benchmark on that platform to check.

@mknyszek @prattmic

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x222 pc=0x10fb8bb]

goroutine 21 [running]:
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:54 +0x1b*B).startRSSSampler.func1()
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:290 +0x13f
created by*B).startRSSSampler
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:280 +0xad
runtime: marked free object in span 0x13852c8, elemsize=416 freeindex=3 (bad use of unsafe.Pointer? try -d=checkptr)
0xc0000ca000 alloc marked  
0xc0000ca1a0 alloc marked  
0xc0000ca340 alloc marked  
0xc0000ca4e0 free  unmarked
0xc0000ca680 free  unmarked
0xc0000ca820 free  unmarked
0xc0000ca9c0 free  unmarked
0xc0000cab60 free  unmarked
0xc0000cad00 free  unmarked
0xc0000caea0 free  unmarked
0xc0000cb040 free  unmarked
0xc0000cb1e0 free  unmarked
0xc0000cb380 free  unmarked
0xc0000cb520 free  unmarked
0xc0000cb6c0 free  unmarked
0xc0000cb860 free  unmarked
0xc0000cba00 free  unmarked
0xc0000cbba0 free  unmarked
0xc0000cbd40 free  marked   zombie
0x000000c0000cbd40:  0x000000c0000cbd50  0x0000000001097dd7 <time.Now+0x0000000000000017> 
0x000000c0000cbd50:  0x0000000300000002  0x00000000010f921e <*B).StartTimer+0x00000000000000fe> 
0x000000c0000cbd60:  0x000000c0000cbd98  0x0000000001125754 <main.main.func1+0x0000000000000034> 
0x000000c0000cbd70:  0x000000000115ef01  0x0000000000000005 
0x000000c0000cbd80:  0x000000c0000d6008  0x000000c00007e000 
0x000000c0000cbd90:  0x000000c0000b8210  0x000000c0000cbef8 
0x000000c0000cbda0:  0x00000000010fae6d <>  0x000000c0000b8210 
0x000000c0000cbdb0:  0x000000c000098330  0x000000000115ef01 
0x000000c0000cbdc0:  0x0000000000000004  0x0000000001716368 
0x000000c0000cbdd0:  0x000000000137a5b8  0x0000000000000004 
0x000000c0000cbde0:  0x0000000000000002  0x0000000000000003 
0x000000c0000cbdf0:  0x40051c963d442e7e  0x000000000115ef01 
0x000000c0000cbe00:  0x000000c00008c0c0  0xbfec261da7059352 
0x000000c0000cbe10:  0xbfec261da7059352  0x000000c0000b8210 
0x000000c0000cbe20:  0x000000000125d520  0xbfea12f684bda130 
0x000000c0000cbe30:  0xbfea12f684bda130  0xbfda12f684bda130 
0x000000c0000cbe40:  0x401471c71c71c71d  0xbfe5555555555555 
0x000000c0000cbe50:  0xbfe5555555555555  0xbfd5555555555555 
0x000000c0000cbe60:  0x4018000000000000  0xbfe6a09e667f3bcc 
0x000000c0000cbe70:  0x3fe6a09e667f3bcc  0x0000000000000000 
0x000000c0000cbe80:  0x8000000000000000  0xbfce2b7dddfefa65 
0x000000c0000cbe90:  0xbfce2b7dddfefa65  0x3fee2b7dddfefa65 
0x000000c0000cbea0:  0x8000000000000000  0x3fe5555555555555 
0x000000c0000cbeb0:  0x3fe5555555555555  0x3fd5555555555555 
0x000000c0000cbec0:  0xc018000000000000  0x0000000000000000 
0x000000c0000cbed0:  0x0000000000000000  0x0000000000000000 
fatal error: found pointer to free object

runtime stack:
runtime.throw({0x116494f?, 0xc0000cbee0?})
        /Users/khr/go1.20/src/runtime/panic.go:1047 +0x5d fp=0x7ff7bfeff488 sp=0x7ff7bfeff458 pc=0x10339bd
        /Users/khr/go1.20/src/runtime/mgcsweep.go:846 +0x2e5 fp=0x7ff7bfeff508 sp=0x7ff7bfeff488 pc=0x1023625
runtime.(*sweepLocked).sweep(0x0?, 0x0)
        /Users/khr/go1.20/src/runtime/mgcsweep.go:634 +0x9f6 fp=0x7ff7bfeff5f8 sp=0x7ff7bfeff508 pc=0x1022f96
runtime.(*mcentral).uncacheSpan(0x7ff7bfeff668?, 0x12654a0?)
        /Users/khr/go1.20/src/runtime/mcentral.go:228 +0xa5 fp=0x7ff7bfeff620 sp=0x7ff7bfeff5f8 pc=0x1015665
        /Users/khr/go1.20/src/runtime/mcache.go:291 +0x145 fp=0x7ff7bfeff688 sp=0x7ff7bfeff620 pc=0x10150c5
        /Users/khr/go1.20/src/runtime/mcache.go:328 +0x39 fp=0x7ff7bfeff6b0 sp=0x7ff7bfeff688 pc=0x10151b9
        /Users/khr/go1.20/src/runtime/proc.go:5096 +0x2c fp=0x7ff7bfeff6c8 sp=0x7ff7bfeff6b0 pc=0x1040e2c
        /Users/khr/go1.20/src/runtime/proc.go:2335 +0xb5 fp=0x7ff7bfeff6f8 sp=0x7ff7bfeff6c8 pc=0x103a4f5
        /Users/khr/go1.20/src/runtime/proc.go:3007 +0xabc fp=0x7ff7bfeff800 sp=0x7ff7bfeff6f8 pc=0x103bd1c
        /Users/khr/go1.20/src/runtime/proc.go:3360 +0xb1 fp=0x7ff7bfeff838 sp=0x7ff7bfeff800 pc=0x103cb51
        /Users/khr/go1.20/src/runtime/proc.go:3511 +0x12d fp=0x7ff7bfeff868 sp=0x7ff7bfeff838 pc=0x103d06d
        /Users/khr/go1.20/src/runtime/asm_amd64.s:452 +0x43 fp=0x7ff7bfeff878 sp=0x7ff7bfeff868 pc=0x1063283

goroutine 1 [runnable]:
runtime.gopark(0x1014070?, 0xe?, 0x90?, 0xe?, 0xb?)
        /Users/khr/go1.20/src/runtime/proc.go:381 +0xd6 fp=0xc000124608 sp=0xc0001245e8 pc=0x1036676
        /Users/khr/go1.20/src/runtime/mgcmark.go:651 +0xeb fp=0xc000124640 sp=0xc000124608 pc=0x101c00b
        /Users/khr/go1.20/src/runtime/mgcmark.go:508 +0x23e fp=0xc0001246a0 sp=0xc000124640 pc=0x101ba1e
        /Users/khr/go1.20/src/runtime/malloc.go:1217 +0x5a fp=0xc0001246c8 sp=0xc0001246a0 pc=0x100cd9a
runtime.mallocgc(0x271000, 0x1134f40, 0x1)
        /Users/khr/go1.20/src/runtime/malloc.go:932 +0xd0 fp=0xc000124730 sp=0xc0001246c8 pc=0x100c5b0
runtime.makeslice(0x126a1e0?, 0x101c000124798?, 0x2958b5e8?)
        /Users/khr/go1.20/src/runtime/slice.go:103 +0x52 fp=0xc000124758 sp=0xc000124730 pc=0x104b412
image.NewRGBA({{0x0?, 0x0?}, {0x320?, 0x320?}})
        /Users/khr/go1.20/src/image/image.go:215 +0x65 fp=0xc0001247d0 sp=0xc000124758 pc=0x10beec5, 0x320, {0x118f5e0, 0xc0000a8040}, 0x1)
        /Users/khr/gopath/pkg/mod/ +0x627 fp=0xc000124b80 sp=0xc0001247d0 pc=0x111b1a7*RotateAnimation).RenderNext(0xc00007e000)
        /Users/khr/gowork/tmp/benchmarks/third_party/fogleman-fauxgl/animate.go:77 +0x3f1 fp=0xc000124d70 sp=0xc000124b80 pc=0x1125211
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/fogleman-fauxgl/main.go:46 +0x5c fp=0xc000124da8 sp=0xc000124d70 pc=0x112577c{0x116509a, 0x1e}, 0xc0000cbf58, {0x1261240, 0x8, 0xc0000f5f70?})
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:455 +0x22d fp=0xc000124f08 sp=0xc000124da8 pc=0x10fae6d
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/fogleman-fauxgl/main.go:42 +0x1c9 fp=0xc000124f80 sp=0xc000124f08 pc=0x11256a9
        /Users/khr/go1.20/src/runtime/proc.go:250 +0x207 fp=0xc000124fe0 sp=0xc000124f80 pc=0x1036247
        /Users/khr/go1.20/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000124fe8 sp=0xc000124fe0 pc=0x10653a1
unexpected fault address 0xb01dfacedebac1e
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x10fb8bb]

goroutine 8 [running]:
runtime.throw({0x115f03f?, 0xc00007a0c0?})
        /Users/khr/go1.20/src/runtime/panic.go:1047 +0x5d fp=0xc00017de90 sp=0xc00017de60 pc=0x10339bd
        /Users/khr/go1.20/src/runtime/signal_unix.go:851 +0x28a fp=0xc00017def0 sp=0xc00017de90 pc=0x104a48a
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:54 +0x1b fp=0xc00017df08 sp=0xc00017def0 pc=0x10fb8bb*B).startRSSSampler.func1()
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:290 +0x13f fp=0xc00017ffe0 sp=0xc00017df08 pc=0x10f997f
        /Users/khr/go1.20/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00017ffe8 sp=0xc00017ffe0 pc=0x10653a1
created by*B).startRSSSampler
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:280 +0xad

goroutine 1 [chan receive]:
runtime.gopark(0xc000174a30?, 0x1017691?, 0x20?, 0x3a?, 0xc000174a98?)
        /Users/khr/go1.20/src/runtime/proc.go:381 +0xd6 fp=0xc000174a18 sp=0xc0001749f8 pc=0x1036676
runtime.chanrecv(0xc000134420, 0xc000174b08, 0x1)
        /Users/khr/go1.20/src/runtime/chan.go:583 +0x49d fp=0xc000174aa8 sp=0xc000174a18 pc=0x100611d
runtime.chanrecv1(0x1017691?, 0x10?)
        /Users/khr/go1.20/src/runtime/chan.go:442 +0x18 fp=0xc000174ad0 sp=0xc000174aa8 pc=0x1005c58*Context).DrawTriangles(0xc00690c000, {0xc000180000, 0x371aa, 0x371aa})
        /Users/khr/gopath/pkg/mod/ +0x191 fp=0xc000174b38 sp=0xc000174ad0 pc=0x110e771*Context).DrawMesh(0xc00690c000?, 0xc00010e000)
        /Users/khr/gopath/pkg/mod/ +0x31 fp=0xc000174b80 sp=0xc000174b38 pc=0x110e991*RotateAnimation).RenderNext(0xc000166000)
        /Users/khr/gowork/tmp/benchmarks/third_party/fogleman-fauxgl/animate.go:73 +0x3c5 fp=0xc000174d70 sp=0xc000174b80 pc=0x11251e5
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/fogleman-fauxgl/main.go:46 +0x5c fp=0xc000174da8 sp=0xc000174d70 pc=0x112577c{0x116509a, 0x1e}, 0xc000127f58, {0x1261240, 0x8, 0xc000145f70?})
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:455 +0x22d fp=0xc000174f08 sp=0xc000174da8 pc=0x10fae6d
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/fogleman-fauxgl/main.go:42 +0x1c9 fp=0xc000174f80 sp=0xc000174f08 pc=0x11256a9
        /Users/khr/go1.20/src/runtime/proc.go:250 +0x207 fp=0xc000174fe0 sp=0xc000174f80 pc=0x1036247
        /Users/khr/go1.20/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000174fe8 sp=0xc000174fe0 pc=0x10653a1
@randall77 randall77 added this to the Go1.20.2 milestone Feb 14, 2023
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Feb 14, 2023
Copy link
Contributor Author

GOGC=off fixes it.
GOMAXPROCS=1 makes it always fail in the driver segfault mode.

Reports a data race under the race detector. This may be the underlying issue.

Read at 0x00c00bb874d0 by goroutine 44:*Context).rasterize()
      /Users/khr/gopath/pkg/mod/ +0x99c*Context).drawClippedTriangle()
      /Users/khr/gopath/pkg/mod/ +0xa49*Context).DrawTriangle()
      /Users/khr/gopath/pkg/mod/ +0x964*Context).DrawTriangles.func1()
      /Users/khr/gopath/pkg/mod/ +0xda*Context).DrawTriangles.func2()
      /Users/khr/gopath/pkg/mod/ +0x47

Previous write at 0x00c00bb874d0 by goroutine 50:*Context).rasterize()
      /Users/khr/gopath/pkg/mod/ +0x1068*Context).drawClippedTriangle()
      /Users/khr/gopath/pkg/mod/ +0xa49*Context).DrawTriangle()
      /Users/khr/gopath/pkg/mod/ +0x964*Context).DrawTriangles.func1()
      /Users/khr/gopath/pkg/mod/ +0xda*Context).DrawTriangles.func2()
      /Users/khr/gopath/pkg/mod/ +0x47

So just a bad benchmark, then?
In fact, at context.go:232, the code says:

			if dc.ReadDepth && bz > dc.DepthBuffer[i] { // safe w/out lock?

So, the answer might be no.


Copy link

Yeah, that's probably it, it just has races. To be fair, I'm not totally sure this code is super well-maintained these days. We might want to consider just moving away from this benchmark. (Originally we ran it as sort of an interesting math/interface/etc. benchmark, but I'm not sure we ever got any super strong signal from it.)

I've been thinking about revamping the Sweet benchmarks a little and replacing a bunch of the benchmarks that don't seem to give us a strong signal one way or the other. We also have quite a few bent benchmarks that I think are noisy in unhelpful ways.

I'll open a bug for this cleanup.

Copy link
Contributor Author

Hm, fixing that race doesn't completely fix the problem. I did this:

diff --git a/context.go b/context.go
index 38467dd..5dbc535 100644
--- a/context.go
+++ b/context.go
@@ -6,6 +6,8 @@ import (
+       "sync/atomic"
+       "unsafe"
 type Face int
@@ -148,6 +150,13 @@ func edge(a, b, c Vector) float64 {
        return (b.X-c.X)*(a.Y-c.Y) - (b.Y-c.Y)*(a.X-c.X)
+func atomicLoadFloat64(p *float64) float64 {
+       return math.Float64frombits(atomic.LoadUint64((*uint64)(unsafe.Pointer(p))))
+func atomicStoreFloat64(p *float64, v float64) {
+       atomic.StoreUint64((*uint64)(unsafe.Pointer(p)), math.Float64bits(v))
 func (dc *Context) rasterize(v0, v1, v2 Vertex, s0, s1, s2 Vector) RasterizeInfo {
        var info RasterizeInfo
@@ -229,7 +238,7 @@ func (dc *Context) rasterize(v0, v1, v2 Vertex, s0, s1, s2 Vector) RasterizeInfo
                        z := b0*s0.Z + b1*s1.Z + b2*s2.Z
                        bz := z + dc.DepthBias
-                       if dc.ReadDepth && bz > dc.DepthBuffer[i] { // safe w/out lock?
+                       if dc.ReadDepth && bz > atomicLoadFloat64(&dc.DepthBuffer[i]) { // safe w/out lock? Nope.
                        // perspective-correct interpolation of vertex data
@@ -249,7 +258,7 @@ func (dc *Context) rasterize(v0, v1, v2 Vertex, s0, s1, s2 Vector) RasterizeInfo
                                if dc.WriteDepth {
                                        // update depth buffer
-                                       dc.DepthBuffer[i] = z
+                                       atomicStoreFloat64(&dc.DepthBuffer[i], z)
                                if dc.WriteColor {
                                        // update color buffer

Which makes the race detector happy. But the program still crashes in the ways described above.

Copy link

I forgot to tag the CLs properly (i.e. I was missing golang/go as a prefix). This benchmark is now gone.

Copy link
Contributor Author

I think this still may be demonstrating a runtime bug on Darwin.
Even if the benchmark is ~useless, it's a test case that should not crash this way.
(Or we could demonstrate that this benchmark has a different race condition, or uses unsafe incorrectly, etc.)

@randall77 randall77 reopened this Feb 16, 2023
Copy link

That's a good point, I missed that in your last reply.

@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 28, 2023
Copy link
Contributor Author

The benchmark has been removed from the benchmark suite. To reproduce you now need to checkout x/benchmarks at 2874b5b8249238c444b75dad5fe805f266f1488b.

I bisected to find the CL that fixed this issue at tip. It was . Its backport is already in for 1.20.2, so this should be fixed when 1.20.2 comes out.

I will leave this issue open and recheck after 1.20.2 is out.

@randall77 randall77 assigned randall77 and unassigned mknyszek Mar 2, 2023
Copy link

mknyszek commented Mar 2, 2023

Thanks Keith! The removal was perhaps a bit premature in that it made it harder to reproduce the issue. Sorry about that. :(

@gopherbot gopherbot modified the milestones: Go1.20.2, Go1.20.3 Mar 7, 2023
Copy link
Contributor Author

Ok, this looks fixed in 1.20.2.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 17, 2023
@dmitshur dmitshur modified the milestones: Go1.20.3, Go1.20.2 Mar 17, 2023
@golang golang locked and limited conversation to collaborators Mar 16, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
None yet

No branches or pull requests

4 participants