runtime: failure of sweet/fogleman-fauxgl on darwin/amd64 on go1.20

[randall77]

To reproduce:

git clone https://go.googlesource.com/benchmarks
cd benchmarks/sweet
go build ./cmd/sweet
echo "[[config]]" > config.toml
echo "  name = \"config120\"" >> config.toml
echo "  goroot = \"/Users/khr/go1.20\"" >> config.toml
./sweet run -run=fogleman-fauxgl -count=10 config.toml

Crashes in a bunch of random ways, all gc-adjacent. Zombie objects, 0xb01dfacedebac1e errors, driver segfaults, etc. (A few traces pasted below.)

Fails on both 1.20 and 1.20.1. Seems to work fine on both 1.19.4 and tip.
I'm running Ventura 13.0.1 (Darwin Kernel Version 22.1.0).
Seems to work fine on linux.

Possibly related to #54760 ? Seems somewhat different but the fix there turned this benchmark off on arm64. I wonder if darwin/arm64 fails also, but we need to reenable this benchmark on that platform to check.

@mknyszek @prattmic

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x222 pc=0x10fb8bb]

goroutine 21 [running]:
golang.org/x/benchmarks/sweet/benchmarks/internal/driver.init.func3.1()
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:54 +0x1b
golang.org/x/benchmarks/sweet/benchmarks/internal/driver.(*B).startRSSSampler.func1()
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:290 +0x13f
created by golang.org/x/benchmarks/sweet/benchmarks/internal/driver.(*B).startRSSSampler
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:280 +0xad

runtime: marked free object in span 0x13852c8, elemsize=416 freeindex=3 (bad use of unsafe.Pointer? try -d=checkptr)
0xc0000ca000 alloc marked  
0xc0000ca1a0 alloc marked  
0xc0000ca340 alloc marked  
0xc0000ca4e0 free  unmarked
0xc0000ca680 free  unmarked
0xc0000ca820 free  unmarked
0xc0000ca9c0 free  unmarked
0xc0000cab60 free  unmarked
0xc0000cad00 free  unmarked
0xc0000caea0 free  unmarked
0xc0000cb040 free  unmarked
0xc0000cb1e0 free  unmarked
0xc0000cb380 free  unmarked
0xc0000cb520 free  unmarked
0xc0000cb6c0 free  unmarked
0xc0000cb860 free  unmarked
0xc0000cba00 free  unmarked
0xc0000cbba0 free  unmarked
0xc0000cbd40 free  marked   zombie
0x000000c0000cbd40:  0x000000c0000cbd50  0x0000000001097dd7 <time.Now+0x0000000000000017> 
0x000000c0000cbd50:  0x0000000300000002  0x00000000010f921e <golang.org/x/benchmarks/sweet/benchmarks/internal/driver.(*B).StartTimer+0x00000000000000fe> 
0x000000c0000cbd60:  0x000000c0000cbd98  0x0000000001125754 <main.main.func1+0x0000000000000034> 
0x000000c0000cbd70:  0x000000000115ef01  0x0000000000000005 
0x000000c0000cbd80:  0x000000c0000d6008  0x000000c00007e000 
0x000000c0000cbd90:  0x000000c0000b8210  0x000000c0000cbef8 
0x000000c0000cbda0:  0x00000000010fae6d <golang.org/x/benchmarks/sweet/benchmarks/internal/driver.RunBenchmark+0x000000000000022d>  0x000000c0000b8210 
0x000000c0000cbdb0:  0x000000c000098330  0x000000000115ef01 
0x000000c0000cbdc0:  0x0000000000000004  0x0000000001716368 
0x000000c0000cbdd0:  0x000000000137a5b8  0x0000000000000004 
0x000000c0000cbde0:  0x0000000000000002  0x0000000000000003 
0x000000c0000cbdf0:  0x40051c963d442e7e  0x000000000115ef01 
0x000000c0000cbe00:  0x000000c00008c0c0  0xbfec261da7059352 
0x000000c0000cbe10:  0xbfec261da7059352  0x000000c0000b8210 
0x000000c0000cbe20:  0x000000000125d520  0xbfea12f684bda130 
0x000000c0000cbe30:  0xbfea12f684bda130  0xbfda12f684bda130 
0x000000c0000cbe40:  0x401471c71c71c71d  0xbfe5555555555555 
0x000000c0000cbe50:  0xbfe5555555555555  0xbfd5555555555555 
0x000000c0000cbe60:  0x4018000000000000  0xbfe6a09e667f3bcc 
0x000000c0000cbe70:  0x3fe6a09e667f3bcc  0x0000000000000000 
0x000000c0000cbe80:  0x8000000000000000  0xbfce2b7dddfefa65 
0x000000c0000cbe90:  0xbfce2b7dddfefa65  0x3fee2b7dddfefa65 
0x000000c0000cbea0:  0x8000000000000000  0x3fe5555555555555 
0x000000c0000cbeb0:  0x3fe5555555555555  0x3fd5555555555555 
0x000000c0000cbec0:  0xc018000000000000  0x0000000000000000 
0x000000c0000cbed0:  0x0000000000000000  0x0000000000000000 
fatal error: found pointer to free object

runtime stack:
runtime.throw({0x116494f?, 0xc0000cbee0?})
        /Users/khr/go1.20/src/runtime/panic.go:1047 +0x5d fp=0x7ff7bfeff488 sp=0x7ff7bfeff458 pc=0x10339bd
runtime.(*mspan).reportZombies(0x13852c8)
        /Users/khr/go1.20/src/runtime/mgcsweep.go:846 +0x2e5 fp=0x7ff7bfeff508 sp=0x7ff7bfeff488 pc=0x1023625
runtime.(*sweepLocked).sweep(0x0?, 0x0)
        /Users/khr/go1.20/src/runtime/mgcsweep.go:634 +0x9f6 fp=0x7ff7bfeff5f8 sp=0x7ff7bfeff508 pc=0x1022f96
runtime.(*mcentral).uncacheSpan(0x7ff7bfeff668?, 0x12654a0?)
        /Users/khr/go1.20/src/runtime/mcentral.go:228 +0xa5 fp=0x7ff7bfeff620 sp=0x7ff7bfeff5f8 pc=0x1015665
runtime.(*mcache).releaseAll(0x137a108)
        /Users/khr/go1.20/src/runtime/mcache.go:291 +0x145 fp=0x7ff7bfeff688 sp=0x7ff7bfeff620 pc=0x10150c5
runtime.(*mcache).prepareForSweep(0x137a108)
        /Users/khr/go1.20/src/runtime/mcache.go:328 +0x39 fp=0x7ff7bfeff6b0 sp=0x7ff7bfeff688 pc=0x10151b9
runtime.acquirep(0xc000026000)
        /Users/khr/go1.20/src/runtime/proc.go:5096 +0x2c fp=0x7ff7bfeff6c8 sp=0x7ff7bfeff6b0 pc=0x1040e2c
runtime.stopm()
        /Users/khr/go1.20/src/runtime/proc.go:2335 +0xb5 fp=0x7ff7bfeff6f8 sp=0x7ff7bfeff6c8 pc=0x103a4f5
runtime.findRunnable()
        /Users/khr/go1.20/src/runtime/proc.go:3007 +0xabc fp=0x7ff7bfeff800 sp=0x7ff7bfeff6f8 pc=0x103bd1c
runtime.schedule()
        /Users/khr/go1.20/src/runtime/proc.go:3360 +0xb1 fp=0x7ff7bfeff838 sp=0x7ff7bfeff800 pc=0x103cb51
runtime.park_m(0xc000006340?)
        /Users/khr/go1.20/src/runtime/proc.go:3511 +0x12d fp=0x7ff7bfeff868 sp=0x7ff7bfeff838 pc=0x103d06d
runtime.mcall()
        /Users/khr/go1.20/src/runtime/asm_amd64.s:452 +0x43 fp=0x7ff7bfeff878 sp=0x7ff7bfeff868 pc=0x1063283

goroutine 1 [runnable]:
runtime.gopark(0x1014070?, 0xe?, 0x90?, 0xe?, 0xb?)
        /Users/khr/go1.20/src/runtime/proc.go:381 +0xd6 fp=0xc000124608 sp=0xc0001245e8 pc=0x1036676
runtime.goparkunlock(...)
        /Users/khr/go1.20/src/runtime/proc.go:387
runtime.gcParkAssist()
        /Users/khr/go1.20/src/runtime/mgcmark.go:651 +0xeb fp=0xc000124640 sp=0xc000124608 pc=0x101c00b
runtime.gcAssistAlloc(0xc000006340)
        /Users/khr/go1.20/src/runtime/mgcmark.go:508 +0x23e fp=0xc0001246a0 sp=0xc000124640 pc=0x101ba1e
runtime.deductAssistCredit(0xc0067b3000?)
        /Users/khr/go1.20/src/runtime/malloc.go:1217 +0x5a fp=0xc0001246c8 sp=0xc0001246a0 pc=0x100cd9a
runtime.mallocgc(0x271000, 0x1134f40, 0x1)
        /Users/khr/go1.20/src/runtime/malloc.go:932 +0xd0 fp=0xc000124730 sp=0xc0001246c8 pc=0x100c5b0
runtime.makeslice(0x126a1e0?, 0x101c000124798?, 0x2958b5e8?)
        /Users/khr/go1.20/src/runtime/slice.go:103 +0x52 fp=0xc000124758 sp=0xc000124730 pc=0x104b412
image.NewRGBA({{0x0?, 0x0?}, {0x320?, 0x320?}})
        /Users/khr/go1.20/src/image/image.go:215 +0x65 fp=0xc0001247d0 sp=0xc000124758 pc=0x10beec5
github.com/nfnt/resize.Resize(0x320, 0x320, {0x118f5e0, 0xc0000a8040}, 0x1)
        /Users/khr/gopath/pkg/mod/github.com/nfnt/resize@v0.0.0-20180221191011-83c6a9932646/resize.go:144 +0x627 fp=0xc000124b80 sp=0xc0001247d0 pc=0x111b1a7
golang.org/x/benchmarks/third_party/fogleman-fauxgl.(*RotateAnimation).RenderNext(0xc00007e000)
        /Users/khr/gowork/tmp/benchmarks/third_party/fogleman-fauxgl/animate.go:77 +0x3f1 fp=0xc000124d70 sp=0xc000124b80 pc=0x1125211
main.main.func1(0xc0000b8210?)
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/fogleman-fauxgl/main.go:46 +0x5c fp=0xc000124da8 sp=0xc000124d70 pc=0x112577c
golang.org/x/benchmarks/sweet/benchmarks/internal/driver.RunBenchmark({0x116509a, 0x1e}, 0xc0000cbf58, {0x1261240, 0x8, 0xc0000f5f70?})
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:455 +0x22d fp=0xc000124f08 sp=0xc000124da8 pc=0x10fae6d
main.main()
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/fogleman-fauxgl/main.go:42 +0x1c9 fp=0xc000124f80 sp=0xc000124f08 pc=0x11256a9
runtime.main()
        /Users/khr/go1.20/src/runtime/proc.go:250 +0x207 fp=0xc000124fe0 sp=0xc000124f80 pc=0x1036247
runtime.goexit()
        /Users/khr/go1.20/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000124fe8 sp=0xc000124fe0 pc=0x10653a1

unexpected fault address 0xb01dfacedebac1e
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x10fb8bb]

goroutine 8 [running]:
runtime.throw({0x115f03f?, 0xc00007a0c0?})
        /Users/khr/go1.20/src/runtime/panic.go:1047 +0x5d fp=0xc00017de90 sp=0xc00017de60 pc=0x10339bd
runtime.sigpanic()
        /Users/khr/go1.20/src/runtime/signal_unix.go:851 +0x28a fp=0xc00017def0 sp=0xc00017de90 pc=0x104a48a
golang.org/x/benchmarks/sweet/benchmarks/internal/driver.init.func3.1()
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:54 +0x1b fp=0xc00017df08 sp=0xc00017def0 pc=0x10fb8bb
golang.org/x/benchmarks/sweet/benchmarks/internal/driver.(*B).startRSSSampler.func1()
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:290 +0x13f fp=0xc00017ffe0 sp=0xc00017df08 pc=0x10f997f
runtime.goexit()
        /Users/khr/go1.20/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00017ffe8 sp=0xc00017ffe0 pc=0x10653a1
created by golang.org/x/benchmarks/sweet/benchmarks/internal/driver.(*B).startRSSSampler
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:280 +0xad

goroutine 1 [chan receive]:
runtime.gopark(0xc000174a30?, 0x1017691?, 0x20?, 0x3a?, 0xc000174a98?)
        /Users/khr/go1.20/src/runtime/proc.go:381 +0xd6 fp=0xc000174a18 sp=0xc0001749f8 pc=0x1036676
runtime.chanrecv(0xc000134420, 0xc000174b08, 0x1)
        /Users/khr/go1.20/src/runtime/chan.go:583 +0x49d fp=0xc000174aa8 sp=0xc000174a18 pc=0x100611d
runtime.chanrecv1(0x1017691?, 0x10?)
        /Users/khr/go1.20/src/runtime/chan.go:442 +0x18 fp=0xc000174ad0 sp=0xc000174aa8 pc=0x1005c58
github.com/fogleman/fauxgl.(*Context).DrawTriangles(0xc00690c000, {0xc000180000, 0x371aa, 0x371aa})
        /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:430 +0x191 fp=0xc000174b38 sp=0xc000174ad0 pc=0x110e771
github.com/fogleman/fauxgl.(*Context).DrawMesh(0xc00690c000?, 0xc00010e000)
        /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:436 +0x31 fp=0xc000174b80 sp=0xc000174b38 pc=0x110e991
golang.org/x/benchmarks/third_party/fogleman-fauxgl.(*RotateAnimation).RenderNext(0xc000166000)
        /Users/khr/gowork/tmp/benchmarks/third_party/fogleman-fauxgl/animate.go:73 +0x3c5 fp=0xc000174d70 sp=0xc000174b80 pc=0x11251e5
main.main.func1(0xc000114210?)
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/fogleman-fauxgl/main.go:46 +0x5c fp=0xc000174da8 sp=0xc000174d70 pc=0x112577c
golang.org/x/benchmarks/sweet/benchmarks/internal/driver.RunBenchmark({0x116509a, 0x1e}, 0xc000127f58, {0x1261240, 0x8, 0xc000145f70?})
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/internal/driver/driver.go:455 +0x22d fp=0xc000174f08 sp=0xc000174da8 pc=0x10fae6d
main.main()
        /Users/khr/gowork/tmp/benchmarks/sweet/benchmarks/fogleman-fauxgl/main.go:42 +0x1c9 fp=0xc000174f80 sp=0xc000174f08 pc=0x11256a9
runtime.main()
        /Users/khr/go1.20/src/runtime/proc.go:250 +0x207 fp=0xc000174fe0 sp=0xc000174f80 pc=0x1036247
runtime.goexit()
        /Users/khr/go1.20/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000174fe8 sp=0xc000174fe0 pc=0x10653a1

[randall77]

GOGC=off fixes it.
GOMAXPROCS=1 makes it always fail in the driver segfault mode.

Reports a data race under the race detector. This may be the underlying issue.

==================
WARNING: DATA RACE
Read at 0x00c00bb874d0 by goroutine 44:
  github.com/fogleman/fauxgl.(*Context).rasterize()
      /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:232 +0x99c
  github.com/fogleman/fauxgl.(*Context).drawClippedTriangle()
      /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:347 +0xa49
  github.com/fogleman/fauxgl.(*Context).DrawTriangle()
      /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:387 +0x964
  github.com/fogleman/fauxgl.(*Context).DrawTriangles.func1()
      /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:421 +0xda
  github.com/fogleman/fauxgl.(*Context).DrawTriangles.func2()
      /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:426 +0x47

Previous write at 0x00c00bb874d0 by goroutine 50:
  github.com/fogleman/fauxgl.(*Context).rasterize()
      /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:252 +0x1068
  github.com/fogleman/fauxgl.(*Context).drawClippedTriangle()
      /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:347 +0xa49
  github.com/fogleman/fauxgl.(*Context).DrawTriangle()
      /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:387 +0x964
  github.com/fogleman/fauxgl.(*Context).DrawTriangles.func1()
      /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:421 +0xda
  github.com/fogleman/fauxgl.(*Context).DrawTriangles.func2()
      /Users/khr/gopath/pkg/mod/github.com/fogleman/fauxgl@v0.0.0-20200818143847-27cddc103802/context.go:426 +0x47

So just a bad benchmark, then?
In fact, at context.go:232, the code says:

			if dc.ReadDepth && bz > dc.DepthBuffer[i] { // safe w/out lock?

So, the answer might be no.

@fogleman

[mknyszek]

Yeah, that's probably it, it just has races. To be fair, I'm not totally sure this code is super well-maintained these days. We might want to consider just moving away from this benchmark. (Originally we ran it as sort of an interesting math/interface/etc. benchmark, but I'm not sure we ever got any super strong signal from it.)

I've been thinking about revamping the Sweet benchmarks a little and replacing a bunch of the benchmarks that don't seem to give us a strong signal one way or the other. We also have quite a few bent benchmarks that I think are noisy in unhelpful ways.

I'll open a bug for this cleanup.

[randall77]

Hm, fixing that race doesn't completely fix the problem. I did this:

diff --git a/context.go b/context.go
index 38467dd..5dbc535 100644
--- a/context.go
+++ b/context.go
@@ -6,6 +6,8 @@ import (
        "math"
        "runtime"
        "sync"
+       "sync/atomic"
+       "unsafe"
 )
 
 type Face int
@@ -148,6 +150,13 @@ func edge(a, b, c Vector) float64 {
        return (b.X-c.X)*(a.Y-c.Y) - (b.Y-c.Y)*(a.X-c.X)
 }
 
+func atomicLoadFloat64(p *float64) float64 {
+       return math.Float64frombits(atomic.LoadUint64((*uint64)(unsafe.Pointer(p))))
+}
+func atomicStoreFloat64(p *float64, v float64) {
+       atomic.StoreUint64((*uint64)(unsafe.Pointer(p)), math.Float64bits(v))
+}
+
 func (dc *Context) rasterize(v0, v1, v2 Vertex, s0, s1, s2 Vector) RasterizeInfo {
        var info RasterizeInfo
 
@@ -229,7 +238,7 @@ func (dc *Context) rasterize(v0, v1, v2 Vertex, s0, s1, s2 Vector) RasterizeInfo
                        info.TotalPixels++
                        z := b0*s0.Z + b1*s1.Z + b2*s2.Z
                        bz := z + dc.DepthBias
-                       if dc.ReadDepth && bz > dc.DepthBuffer[i] { // safe w/out lock?
+                       if dc.ReadDepth && bz > atomicLoadFloat64(&dc.DepthBuffer[i]) { // safe w/out lock? Nope.
                                continue
                        }
                        // perspective-correct interpolation of vertex data
@@ -249,7 +258,7 @@ func (dc *Context) rasterize(v0, v1, v2 Vertex, s0, s1, s2 Vector) RasterizeInfo
                                info.UpdatedPixels++
                                if dc.WriteDepth {
                                        // update depth buffer
-                                       dc.DepthBuffer[i] = z
+                                       atomicStoreFloat64(&dc.DepthBuffer[i], z)
                                }
                                if dc.WriteColor {
                                        // update color buffer

Which makes the race detector happy. But the program still crashes in the ways described above.

[mknyszek]

I forgot to tag the CLs properly (i.e. I was missing golang/go as a prefix). This benchmark is now gone.

[randall77]

I think this still may be demonstrating a runtime bug on Darwin.
Even if the benchmark is ~useless, it's a test case that should not crash this way.
(Or we could demonstrate that this benchmark has a different race condition, or uses unsafe incorrectly, etc.)

[mknyszek]

That's a good point, I missed that in your last reply.

[randall77]

The benchmark has been removed from the benchmark suite. To reproduce you now need to checkout x/benchmarks at 2874b5b8249238c444b75dad5fe805f266f1488b.

I bisected to find the CL that fixed this issue at tip. It was https://go-review.googlesource.com/c/go/+/467015 . Its backport is already in for 1.20.2, so this should be fixed when 1.20.2 comes out.

I will leave this issue open and recheck after 1.20.2 is out.

[mknyszek]

Thanks Keith! The removal was perhaps a bit premature in that it made it harder to reproduce the issue. Sorry about that. :(

[randall77]

Ok, this looks fixed in 1.20.2.