runtime: `SIGSEGV` via go test with -race -cover and -tags netgo on alpine linux/arm64

[GeorgeMac]

unexpected fault address 0x1412000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1412000 pc=0xa7e50]

goroutine 7 [running]:
runtime.throw({0x28990c?, 0x0?})
	/usr/local/go/src/runtime/panic.go:1047 +0x40 fp=0xc000204b40 sp=0xc000204b10 pc=0x72540
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:851 +0x1a8 fp=0xc000204b80 sp=0xc000204b40 pc=0x8a638
racecallatomic()
	/usr/local/go/src/runtime/race_arm64.s:351 +0x10 fp=0xc000204ba0 sp=0xc000204b90 pc=0xa7e50
sync/atomic.AddInt32()
	/usr/local/go/src/runtime/race_arm64.s:286 +0x18 fp=0xc000204bb0 sp=0xc000204ba0 pc=0xa8248
sync/atomic.AddUint32(0x83d91a7?, 0xdbba11c8?)
	<autogenerated>:1 +0x18 fp=0xc000204be0 sp=0xc000204bb0 pc=0xa96d8
go.flipt.io/flipt/internal/storage/oplock/memory.(*Service).TryAcquire(0xc00004b0b0, {0xc000204de8?, 0x0?}, {0x289720, 0x4}, 0x1?)
	/src/internal/storage/oplock/memory/memory.go:43 +0x280 fp=0xc000204d80 sp=0xc000204be0 pc=0x22b5b0
go.flipt.io/flipt/internal/storage/oplock/testing.Harness.func1()
	/src/internal/storage/oplock/testing/testing.go:50 +0x150 fp=0xc000204f20 sp=0xc000204d80 pc=0x22b200
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x70 fp=0xc000204fd0 sp=0xc000204f20 pc=0x22a450
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000204fd0 sp=0xc000204fd0 pc=0xa7134
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0x108

What version of Go are you using (go version)?

$ go version
go version go1.20.2 linux/arm64

Does this issue reproduce with the latest release?

Yes. Infact I can't reproduce it on go 1.20.1

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm64"
GOVCS=""
GOVERSION="go1.20.2"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/src/go.mod"
GOWORK="/src/go.work"
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3327561268=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I can comfortably reproduce it in this particular open source package:
https://github.com/flipt-io/flipt/tree/main/internal/storage/oplock/memory

I can consistently reproduce a SIGSEV when I perform go test -cover -race -tags netgo ..

It requires all three of these flags to be present to trigger it.

What did you expect to see?

No segfault. Works for golang:1.20.1-alpine3.16 docker image. However, golang:1.20.2-alpine3:16 fails consistently with the error seen above.

What did you see instead?

A SIGSEGV fatal error.

[ianlancetaylor]

CC @thanm

[thanm]

Thanks for the report.

I tried reproducing this on our linux/arm64 builder, but so far no luck.

It is a bit surprising that you're seeing this only with Go 1.20.2 and not Go 1.20.1. There is a coverage related bugfix in that sequence of commits (202a1a5706 ... aee9a19c55) but it is in the data reader, has nothing to do with the runtime.

I looked at the assembly we're generating for the function in question (go.flipt.io/flipt/internal/storage/oplock/memory.(*Service).TryAcquire)and I don't see any immediate red flags.

Not sure what to do next to debug this, without a way to repro the crash.

[GeorgeMac]

Hmmm intruiging. The only other thing of note was that I could only repro this on alpine in particular.
Im definitely way out of my depth, but could the whole Alpine uses libc-musl be a part of the equation?
Could that be a difference to the linux/arm64 builder you used?

[bcmills]

We do have a linux-amd64-alpine builder. @thanm, you might try a gomote on that and see if it reproduces there.

[bcmills]

(Probably not, though, what with it being amd64 instead of arm64. 🤷‍♂️)

[bcmills]

@GeorgeMac, I suspect that this could be a bad interaction between coverage flushing and a race in your test.

The test function can exit the goroutine here without ever waiting for the errgroup result:
https://github.com/flipt-io/flipt/blob/f04ddb10cdd98cec122ca3242a67743bec557eef/internal/storage/oplock/testing/testing.go#L77

If it takes that exit path, then it seems plausible that the goroutine in the errgroup may race with the test's main completing and flushing coverage results. Maybe that triggers some other bug in coverage reporting?

[bcmills]

It would be helpful to see what other goroutines are running at the time of the crash.

[GeorgeMac]

Here are all the goroutine stacks from the crash (below), is there something more I can do to get you information on other goroutines at the time of the crash?

I just changed the require to an assert too (to avoid exiting before the Wait) and I get the same crash:

unexpected fault address 0x1412000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1412000 pc=0xa7e50]

goroutine 36 [running]:
runtime.throw({0x28990c?, 0x0?})
	/usr/local/go/src/runtime/panic.go:1047 +0x40 fp=0xc000041b40 sp=0xc000041b10 pc=0x72540
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:851 +0x1a8 fp=0xc000041b80 sp=0xc000041b40 pc=0x8a638
racecallatomic()
	/usr/local/go/src/runtime/race_arm64.s:351 +0x10 fp=0xc000041ba0 sp=0xc000041b90 pc=0xa7e50
sync/atomic.AddInt32()
	/usr/local/go/src/runtime/race_arm64.s:286 +0x18 fp=0xc000041bb0 sp=0xc000041ba0 pc=0xa8248
sync/atomic.AddUint32(0x6fa63f5?, 0xdbbe4017?)
	<autogenerated>:1 +0x18 fp=0xc000041be0 sp=0xc000041bb0 pc=0xa96d8
go.flipt.io/flipt/internal/storage/oplock/memory.(*Service).TryAcquire(0xc00011b080, {0xc000041de8?, 0x0?}, {0x289720, 0x4}, 0x1?)
	/src/internal/storage/oplock/memory/memory.go:43 +0x280 fp=0xc000041d80 sp=0xc000041be0 pc=0x22b480
go.flipt.io/flipt/internal/storage/oplock/testing.Harness.func1()
	/src/internal/storage/oplock/testing/testing.go:50 +0x150 fp=0xc000041f20 sp=0xc000041d80 pc=0x22b0d0
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x70 fp=0xc000041fd0 sp=0xc000041f20 pc=0x22a320
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000041fd0 sp=0xc000041fd0 pc=0xa7134
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0x108

goroutine 1 [chan receive]:
runtime.gopark(0x522e70?, 0xffffffff?, 0x8?, 0x77?, 0x49a2c?)
	/usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0xc000187670 sp=0xc000187650 pc=0x74fa0
runtime.chanrecv(0xc0001763f0, 0xc00018778e, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x408 fp=0xc000187710 sp=0xc000187670 pc=0x42b48
runtime.chanrecv1(0x461060?, 0x2466a0?)
	/usr/local/go/src/runtime/chan.go:442 +0x14 fp=0xc000187740 sp=0xc000187710 pc=0x42704
testing.(*T).Run(0xc00010ed00, {0x28b812, 0xc}, 0x2b4b08)
	/usr/local/go/src/testing/testing.go:1630 +0x5d4 fp=0xc000187880 sp=0xc000187740 pc=0x180b64
testing.runTests.func1(0x0?)
	/usr/local/go/src/testing/testing.go:2036 +0x84 fp=0xc0001878e0 sp=0xc000187880 pc=0x183ab4
testing.tRunner(0xc00010ed00, 0xc000187b38)
	/usr/local/go/src/testing/testing.go:1576 +0x184 fp=0xc0001879f0 sp=0xc0001878e0 pc=0x17fa14
testing.runTests(0xc000131180?, {0x456330, 0x1, 0x1}, {0xc000187c08?, 0x49a2c?, 0x461420?})
	/usr/local/go/src/testing/testing.go:2034 +0x6ac fp=0xc000187b60 sp=0xc0001879f0 pc=0x18394c
testing.(*M).Run(0xc000131180)
	/usr/local/go/src/testing/testing.go:1906 +0x8e4 fp=0xc000187ec0 sp=0xc000187b60 pc=0x1819c4
main.main()
	_testmain.go:78 +0x2d4 fp=0xc000187f70 sp=0xc000187ec0 pc=0x22bfa4
runtime.main()
	/usr/local/go/src/runtime/proc.go:250 +0x200 fp=0xc000187fd0 sp=0xc000187f70 pc=0x74b80
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000187fd0 sp=0xc000187fd0 pc=0xa7134

goroutine 2 [force gc (idle)]:
runtime.gopark(0x461c40?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0xc0000327a0 sp=0xc000032780 pc=0x74fa0
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc0000327d0 sp=0xc0000327a0 pc=0x74df0
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc0000327d0 sp=0xc0000327d0 pc=0xa7134
created by runtime.init.6
	/usr/local/go/src/runtime/proc.go:293 +0x24

goroutine 18 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0xc000040f60 sp=0xc000040f40 pc=0x74fa0
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:278 +0x98 fp=0xc000040fb0 sp=0xc000040f60 pc=0x5fca8
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:178 +0x28 fp=0xc000040fd0 sp=0xc000040fb0 pc=0x54d88
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000040fd0 sp=0xc000040fd0 pc=0xa7134
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:178 +0x6c

goroutine 19 [GC scavenge wait]:
runtime.gopark(0xc00008e000?, 0x2f2b30?, 0x1?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0xc000104f50 sp=0xc000104f30 pc=0x74fa0
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x4614a0)
	/usr/local/go/src/runtime/mgcscavenge.go:400 +0x5c fp=0xc000104f80 sp=0xc000104f50 pc=0x5dc5c
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:628 +0x44 fp=0xc000104fb0 sp=0xc000104f80 pc=0x5e174
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:179 +0x28 fp=0xc000104fd0 sp=0xc000104fb0 pc=0x54d28
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000104fd0 sp=0xc000104fd0 pc=0xa7134
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:179 +0xac

goroutine 34 [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0xc000046d80 sp=0xc000046d60 pc=0x74fa0
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x124 fp=0xc000046fd0 sp=0xc000046d80 pc=0x53e54
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000046fd0 sp=0xc000046fd0 pc=0xa7134
created by runtime.createfing
	/usr/local/go/src/runtime/mfinal.go:163 +0x80

goroutine 35 [chan receive]:
runtime.gopark(0x0?, 0xc000202030?, 0x18?, 0x0?, 0x4718?)
	/usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0xc000045b40 sp=0xc000045b20 pc=0x74fa0
runtime.chanrecv(0xc000134240, 0xc000045e00, 0x1)
	/usr/local/go/src/runtime/chan.go:583 +0x408 fp=0xc000045be0 sp=0xc000045b40 pc=0x42b48
runtime.chanrecv2(0x0?, 0x1?)
	/usr/local/go/src/runtime/chan.go:447 +0x14 fp=0xc000045c10 sp=0xc000045be0 pc=0x42724
go.flipt.io/flipt/internal/storage/oplock/testing.Harness(0xc00010eea0, {0x2f4f50?, 0xc00011b080})
	/src/internal/storage/oplock/testing/testing.go:66 +0x5f0 fp=0xc000045e40 sp=0xc000045c10 pc=0x22aab0
go.flipt.io/flipt/internal/storage/oplock/memory.Test_Harness(0x0?)
	/src/internal/storage/oplock/memory/memory_test.go:10 +0xf8 fp=0xc000045e90 sp=0xc000045e40 pc=0x22b918
testing.tRunner(0xc00010eea0, 0x2b4b08)
	/usr/local/go/src/testing/testing.go:1576 +0x184 fp=0xc000045fa0 sp=0xc000045e90 pc=0x17fa14
testing.(*T).Run.func1()
	/usr/local/go/src/testing/testing.go:1629 +0x44 fp=0xc000045fd0 sp=0xc000045fa0 pc=0x180d54
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000045fd0 sp=0xc000045fd0 pc=0xa7134
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1629 +0x5b8

goroutine 37 [sync.Mutex.Lock]:
runtime.gopark(0x496b40?, 0xffffaf070f40?, 0x68?, 0xb?, 0x4f150?)
	/usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0xc000100ad0 sp=0xc000100ab0 pc=0x74fa0
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.semacquire1(0xc00011b084, 0x80?, 0x3, 0x1, 0xbc?)
	/usr/local/go/src/runtime/sema.go:160 +0x20c fp=0xc000100b30 sp=0xc000100ad0 pc=0x878ac
sync.runtime_SemacquireMutex(0xc000100b88?, 0x3c?, 0xc000100bb8?)
	/usr/local/go/src/runtime/sema.go:77 +0x28 fp=0xc000100b70 sp=0xc000100b30 pc=0xa3328
sync.(*Mutex).lockSlow(0xc00011b080)
	/usr/local/go/src/sync/mutex.go:171 +0x1e8 fp=0xc000100bc0 sp=0xc000100b70 pc=0xc1ce8
sync.(*Mutex).Lock(0xc00011b080)
	/usr/local/go/src/sync/mutex.go:90 +0x60 fp=0xc000100be0 sp=0xc000100bc0 pc=0xc1ad0
go.flipt.io/flipt/internal/storage/oplock/memory.(*Service).TryAcquire(0xc00011b080, {0xc000100de8?, 0x0?}, {0x289720, 0x4}, 0x1?)
	/src/internal/storage/oplock/memory/memory.go:29 +0xa4 fp=0xc000100d80 sp=0xc000100be0 pc=0x22b2a4
go.flipt.io/flipt/internal/storage/oplock/testing.Harness.func1()
	/src/internal/storage/oplock/testing/testing.go:50 +0x150 fp=0xc000100f20 sp=0xc000100d80 pc=0x22b0d0
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x70 fp=0xc000100fd0 sp=0xc000100f20 pc=0x22a320
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000100fd0 sp=0xc000100fd0 pc=0xa7134
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0x108

goroutine 38 [sync.Mutex.Lock]:
runtime.gopark(0x465cc0?, 0x10?, 0x60?, 0xc0?, 0x49144?)
	/usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0xc000047ad0 sp=0xc000047ab0 pc=0x74fa0
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.semacquire1(0xc00011b084, 0x80?, 0x3, 0x1, 0x4c?)
	/usr/local/go/src/runtime/sema.go:160 +0x20c fp=0xc000047b30 sp=0xc000047ad0 pc=0x878ac
sync.runtime_SemacquireMutex(0xc000047b88?, 0x3c?, 0xc000047bb8?)
	/usr/local/go/src/runtime/sema.go:77 +0x28 fp=0xc000047b70 sp=0xc000047b30 pc=0xa3328
sync.(*Mutex).lockSlow(0xc00011b080)
	/usr/local/go/src/sync/mutex.go:171 +0x1e8 fp=0xc000047bc0 sp=0xc000047b70 pc=0xc1ce8
sync.(*Mutex).Lock(0xc00011b080)
	/usr/local/go/src/sync/mutex.go:90 +0x60 fp=0xc000047be0 sp=0xc000047bc0 pc=0xc1ad0
go.flipt.io/flipt/internal/storage/oplock/memory.(*Service).TryAcquire(0xc00011b080, {0xc000047de8?, 0x0?}, {0x289720, 0x4}, 0x1?)
	/src/internal/storage/oplock/memory/memory.go:29 +0xa4 fp=0xc000047d80 sp=0xc000047be0 pc=0x22b2a4
go.flipt.io/flipt/internal/storage/oplock/testing.Harness.func1()
	/src/internal/storage/oplock/testing/testing.go:50 +0x150 fp=0xc000047f20 sp=0xc000047d80 pc=0x22b0d0
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x70 fp=0xc000047fd0 sp=0xc000047f20 pc=0x22a320
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000047fd0 sp=0xc000047fd0 pc=0xa7134
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0x108

goroutine 39 [sync.Mutex.Lock]:
runtime.gopark(0x0?, 0x0?, 0x18?, 0x5b?, 0x49144?)
	/usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0xc000105ad0 sp=0xc000105ab0 pc=0x74fa0
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.semacquire1(0xc00011b084, 0x80?, 0x3, 0x1, 0xcc?)
	/usr/local/go/src/runtime/sema.go:160 +0x20c fp=0xc000105b30 sp=0xc000105ad0 pc=0x878ac
sync.runtime_SemacquireMutex(0xc000105b88?, 0x3c?, 0xc000105bb8?)
	/usr/local/go/src/runtime/sema.go:77 +0x28 fp=0xc000105b70 sp=0xc000105b30 pc=0xa3328
sync.(*Mutex).lockSlow(0xc00011b080)
	/usr/local/go/src/sync/mutex.go:171 +0x1e8 fp=0xc000105bc0 sp=0xc000105b70 pc=0xc1ce8
sync.(*Mutex).Lock(0xc00011b080)
	/usr/local/go/src/sync/mutex.go:90 +0x60 fp=0xc000105be0 sp=0xc000105bc0 pc=0xc1ad0
go.flipt.io/flipt/internal/storage/oplock/memory.(*Service).TryAcquire(0xc00011b080, {0xc000105de8?, 0x0?}, {0x289720, 0x4}, 0x1?)
	/src/internal/storage/oplock/memory/memory.go:29 +0xa4 fp=0xc000105d80 sp=0xc000105be0 pc=0x22b2a4
go.flipt.io/flipt/internal/storage/oplock/testing.Harness.func1()
	/src/internal/storage/oplock/testing/testing.go:50 +0x150 fp=0xc000105f20 sp=0xc000105d80 pc=0x22b0d0
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x70 fp=0xc000105fd0 sp=0xc000105f20 pc=0x22a320
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000105fd0 sp=0xc000105fd0 pc=0xa7134
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0x108

goroutine 40 [sync.Mutex.Lock]:
runtime.gopark(0xc000101b18?, 0x48f18?, 0x18?, 0x1b?, 0x49144?)
	/usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0xc000101ad0 sp=0xc000101ab0 pc=0x74fa0
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.semacquire1(0xc00011b084, 0x80?, 0x3, 0x1, 0x6e?)
	/usr/local/go/src/runtime/sema.go:160 +0x20c fp=0xc000101b30 sp=0xc000101ad0 pc=0x878ac
sync.runtime_SemacquireMutex(0xc000101b88?, 0x3c?, 0xc000101bb8?)
	/usr/local/go/src/runtime/sema.go:77 +0x28 fp=0xc000101b70 sp=0xc000101b30 pc=0xa3328
sync.(*Mutex).lockSlow(0xc00011b080)
	/usr/local/go/src/sync/mutex.go:171 +0x1e8 fp=0xc000101bc0 sp=0xc000101b70 pc=0xc1ce8
sync.(*Mutex).Lock(0xc00011b080)
	/usr/local/go/src/sync/mutex.go:90 +0x60 fp=0xc000101be0 sp=0xc000101bc0 pc=0xc1ad0
go.flipt.io/flipt/internal/storage/oplock/memory.(*Service).TryAcquire(0xc00011b080, {0xc000101de8?, 0x0?}, {0x289720, 0x4}, 0x1?)
	/src/internal/storage/oplock/memory/memory.go:29 +0xa4 fp=0xc000101d80 sp=0xc000101be0 pc=0x22b2a4
go.flipt.io/flipt/internal/storage/oplock/testing.Harness.func1()
	/src/internal/storage/oplock/testing/testing.go:50 +0x150 fp=0xc000101f20 sp=0xc000101d80 pc=0x22b0d0
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x70 fp=0xc000101fd0 sp=0xc000101f20 pc=0x22a320
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0xc000101fd0 sp=0xc000101fd0 pc=0xa7134
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0x108
FAIL	go.flipt.io/flipt/internal/storage/oplock/memory	2.048s
FAIL