Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: memory corruption on darwin/amd64 #60449

Open
gopherbot opened this issue May 26, 2023 · 42 comments
Open

runtime: memory corruption on darwin/amd64 #60449

gopherbot opened this issue May 26, 2023 · 42 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone

Comments

@gopherbot
Copy link
Contributor

gopherbot commented May 26, 2023

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && pkg == "cmd/compile/internal/ssa") ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

Issue created automatically to collect these failures.

Example (log):

watchflakes

@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 26, 2023
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "exp" && section == "go test -short ./..." && test == ""
2023-05-22 17:56 darwin-amd64-12_0 exp@2e198f4a go@f0e356a0 (log)

watchflakes

@bcmills
Copy link
Contributor

bcmills commented Jun 13, 2023

Runtime heap corruption in cmd/go during go test. Looks like a runtime bug to me?

@bcmills bcmills added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jun 13, 2023
@mknyszek
Copy link
Contributor

@prattmic notes that this is a failure on Darwin and there's a fork/exec goroutine here. Sounds like an existing issue possibly (#59995)?

@mknyszek
Copy link
Contributor

We should probably aggregate such issues into a single issue.

@mknyszek mknyszek self-assigned this Jun 14, 2023
@bcmills
Copy link
Contributor

bcmills commented Jun 14, 2023

This failure mode looks substantially different from the other Darwin cluster described in #54461 (comment) (which includes #59995).

That cluster of failures is characterized by a goroutine stuck copying a pipe connected to a dead child process: either stdout, stderr, or (more rarely) the one created by syscall.forkExecPipe, but there is typically no sign of memory corruption. In contrast, the log attached to this issue shows memory corruption, and has a syscall.Wait4 frame on the stack that suggests that the child process is probably still alive.

So I think this issue is probably independent of #59995.

@mknyszek
Copy link
Contributor

Agreed. I changed my mind after posting the last message and looking at the other failures again, but then I had a meeting. :P

@mknyszek mknyszek added this to the Go1.21 milestone Jun 14, 2023
@mknyszek
Copy link
Contributor

I was going back through issues assigned to me (oops, I have a backlog) and I noticed that #58715 is also memory corruption, but none of those failures include forkExec. Just mentioning it here since it might be related.

@bcmills bcmills changed the title go test -short ./...: unrecognized failures runtime: found pointer to free object on darwin/amd64 Jun 20, 2023
@bcmills
Copy link
Contributor

bcmills commented Jun 20, 2023

#55167 (comment) looks like plausibly the same kind of corruption; updated the watchflakes pattern to catch it more specifically.

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && `fatal error: found pointer to free object`
2023-06-16 20:47 darwin-amd64-10_15 go@0183c1aa cmd/go.TestScript (log)
vcs-test.golang.org rerouted to http://127.0.0.1:54179
https://vcs-test.golang.org rerouted to https://127.0.0.1:54180
go test proxy running at GOPROXY=http://127.0.0.1:54181/mod
runtime: marked free object in span 0x1e6f868, elemsize=48 freeindex=106 (bad use of unsafe.Pointer? try -d=checkptr)
0xc000042000 alloc marked  
0xc000042030 alloc marked  
0xc000042060 alloc marked  
0xc000042090 alloc marked  
0xc0000420c0 alloc marked  
0xc0000420f0 alloc marked  
...
fatal error: found pointer to free object

runtime stack:
runtime.throw({0x17b70aa?, 0xc000043d10?})
	/tmp/buildlet/go/src/runtime/panic.go:1077 +0x5c fp=0x70000d701c50 sp=0x70000d701c20 pc=0x103b47c
runtime.(*mspan).reportZombies(0x1e6f868)
	/tmp/buildlet/go/src/runtime/mgcsweep.go:861 +0x2e5 fp=0x70000d701cd0 sp=0x70000d701c50 pc=0x102a745
runtime.(*sweepLocked).sweep(0x2030002?, 0x0)
	/tmp/buildlet/go/src/runtime/mgcsweep.go:644 +0xa88 fp=0x70000d701dd8 sp=0x70000d701cd0 pc=0x1029f68
runtime.(*mcentral).uncacheSpan(0x70000d701e50?, 0x1031946?)
...
	/tmp/buildlet/go/src/os/file_unix.go:315 +0x75 fp=0xc000856d10 sp=0xc000856cb8 pc=0x10b9815
os.(*File).Close(...)
	/tmp/buildlet/go/src/os/file_posix.go:23
os.WriteFile({0xc000b882a0?, 0x64?}, {0xc00080de81, 0xe, 0xf}, 0x18e2578?)
	/tmp/buildlet/go/src/os/file.go:766 +0x7d fp=0xc000856d58 sp=0xc000856d10 pc=0x10b819d
cmd/go/internal/script.(*State).ExtractFiles(0xc000e7ad00, 0xc000538a80?)
	/tmp/buildlet/go/src/cmd/go/internal/script/state.go:171 +0x1c9 fp=0xc000856e20 sp=0xc000856d58 pc=0x16421c9
cmd/go_test.TestScript.func3(0xc0005d7ba0)
	/tmp/buildlet/go/src/cmd/go/script_test.go:128 +0x2a5 fp=0xc000856f70 sp=0xc000856e20 pc=0x166ff25
testing.tRunner(0xc0005d7ba0, 0xc0004daeb0)

watchflakes

@bcmills bcmills changed the title runtime: found pointer to free object on darwin/amd64 runtime: pointer corruption on darwin/amd64 Jun 20, 2023
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap`)
2023-07-24 17:08 darwin-amd64-12_0 tools@67ba5997 go@df0a1297 x/tools/go/packages.TestAll (log)
runtime: pointer 0xc01477ee10 to unused region of span span.base()=0xc01477e000 span.limit=0xc01477ffe0 span.state=1
runtime: found in object at *(0xc0394e8000+0x17d8)
object=0xc0394e8000 s.base()=0xc039408000 s.limit=0xc03a370000 s.spanclass=0 s.elemsize=16154624 s.state=mSpanInUse
 *(object+0) = 0x4
 *(object+8) = 0x1350008
 *(object+16) = 0x14d37c0
 *(object+24) = 0x1351c90
 *(object+32) = 0xc037d90e38
 *(object+40) = 0x5
 *(object+48) = 0x13500f8
...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x12f6b3f?, 0x6?})
	/tmp/buildlet/go/src/runtime/panic.go:1077 +0x5c fp=0x700005b78d90 sp=0x700005b78d60 pc=0x10360dc
runtime.badPointer(0x5f407c8, 0x700005b78e10?, 0xc0394e8000, 0xc00791dfe0?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:321 +0x165 fp=0x700005b78de8 sp=0x700005b78d90 pc=0x1013885
runtime.findObject(0xc02fca9280?, 0x10563de?, 0x106c4a0?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:364 +0xa6 fp=0x700005b78e20 sp=0x700005b78de8 pc=0x1013a06
runtime.scanobject(0xc0394e8000, 0xc000030140)
...
	/tmp/buildlet/go/src/runtime/chan.go:442 +0x12 fp=0xc00010ac98 sp=0xc00010ac70 pc=0x1005f12
testing.(*T).Run(0xc00cc761a0, {0x12e60ab?, 0xc000171e20?}, 0xc00d581278)
	/tmp/buildlet/go/src/testing/testing.go:1649 +0x3c8 fp=0xc00010ad58 sp=0xc00010ac98 pc=0x10eec48
golang.org/x/tools/go/packages_test.testLoadDifferentPatterns(0xc00cc761a0, {0x1350d40, 0x150c880})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/overlay_test.go:997 +0x517 fp=0xc00010aed8 sp=0xc00010ad58 pc=0x1257a97
golang.org/x/tools/go/packages_test.testAllOrModulesParallel.func1(0xc00cc761a0, {0x1350d40, 0x150c880})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages_test.go:81 +0x138 fp=0xc00010af30 sp=0xc00010aed8 pc=0x1258f78
golang.org/x/tools/go/packages/packagestest.TestAll.func1(0xc00cc761a0)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packagestest/export.go:162 +0x51 fp=0xc00010af70 sp=0xc00010af30 pc=0x1247c91
testing.tRunner(0xc00cc761a0, 0xc00e8fd5c0)

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap` || `fatal error: workbuf is not empty`)
2023-06-23 21:36 darwin-amd64-10_15 telemetry@5a4cd7a1 go@bca81759 (log)
fatal error: workbuf is not empty

runtime stack:
runtime.throw({0x16519f6?, 0x700003f65e08?})
	/tmp/buildlet/go/src/runtime/panic.go:1047 +0x5d fp=0x700003f65de0 sp=0x700003f65db0 pc=0x103449d
runtime.(*workbuf).checkempty(0x4582128?)
	/tmp/buildlet/go/src/runtime/mgcwork.go:344 +0x36 fp=0x700003f65e00 sp=0x700003f65de0 pc=0x1025a16
runtime.getempty()
	/tmp/buildlet/go/src/runtime/mgcwork.go:357 +0x1e8 fp=0x700003f65e60 sp=0x700003f65e00 pc=0x1025c28
runtime.handoff(0xc001c81000)
	/tmp/buildlet/go/src/runtime/mgcwork.go:439 +0x25 fp=0x700003f65e90 sp=0x700003f65e60 pc=0x1025e25
runtime.(*gcWork).balance(0xc000035238)
	/tmp/buildlet/go/src/runtime/mgcwork.go:297 +0x45 fp=0x700003f65ea8 sp=0x700003f65e90 pc=0x1025905
runtime.gcDrain(0xc000035238, 0x3)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1085 +0x125 fp=0x700003f65f08 sp=0x700003f65ea8 pc=0x101e7e5
runtime.gcBgMarkWorker.func2()
	/tmp/buildlet/go/src/runtime/mgc.go:1348 +0xad fp=0x700003f65f58 sp=0x700003f65f08 pc=0x101af6d
runtime.systemstack()
	/tmp/buildlet/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x700003f65f60 sp=0x700003f65f58 pc=0x10655c9

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap` || `fatal error: workbuf is not empty`)
2023-07-27 16:59 darwin-amd64-12_0 tools@bacac149 go@77863e42 x/tools/gopls/test.TestStringChanges (log)
runtime: marked free object in span 0x6bf8728, elemsize=16 freeindex=10 (bad use of unsafe.Pointer? try -d=checkptr)
0xc00027c000 alloc marked  
0xc00027c010 alloc marked  
0xc00027c020 alloc marked  
0xc00027c030 alloc marked  
0xc00027c040 alloc marked  
0xc00027c050 alloc marked  
0xc00027c060 alloc marked  
0xc00027c070 alloc marked  
0xc00027c080 alloc marked  
...
fatal error: found pointer to free object

runtime stack:
runtime.throw({0x1248123?, 0xc00027d4d0?})
	/tmp/buildlet/go/src/runtime/panic.go:1077 +0x5c fp=0x700010f8b8d8 sp=0x700010f8b8a8 pc=0x103ac7c
runtime.(*mspan).reportZombies(0x6bf8728)
	/tmp/buildlet/go/src/runtime/mgcsweep.go:861 +0x2e5 fp=0x700010f8b958 sp=0x700010f8b8d8 pc=0x102a025
runtime.(*sweepLocked).sweep(0x700010f8ba90?, 0x0)
	/tmp/buildlet/go/src/runtime/mgcsweep.go:644 +0xa88 fp=0x700010f8ba60 sp=0x700010f8b958 pc=0x1029848
runtime.(*mcentral).uncacheSpan(0xc000042900?, 0x700010f8bab8?)
...
	/tmp/buildlet/gopath/pkg/mod/github.com/google/go-cmp@v0.5.9/cmp/report.go:47 +0x89 fp=0xc0002a3ca8 sp=0xc0002a3c38 pc=0x11b38a9
github.com/google/go-cmp/cmp.Diff({0x121a3c0, 0xc00018ea80}, {0x121a3c0, 0xc00018ee00}, {0xc0001c2fc0?, 0x1252abc?, 0xc000736000?})
	/tmp/buildlet/gopath/pkg/mod/github.com/google/go-cmp@v0.5.9/cmp/compare.go:130 +0x16f fp=0xc0002a3d08 sp=0xc0002a3ca8 pc=0x11aa0ef
golang.org/x/tools/gopls/test_test.tryChange(0x133f, 0x1348, {0x1240e31, 0x2})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/test/json_test.go:134 +0x366 fp=0xc0002a3e38 sp=0xc0002a3d08 pc=0x11dd026
golang.org/x/tools/gopls/test_test.allDeltas(0xc0000f0340, {0xc0000185a0, 0x12, 0x100000001?}, {0xc0002a3f40, 0x2, 0xfc90b1b56b?})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/test/json_test.go:94 +0x136 fp=0xc0002a3f00 sp=0xc0002a3e38 pc=0x11dcb16
golang.org/x/tools/gopls/test_test.TestStringChanges(0x0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/test/json_test.go:67 +0x11c fp=0xc0002a3f70 sp=0xc0002a3f00 pc=0x11dc73c
testing.tRunner(0xc0000f0340, 0x1253838)

watchflakes

@gopherbot gopherbot modified the milestones: Go1.21, Go1.22 Aug 8, 2023
@mknyszek mknyszek changed the title runtime: pointer corruption on darwin/amd64 runtime: memory corruption on darwin/amd64 Aug 23, 2023
@mknyszek
Copy link
Contributor

These represent more general runtime memory corruption, but I suspect this is all part of the same issue.

2023-08-21 23:20 darwin-amd64-11_0 go@e3c80f57 cmd/go.TestScript (log)
2023-08-22 08:38 darwin-amd64-longtest go@05f95115 runtime.TestEINTR (log)

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap` || `fatal error: workbuf is not empty` || `schedule: holding locks` || `fatal error: bad summary data`)
2023-08-21 23:20 darwin-amd64-11_0 go@e3c80f57 cmd/go.TestScript (log)
vcs-test.golang.org rerouted to http://127.0.0.1:53045
https://vcs-test.golang.org rerouted to https://127.0.0.1:53046
go test proxy running at GOPROXY=http://127.0.0.1:53047/mod
--- FAIL: TestScript (1.07s)
    --- FAIL: TestScript/mod_list_test (1.11s)
        script_test.go:132: 2023-08-21T23:59:53Z
        script_test.go:134: $WORK=/tmp/buildlet/tmp/cmd-go-test-340886821/tmpdir633785442/mod_list_test250661805
        script_test.go:156: 
            > env GO111MODULE=on
            # go list -compiled -test must handle test-only packages
...
            fatal error: bad summary data

            runtime stack:
            runtime.throw({0x880bda5?, 0x7000100efed0?})
            	/tmp/buildlet/go/src/runtime/panic.go:1018 +0x5c fp=0x7000100efe08 sp=0x7000100efdd8 pc=0x81c693c
            runtime.(*pageAlloc).alloc(0x8e82ca8, 0x4)
            	/tmp/buildlet/go/src/runtime/mpagealloc.go:896 +0x26a fp=0x7000100efe50 sp=0x7000100efe08 pc=0x81bbe0a
            runtime.(*mheap).allocSpan(0x8e82ca0, 0x4, 0x0, 0x8a?)
            	/tmp/buildlet/go/src/runtime/mheap.go:1247 +0x192 fp=0x7000100efef0 sp=0x7000100efe50 pc=0x81b8052
            runtime.(*mheap).alloc.func1()
...
            cmd/go/internal/work.ActorFunc.Act(0x8a73460?, 0x8e99720?, {0x8a73460?, 0x8e99720?}, 0x0?)
            	/tmp/buildlet/go/src/cmd/go/internal/work/action.go:78 +0x2d fp=0xc00009fd70 sp=0xc00009fd40 pc=0x870282d
            cmd/go/internal/work.(*Builder).Do.func3({0x8a73460, 0x8e99720}, 0xc000479080)
            	/tmp/buildlet/go/src/cmd/go/internal/work/exec.go:151 +0x80c fp=0xc00009ff20 sp=0xc00009fd70 pc=0x871106c
            cmd/go/internal/work.(*Builder).Do.func4()
            	/tmp/buildlet/go/src/cmd/go/internal/work/exec.go:219 +0xb9 fp=0xc00009ffe0 sp=0xc00009ff20 pc=0x8710699
            runtime.goexit()
            	/tmp/buildlet/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00009ffe8 sp=0xc00009ffe0 pc=0x81fd701
            created by cmd/go/internal/work.(*Builder).Do in goroutine 1
            	/tmp/buildlet/go/src/cmd/go/internal/work/exec.go:205 +0x3fe
2023-08-22 08:38 darwin-amd64-longtest go@05f95115 runtime.TestEINTR (log)
fatal error: schedule: holding locks

runtime stack:
runtime.throw({0x632a2a8?, 0x66cf960?})
	/tmp/buildlet/go/src/runtime/panic.go:1018 +0x5c fp=0x7ff7baf93388 sp=0x7ff7baf93358 pc=0x5faa69c
runtime.schedule()
	/tmp/buildlet/go/src/runtime/proc.go:3577 +0x276 fp=0x7ff7baf933c0 sp=0x7ff7baf93388 pc=0x5fb50d6
runtime.park_m(0xc0004cf520?)
	/tmp/buildlet/go/src/runtime/proc.go:3765 +0x12d fp=0x7ff7baf93408 sp=0x7ff7baf933c0 pc=0x5fb546d
runtime.mcall()
...
	/tmp/buildlet/go/src/os/exec/exec.go:590 +0x2d fp=0xc00018cd70 sp=0xc00018cd50 pc=0x60b0eed
os/exec.(*Cmd).CombinedOutput(0xc000610000)
	/tmp/buildlet/go/src/os/exec/exec.go:977 +0x94 fp=0xc00018cd98 sp=0xc00018cd70 pc=0x60b2b34
runtime_test.runBuiltTestProg(0xc000824d00, {0xc00002c500, 0x34}, {0x631da22, 0x5}, {0x0, 0x0, 0x0?})
	/tmp/buildlet/go/src/runtime/crash_test.go:76 +0x327 fp=0xc00018ce80 sp=0xc00018cd98 pc=0x6265ae7
runtime_test.runTestProg(0xc000824d00, {0x6320719, 0xb}, {0x631da22, 0x5}, {0x0, 0x0, 0x0})
	/tmp/buildlet/go/src/runtime/crash_test.go:59 +0x19e fp=0xc00018cef0 sp=0xc00018ce80 pc=0x626575e
runtime_test.TestEINTR(0xc000824d00)
	/tmp/buildlet/go/src/runtime/crash_cgo_test.go:736 +0x5c fp=0xc00018cf70 sp=0xc00018cef0 pc=0x62647dc
testing.tRunner(0xc000824d00, 0x6482498)

watchflakes

@mknyszek
Copy link
Contributor

#62165 may be related.

@mknyszek
Copy link
Contributor

Actually, it almost certainly is the same memory corruption. Folding it in.

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2023-11-29 15:35 darwin-amd64-11_0 tools@147b88d0 go@6111acc8 x/tools/go/packages (log)
SIGSEGV: segmentation violation
PC=0x8140827 m=12 sigcode=1 addr=0x0

goroutine 0 gp=0xc00edb4680 m=12 mp=0xc00137d008 [idle]:
runtime.spanOfUnchecked(...)
	/tmp/buildlet/go/src/runtime/mheap.go:709
runtime.scanobject(0xc000031268?, 0xc000031268?)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1407 +0x47 fp=0x700010391ea0 sp=0x700010391e10 pc=0x8140827
runtime.gcDrain(0xc000031268, 0x2)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1242 +0x1f4 fp=0x700010391f08 sp=0x700010391ea0 pc=0x81401f4
...
r11    0x4c
r12    0x700010391e90
r13    0x8190fe1
r14    0xc00edb4680
r15    0x1
rip    0x8140827
rflags 0x10246
cs     0x2b
fs     0x0
gs     0x0

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2023-12-07 19:37 darwin-amd64-11_0 pkgsite@00698da1 go@9b4b3e5a (log)
runtime: pointer 0xc000ced4b9 to unused region of span span.base()=0xc000cea000 span.limit=0xc000cf9f00 span.state=1
runtime: found in object at *(0xc001265d60+0x0)
object=0xc001265d60 s.base()=0xc001264000 s.limit=0xc001265fe0 s.spanclass=14 s.elemsize=80 s.state=mSpanInUse
 *(object+0) = 0xc000ced4b9 <==
 *(object+8) = 0x6
 *(object+16) = 0xc0004047e8
 *(object+24) = 0x8
 *(object+32) = 0xc00133e0a0
 *(object+40) = 0x1d
 *(object+48) = 0xc0000280af
...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0xf8793f0?, 0x6?})
	runtime/panic.go:1023 +0x5c fp=0x700008ed5d88 sp=0x700008ed5d58 pc=0xf2f73dc
runtime.badPointer(0x16cb2980, 0xc000ced4b9, 0xc001265d60, 0x0)
	runtime/mbitmap.go:286 +0x165 fp=0x700008ed5dd8 sp=0x700008ed5d88 pc=0xf2d4e05
runtime.findObject(0xc00002a080?, 0xc001265d60?, 0xf2e8545?)
	runtime/mbitmap.go:329 +0xa6 fp=0x700008ed5e10 sp=0x700008ed5dd8 pc=0xf2d4f86
runtime.scanobject(0xc00003f768?, 0xc00003f768)
...
os/exec.(*Cmd).writerDescriptor.func1()
	os/exec/exec.go:577 +0x34 fp=0xc001a85f60 sp=0xc001a85f00 pc=0xf3e04f4
os/exec.(*Cmd).Start.func2(0xc0005382c0?)
	os/exec/exec.go:724 +0x2c fp=0xc001a85fc8 sp=0xc001a85f60 pc=0xf3e118c
os/exec.(*Cmd).Start.gowrap1()
	os/exec/exec.go:736 +0x24 fp=0xc001a85fe0 sp=0xc001a85fc8 pc=0xf3e1124
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc001a85fe8 sp=0xc001a85fe0 pc=0xf331c21
created by os/exec.(*Cmd).Start in goroutine 7451
	os/exec/exec.go:723 +0x9ab

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2023-12-18 20:56 darwin-amd64-13 tools@ee47fd47 go@c564d4ae (log)

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-01-08 18:46 darwin-amd64-12_0 build@21e06540 go@59ecebf4 (log)

watchflakes

@bcmills
Copy link
Contributor

bcmills commented Jan 10, 2024

I wonder if this might be due to an uninitialized pointer field in a syscall result. That could explain the bad pointers, at least, but I guess it doesn't explain the other failure modes.

@bcmills
Copy link
Contributor

bcmills commented Jan 11, 2024

darwin/amd64 is supposed to be a first class port.

@golang/runtime, should this be marked as a release-blocker for Go 1.22?

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2023-12-11 23:10 darwin-amd64-nocgo tools@f40889dc go@f4ff0e25 x/tools/go/analysis/passes/deepequalerrors (log)
SIGSEGV: segmentation violation
PC=0xe7c69a5 m=7 sigcode=1 addr=0x0

goroutine 0 gp=0xc0002081c0 m=7 mp=0xc000204008 [idle]:
runtime.(*mheap).allocNeedsZero(0xe7a96f6?, 0x800?, 0xc004000000?)
	/tmp/buildlet/go/src/runtime/mheap.go:1024 +0x65 fp=0x70000176ee38 sp=0x70000176ee18 pc=0xe7c69a5
runtime.(*mheap).initSpan(0xeca5740, 0x1586cdc8, 0x0, 0xc, 0xc004010000, 0x1)
	/tmp/buildlet/go/src/runtime/mheap.go:1381 +0x51 fp=0x70000176ee70 sp=0x70000176ee38 pc=0xe7c71b1
runtime.(*mheap).allocSpan(0xeca5740, 0x1, 0x0, 0xc)
	/tmp/buildlet/go/src/runtime/mheap.go:1340 +0x43d fp=0x70000176ef10 sp=0x70000176ee70 pc=0xe7c6fbd
...
r11    0x0
r12    0x80c00407ffff
r13    0xec7bbe0
r14    0xc0002081c0
r15    0x3
rip    0xe7c69a5
rflags 0x10246
cs     0x2b
fs     0x0
gs     0x0

watchflakes

@mknyszek
Copy link
Contributor

darwin/amd64 is supposed to be a first class port.

@golang/runtime, should this be marked as a release-blocker for Go 1.22?

I may be misunderstanding the policy, but I think technically it's not a release-blocker since we released Go 1.21 with the same issue, so it's not new to this release. (The fact that we released Go 1.21 without figuring out what it is is on it's own something that we should maybe revisit, but I think that's where we are today.) I don't want to be the sole judge on this so I'll add it to the next weekly meeting and/or revisit this in triage.

@cherrymui spent a good bit of time investigating this and had a lot of difficulty reproducing. (Mentioning that here because I don't think that was recorded in this issue.) The crash rate is also fairly low, so it's difficult to continue to prioritize this against other things that need fixing, especially since the amount of time this bug could take to resolve is unknown and likely high. Again, not trying to make a decision, just trying to rationalize how we got here.

The other aspect to this is that it doesn't seem to have impacted downstream users much, if at all. It's possible that users are just ignoring it, but IIRC other Darwin issues that appeared on the builders have been reported in the past, such as the CPU profile related one.

This makes me wonder if maybe it's our postsubmit Mac machines specifically that are the problem. It also occurs to me that I haven't seen the same kinds of failures on the LUCI infrastructure. Those builders have been up long enough that I would've expected to see at least one such failure by now. They have a few failures from other unique sources, but I haven't seen anything like this yet.

I will say that it's possible the failure got missed on the LUCI builders because watchflakes doesn't run against the LUCI infrastructure yet. At the very least I can do an ad-hoc-but-systematic analysis of all our LUCI darwin builds and see if anything crops up. If not, then that's an interesting data point.

@mknyszek mknyszek removed their assignment Jan 11, 2024
@bcmills
Copy link
Contributor

bcmills commented Jan 17, 2024

Previously, the darwin failures in #49692 were resolved by some combination of OS updates and perhaps better ventilation of the builder machines.

Obviously we don't have much control over the ventilation of Amazon's EC2 Mac Minis, though. It's unfortunate that so few Macs support ECC memory. 😒

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-01-18 23:21 darwin-amd64-13 build@911ff433 go@2540b143 (log)

watchflakes

@cherrymui
Copy link
Member

cherrymui commented Jan 19, 2024

On the main repo, main branch, watchflakes doesn't find any failure matching this for most recent builds in 45 days. I could run it for longer time period and see.

Update: also no match on LUCI builders in a longer period of time.

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-01-22 20:52 darwin-amd64-11_0 tools@e2ca5942 go@a95136a8 x/tools/go/analysis/passes/stdmethods.TestAnalyzeEncodingXML (log)
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1026791]

runtime stack:
runtime.throw({0x12e4d8e?, 0x14b5700?})
	/tmp/buildlet/go/src/runtime/panic.go:1047 +0x5d fp=0x7ffeefbff6c8 sp=0x7ffeefbff698 pc=0x103535d
runtime.sigpanic()
	/tmp/buildlet/go/src/runtime/signal_unix.go:825 +0x3e9 fp=0x7ffeefbff728 sp=0x7ffeefbff6c8 pc=0x104c409
runtime.(*mheap).allocNeedsZero(0xc00c100000?, 0x7c?, 0x180?)
	/tmp/buildlet/go/src/runtime/mheap.go:1021 +0x71 fp=0x7ffeefbff748 sp=0x7ffeefbff728 pc=0x1026791
...
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:758 +0xd77 fp=0xc00650dc48 sp=0xc00650d9d8 pc=0x1258317
golang.org/x/tools/go/packages.Load(0xc0003c12c0?, {0xc007dd4a70, 0x1, 0x1})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:249 +0x2c7 fp=0xc00650dcc0 sp=0xc00650dc48 pc=0x12566e7
golang.org/x/tools/go/analysis/analysistest.loadPackages(0x14b03e0, {0x0, 0x0}, {0xc007dd4a70, 0x1, 0x1})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/analysis/analysistest/analysistest.go:375 +0x29a fp=0xc00650de50 sp=0xc00650dcc0 pc=0x126361a
golang.org/x/tools/go/analysis/analysistest.Run({0x13375a0, 0xc007abb860}, {0x0, 0x0}, 0x14b03e0, {0xc007dd4a70, 0x1, 0x1})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/analysis/analysistest/analysistest.go:322 +0xbf fp=0xc00650df18 sp=0xc00650de50 pc=0x1262fbf
golang.org/x/tools/go/analysis/passes/stdmethods_test.TestAnalyzeEncodingXML(0x100523d?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/analysis/passes/stdmethods/stdmethods_test.go:20 +0x5f fp=0xc00650df70 sp=0xc00650df18 pc=0x126673f
testing.tRunner(0xc007abb860, 0x12ec930)

watchflakes

@gopherbot gopherbot modified the milestones: Go1.22, Go1.23 Feb 6, 2024
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-02-06 21:16 darwin-amd64-12_0 tools@0d875898 go@f2920803 x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckStd (log)
serve.go:441: debug server listening at http://localhost:49266
serve.go:441: debug server listening at http://localhost:49267
#### Start Gopls Test Logs for "TestRunGovulncheckStd/default"
[Trace - 20:03:18.284 PM] Sending request 'initialize - (1)'.
Params: {"processId":0,"rootUri":"","capabilities":{"workspace":{"workspaceEdit":{"resourceOperations":["rename"]},"didChangeConfiguration":{},"didChangeWatchedFiles":{"dynamicRegistration":true},"configuration":true},"textDocument":{"completion":{"completionItem":{"snippetSupport":true,"tagSupport":{"valueSet":[1]}}},"documentSymbol":{"hierarchicalDocumentSymbolSupport":true},"codeAction":{"codeActionLiteralSupport":{"codeActionKind":{"valueSet":null}}},"publishDiagnostics":{},"semanticTokens":{"requests":{"full":true},"tokenTypes":["namespace","type","class","enum","interface","struct","typeParameter","parameter","variable","property","enumMember","event","function","method","macro","keyword","modifier","comment","string","number","regexp","operator"],"tokenModifiers":["declaration","definition","readonly","static","deprecated","abstract","async","modification","documentation","defaultLibrary"],"formats":null}},"window":{"workDoneProgress":true}},"initializationOptions":{"codelenses":{"run_govulncheck":true},"completionBudget":"0s","diagnosticsDelay":"10ms","env":{"GO111MODULE":"","GOMODCACHE":"","GOPACKAGESDRIVER":"off","GOPATH":"/tmp/buildlet/tmp/gopls-test-76965345/TestRunGovulncheckStd/default/gopath","GOPROXY":"file:///tmp/buildlet/tmp/gopls-test-76965345/TestRunGovulncheckStd/default/proxy","GOSUMDB":"off","GOVULNDB":"file:///tmp/buildlet/tmp/vulndb-test2317938991/ID","_GOPLS_TEST_BINARY_RUN_AS_GOPLS":"true","_GOPLS_TEST_VULNCHECK_GOVERSION":"go1.18"},"verboseWorkDoneProgress":true},"trace":"messages","workspaceFolders":[{"uri":"file:///tmp/buildlet/tmp/gopls-test-76965345/TestRunGovulncheckStd/default/work","name":"work"}]}


[Trace - 20:03:18.284 PM] Received response 'initialize - (1)' in 0ms.
Result: {"capabilities":{"textDocumentSync":{"openClose":true,"change":2,"save":{}},"completionProvider":{"triggerCharacters":["."]},"hoverProvider":true,"signatureHelpProvider":{"triggerCharacters":["(",","]},"definitionProvider":true,"typeDefinitionProvider":true,"implementationProvider":true,"referencesProvider":true,"documentHighlightProvider":true,"documentSymbolProvider":true,"codeActionProvider":true,"codeLensProvider":{},"documentLinkProvider":{},"workspaceSymbolProvider":true,"documentFormattingProvider":true,"renameProvider":true,"foldingRangeProvider":true,"selectionRangeProvider":true,"executeCommandProvider":{"commands":["gopls.add_dependency","gopls.add_import","gopls.add_telemetry_counters","gopls.apply_fix","gopls.change_signature","gopls.check_upgrades","gopls.diagnose_files","gopls.edit_go_directive","gopls.fetch_vulncheck_result","gopls.gc_details","gopls.generate","gopls.go_get_package","gopls.list_imports","gopls.list_known_packages","gopls.maybe_prompt_for_telemetry","gopls.mem_stats","gopls.regenerate_cgo","gopls.remove_dependency","gopls.reset_go_mod_diagnostics","gopls.run_go_work_command","gopls.run_govulncheck","gopls.run_tests","gopls.start_debugging","gopls.start_profile","gopls.stop_profile","gopls.test","gopls.tidy","gopls.toggle_gc_details","gopls.update_go_sum","gopls.upgrade_dependency","gopls.vendor","gopls.views","gopls.workspace_stats"]},"callHierarchyProvider":true,"semanticTokensProvider":{"legend":{"tokenTypes":["namespace","type","class","enum","interface","struct","typeParameter","parameter","variable","property","enumMember","event","function","method","macro","keyword","modifier","comment","string","number","regexp","operator"],"tokenModifiers":["declaration","definition","readonly","static","deprecated","abstract","async","modification","documentation","defaultLibrary"]},"range":true,"full":true},"inlayHintProvider":{},"workspace":{"workspaceFolders":{"supported":true,"changeNotifications":"workspace/didChangeWorkspaceFolders"}}},"serverInfo":{"name":"gopls","version":"{\"GoVersion\":\"go1.21.7\",\"Path\":\"\",\"Main\":{\"Path\":\"\",\"Version\":\"\",\"Sum\":\"\",\"Replace\":null},\"Deps\":null,\"Settings\":null,\"Version\":\"(unknown)\"}"}}

...
--- FAIL: TestRunGovulncheckStd (3.56s)
    --- FAIL: TestRunGovulncheckStd/default (3.53s)
        vuln_test.go:212: waiting on:
            Unmeetable: once "completed work for token 680114519661641662" is met, must have:
            received window/showMessage containing "Found GOSTDLIB"
            no diagnostics for file "go.mod"

            err:condition has final verdict Unmeetable

            state:
...
            #### diagnostics:
            	go.mod (version 1):

            #### outstanding work:
            #### completed work:
            	govulncheck: 1
            	diagnosing initial workspace load: 1
            	Setting up workspace: 1
            	Checking telemetry prompt: 1
            	diagnosing opened files: 1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-02-13 14:38 darwin-amd64-11_0 tools@1b39a8b6 go@1b541502 x/tools/go/packages (log)
runtime: pointer 0xc004212000 to unallocated span span.base()=0xc004212000 span.limit=0xc004213fe0 span.state=0
runtime: found in object at *(0xc015497560+0x58)
object=0xc015497560 s.base()=0xc015496000 s.limit=0xc015497fe0 s.spanclass=16 s.elemsize=96 s.state=mSpanInUse
 *(object+0) = 0xc01540d500
 *(object+8) = 0x0
 *(object+16) = 0x0
 *(object+24) = 0x0
 *(object+32) = 0x0
 *(object+40) = 0x0
 *(object+48) = 0x0
...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x177fe8d?, 0x6?})
	/tmp/buildlet/go/src/runtime/panic.go:1021 +0x5c fp=0x70000644cd88 sp=0x70000644cd58 pc=0x152179c
runtime.badPointer(0x8f10308, 0xc004212000, 0xc015497560, 0x58)
	/tmp/buildlet/go/src/runtime/mbitmap.go:286 +0x165 fp=0x70000644cdd8 sp=0x70000644cd88 pc=0x14fdde5
runtime.findObject(0xc02575eb40?, 0xc015497560?, 0x1511665?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:329 +0xa6 fp=0x70000644ce10 sp=0x70000644cdd8 pc=0x14fdf66
runtime.scanobject(0xc000031268?, 0xc000031268)
...
golang.org/x/tools/go/packages.(*loader).loadRecursive(0xc022737ce0?, 0xc000a217d0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:822 +0xf fp=0xc000a217a0 sp=0xc000a21768 pc=0x17330cf
golang.org/x/tools/go/packages.(*loader).loadRecursive.func1.1(0x0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:829 +0x26 fp=0xc000a217c8 sp=0xc000a217a0 pc=0x1733386
golang.org/x/tools/go/packages.(*loader).loadRecursive.func1.gowrap1()
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:831 +0x24 fp=0xc000a217e0 sp=0xc000a217c8 pc=0x1733324
runtime.goexit({})
	/tmp/buildlet/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000a217e8 sp=0xc000a217e0 pc=0x155b1c1
created by golang.org/x/tools/go/packages.(*loader).loadRecursive.func1 in goroutine 7427
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:828 +0x94

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && pkg == "cmd/compile/internal/ssa") ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-03-27 17:55 darwin-amd64-nocgo go@979b34b3 cmd/compile/internal/ssa [build] (log)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x29 pc=0xc022ed3]

goroutine 34 gp=0xc000104c40 m=0 mp=0xc5076a0 [running]:
panic({0xc2ff3e0?, 0xc4f4360?})
	runtime/panic.go:778 +0x168 fp=0xc000029570 sp=0xc0000294c0 pc=0xbfcbe08
runtime.panicmem(...)
	runtime/panic.go:261
runtime.sigpanic()
	runtime/signal_unix.go:881 +0x378 fp=0xc0000295d0 sp=0xc000029570 pc=0xbfe4ad8
...
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func2(0xc4fef80)
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:333 +0x19e fp=0xc002a85fa0 sp=0xc002a85ea0 pc=0xc2065de
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func3.1(0x5454545454545454?)
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:404 +0x23 fp=0xc002a85fc8 sp=0xc002a85fa0 pc=0xc206403
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func3.gowrap1()
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:406 +0x24 fp=0xc002a85fe0 sp=0xc002a85fc8 pc=0xc2063a4
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc002a85fe8 sp=0xc002a85fe0 pc=0xc003dc1
created by cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func3 in goroutine 1
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:403 +0x47

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && pkg == "cmd/compile/internal/ssa") ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-05-02 22:38 darwin-amd64-11_0 tools@ccdef3cc go@93d8777d x/tools/gopls/internal/test/integration/completion (log)
SIGSEGV: segmentation violation
PC=0x4b07827 m=5 sigcode=1 addr=0x0

goroutine 0 gp=0xc0001041c0 m=5 mp=0xc000100008 [idle]:
runtime.spanOfUnchecked(...)
	/tmp/buildlet/go/src/runtime/mheap.go:709
runtime.scanobject(0xc00005a168?, 0xc00005a168?)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1407 +0x47 fp=0x700011677e80 sp=0x700011677df0 pc=0x4b07827
runtime.gcDrain(0xc00005a168, 0x2)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1242 +0x1f4 fp=0x700011677ee8 sp=0x700011677e80 pc=0x4b071f4
...
r11    0xd0
r12    0x700011677e70
r13    0xc000100530
r14    0xc0001041c0
r15    0x1
rip    0x4b07827
rflags 0x10246
cs     0x2b
fs     0x0
gs     0x0

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && pkg == "cmd/compile/internal/ssa") ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-05-09 15:16 darwin-amd64-11_0 tools@24f3b32f go@dd6dee48 x/tools/go/packages (log)
runtime: pointer 0xc0137b4be0 to unused region of span span.base()=0xc0137b4000 span.limit=0xc0137b6000 span.state=1
runtime: found in object at *(0xc00c0bc000+0xeec8)
object=0xc00c0bc000 s.base()=0xc00c09c000 s.limit=0xc00c410000 s.spanclass=0 s.elemsize=3620864 s.state=mSpanInUse
 *(object+0) = 0xc00f600540
 *(object+8) = 0xc00089c660
 *(object+16) = 0x0
 *(object+24) = 0x0
 *(object+32) = 0x0
 *(object+40) = 0xfe7b8f8
 *(object+48) = 0xc00d9653e0
...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0xfda67d6?, 0x6?})
	/tmp/buildlet/go/src/runtime/panic.go:1023 +0x48 fp=0x70000bf27d88 sp=0x70000bf27d58 pc=0xfb42928
runtime.badPointer(0x172b25a8, 0xc0137b4be0, 0xc00c0bc000, 0xeec8)
	/tmp/buildlet/go/src/runtime/mbitmap.go:1247 +0x165 fp=0x70000bf27dd8 sp=0x70000bf27d88 pc=0xfb1ffc5
runtime.findObject(0xc007ad77a0?, 0xc00df2eae0?, 0xfb64a5e?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:1290 +0xa6 fp=0x70000bf27e10 sp=0x70000bf27dd8 pc=0xfb20146
runtime.scanobject(0xc000037c50?, 0xc000037c50)
...
golang.org/x/tools/go/packages.(*loader).loadRecursive(0xc006a2eee8?, 0xc000a787d0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:924 +0x3b fp=0xc000a787a0 sp=0xc000a78768 pc=0xfd598fb
golang.org/x/tools/go/packages.(*loader).loadRecursive.func1.1(0x0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:931 +0x26 fp=0xc000a787c8 sp=0xc000a787a0 pc=0xfd59b66
golang.org/x/tools/go/packages.(*loader).loadRecursive.func1.gowrap1()
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:933 +0x24 fp=0xc000a787e0 sp=0xc000a787c8 pc=0xfd59b04
runtime.goexit({})
	/tmp/buildlet/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc000a787e8 sp=0xc000a787e0 pc=0xfb7d7e1
created by golang.org/x/tools/go/packages.(*loader).loadRecursive.func1 in goroutine 7646
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:930 +0x94

watchflakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
Status: In Progress
Status: No status
Development

No branches or pull requests

5 participants