cmd/dist,x/build/cmd/coordinator: test logs missing errors or final status

[bcmills]

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|fatal error:|panic:|WARNING: DATA RACE|go tool dist: |(Test .* ran over .* limit)|([^ ]*make.bash: .*(error|fault)))`)

(pulled out from #57995 (comment))

[gopherbot]

Sorry, but there were parse errors in the watch flakes script.
The script I found was:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|go tool dist: |Test ".*" ran over .* limit`)

And the problems were:

script:2.33: invalid regexp: error parsing regexp: missing closing ): `(?m)^(All tests passed|FAIL|build failed|go tool dist: |Test ".*" ran over .* limit`

See https://go.dev/wiki/Watchflakes for details.

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|go tool dist: |Test .* ran over .* limit)`)

2023-06-24 02:06 freebsd-arm-paulzhol go@65092835 (log)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|go tool dist: |Test .* ran over .* limit)`)

2023-05-16 20:09 linux-amd64-racecompile go@ad6dcf9d internal/itoa [build] (log)

==================
WARNING: DATA RACE
Write at 0x00c00018c470 by goroutine 11:
  sync/atomic.CompareAndSwapInt32()
      ../src/runtime/race_amd64.s:310 +0xb
  ??()
      -:0 +0x4b8c58
  cmd/internal/obj.(*Link).LookupABIInit()
      ../src/cmd/internal/obj/sym.go:102 +0xb3
  cmd/compile/internal/base.linksym()
...

Goroutine 10 (running) created at:
  cmd/compile/internal/gc.compileFunctions.func3()
      ../src/cmd/compile/internal/gc/compile.go:165 +0x331

Goroutine 11 (finished) created at:
  cmd/compile/internal/gc.compileFunctions.func3()
      ../src/cmd/compile/internal/gc/compile.go:165 +0x331
==================
Found 2 data race(s)

2023-05-16 20:09 linux-amd64-racecompile go@ad6dcf9d internal/race [build] (log)

==================
WARNING: DATA RACE
Write at 0x00c000104470 by goroutine 13:
  sync/atomic.CompareAndSwapInt32()
      ../src/runtime/race_amd64.s:310 +0xb
  ??()
      -:0 +0x4b8c58
  cmd/internal/obj.(*Link).LookupInit()
      ../src/cmd/internal/obj/sym.go:126 +0x47
  cmd/internal/obj.(*Link).GCLocalsSym()
...

Goroutine 20 (running) created at:
  cmd/compile/internal/gc.compileFunctions.func3()
      ../src/cmd/compile/internal/gc/compile.go:165 +0x331

Goroutine 17 (running) created at:
  cmd/compile/internal/gc.compileFunctions.func3()
      ../src/cmd/compile/internal/gc/compile.go:165 +0x331
==================
Found 11 data race(s)

2023-05-16 20:09 linux-amd64-racecompile go@ad6dcf9d runtime/internal/syscall [build] (log)

==================
WARNING: DATA RACE
Write at 0x00c000214470 by goroutine 14:
  sync/atomic.CompareAndSwapInt32()
      ../src/runtime/race_amd64.s:310 +0xb
  ??()
      -:0 +0x4b8c58
  cmd/internal/obj.(*Link).LookupABIInit()
      ../src/cmd/internal/obj/sym.go:102 +0xb3
  cmd/compile/internal/base.linksym()
...

Goroutine 16 (running) created at:
  cmd/compile/internal/gc.compileFunctions.func3()
      ../src/cmd/compile/internal/gc/compile.go:165 +0x331

Goroutine 15 (finished) created at:
  cmd/compile/internal/gc.compileFunctions.func3()
      ../src/cmd/compile/internal/gc/compile.go:165 +0x331
==================
Found 15 data race(s)

2023-05-24 23:57 plan9-arm go@2a4f4fc5 (log)

fatal error: uncaching span but s.allocCount == 0

runtime stack:
runtime.throw({0x1450b8, 0x24})
	/boot/workdir/go1.4/src/runtime/panic.go:1198 +0x5c
runtime.(*mcentral).uncacheSpan(0x2148e4, 0x10741c78)
	/boot/workdir/go1.4/src/runtime/mcentral.go:193 +0x12c
runtime.(*mcache).releaseAll(0x225088)
	/boot/workdir/go1.4/src/runtime/mcache.go:279 +0x154
runtime.(*mcache).prepareForSweep(0x225088)
	/boot/workdir/go1.4/src/runtime/mcache.go:317 +0x3c
runtime.acquirep(0x10832000)
	/boot/workdir/go1.4/src/runtime/proc.go:5141 +0x2c
runtime.stopm()
	/boot/workdir/go1.4/src/runtime/proc.go:2409 +0x8c
runtime.exitsyscall0(0x10a750e0)
	/boot/workdir/go1.4/src/runtime/proc.go:4115 +0x1d0
runtime.mcall()
	/boot/workdir/go1.4/src/runtime/asm_arm.s:266 +0x48

2023-06-06 21:23 plan9-arm go@1aaf1b21 (log)

panic: send on closed channel

2023-06-07 02:51 freebsd-arm-paulzhol go@ea51e92c (log)

unexpected fault address 0x4d6149
fatal error: fault
[signal SIGBUS: bus error code=0x3 addr=0x4d6149 pc=0x1609c0]

goroutine 4945 [running]:
panic during panic
SIGBUS: bus error
PC=0x7cbd8 m=9 sigcode=3

/tmp/workdir-host-freebsd-arm-paulzhol/go/src/make.bash: line 226:  1079 Bus error               ./cmd/dist/dist bootstrap -a $vflag $GO_DISTFLAGS "$@"

2023-06-14 01:36 freebsd-arm-paulzhol go@c3db64c0 (log)

unexpected fault address 0x4d616f
fatal error: fault
[signal SIGBUS: bus error code=0x3 addr=0x4d616f pc=0x1613bc]

goroutine 4912 [running]:
panic during panic
SIGBUS: bus error
PC=0x7cffc m=8 sigcode=3

/tmp/workdir-host-freebsd-arm-paulzhol/go/src/make.bash: line 226:  1071 Bus error               ./cmd/dist/dist bootstrap -a $vflag $GO_DISTFLAGS "$@"

2023-06-15 23:07 freebsd-arm-paulzhol go@3e7ec131 (log)

/tmp/workdir-host-freebsd-arm-paulzhol/go/src/make.bash: line 226:  1071 Bus error               ./cmd/dist/dist bootstrap -a $vflag $GO_DISTFLAGS "$@"

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|fatal error:|panic:|WARNING: DATA RACE|go tool dist: |(Test .* ran over .* limit)|([^ ]*make.bash: .*error))`)

2023-06-20 17:19 freebsd-arm-paulzhol go@ebbff91f (log)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|fatal error:|panic:|WARNING: DATA RACE|go tool dist: |(Test .* ran over .* limit)|([^ ]*make.bash: .*error))`)

2023-11-02 10:34 freebsd-arm-paulzhol go@54452b96 (log)

/tmp/workdir-host-freebsd-arm-paulzhol/go/src/make.bash: line 226:  1155 Segmentation fault      ./cmd/dist/dist bootstrap -a $vflag $GO_DISTFLAGS "$@"

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|fatal error:|panic:|WARNING: DATA RACE|go tool dist: |(Test .* ran over .* limit)|([^ ]*make.bash: .*error))`)

2023-11-10 15:50 windows-arm64-11 go@9d914015 TestTraceCPUProfile (log)

--- FAIL: TestTraceCPUProfile (2.64s)
    --- FAIL: TestTraceCPUProfile/Default (2.62s)
        reader_test.go:65: unexpected error while reading the trace: expected a proc but didn't have one
        trace_test.go:567: found bad trace; dumping to test log...
        trace_test.go:568: Trace Go1.22
            EventBatch gen=1 m=2144 time=23479457578 size=65458
            ProcStart dt=1 p=2 p_seq=2
            ProcStop dt=1
            ProcStart dt=1 p=2 p_seq=3
            ProcStop dt=1
...
            GoStart dt=1 g=1 g_seq=3938
            GoStop dt=1 reason_string=20 stack=14
            GoStart dt=1 g=1 g_seq=3939
            GoStop dt=1 reason_string=20 stack=14
            GoStart dt=1 g=1 g_seq=3940
            GoStop dt=1 reason_string=20 stack=14
            GoStart dt=1 g=1 g_seq=3941
            GoStop dt=1 reason_st

... log truncated ...

2023-11-10 15:51 windows-arm64-11 go@3b303fa9 TestTraceCPUProfile (log)

--- FAIL: TestTraceCPUProfile (2.70s)
    --- FAIL: TestTraceCPUProfile/Default (2.68s)
        reader_test.go:65: unexpected error while reading the trace: per-M streams are out-of-order
        trace_test.go:567: found bad trace; dumping to test log...
        trace_test.go:568: Trace Go1.22
            EventBatch gen=1 m=9936 time=23496448275 size=65455
            ProcStart dt=1 p=0 p_seq=3
            ProcStop dt=1
            ProcStart dt=1 p=0 p_seq=4
            ProcStop dt=1
...
            GoStop dt=1 reason_string=20 stack=16
            GoStart dt=1 g=1 g_seq=19041
            GoStop dt=1 reason_string=20 stack=16
            GoStart dt=1 g=1 g_seq=19042
            GoStop dt=1 reason_string=20 stack=16
            GoStart dt=1 g=1 g_seq=19043
            GoStop dt=1 reason_string=20 stack=16


... log truncated ...

2023-11-10 16:18 windows-arm64-11 go@abf84221 TestTraceCPUProfile (log)

--- FAIL: TestTraceCPUProfile (2.63s)
    --- FAIL: TestTraceCPUProfile/Default (2.62s)
        reader_test.go:65: unexpected error while reading the trace: per-M streams are out-of-order
        trace_test.go:569: found bad trace; dumping to test log...
        trace_test.go:570: Trace Go1.22
            EventBatch gen=1 m=1420 time=27049777477 size=65457
            ProcStart dt=1 p=0 p_seq=1
            ProcStop dt=1
            ProcStart dt=1 p=0 p_seq=2
            ProcStop dt=1
...
            ProcStop dt=1
            ProcStart dt=1 p=1 p_seq=2698
            ProcStop dt=1
            ProcStart dt=1 p=1 p_seq=2699
            ProcStop dt=1
            ProcStart dt=1 p=1 p_seq=2700
            ProcStop dt=1
            ProcStart dt=1 p=

... log truncated ...

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|fatal error:|panic:|WARNING: DATA RACE|go tool dist: |(Test .* ran over .* limit)|([^ ]*make.bash: .*error))`)

2023-11-13 22:12 freebsd-arm64-dmgk go@773039ed TestTraceCPUProfile (log)

--- FAIL: TestTraceCPUProfile (4.69s)
    --- FAIL: TestTraceCPUProfile/Default (4.65s)
        trace_test.go:177: execution trace did not include all CPU profile samples; 29 in profile, 21 in trace
        trace_test.go:187: execution trace did not include all CPU profile samples for stack "main.cpuHog0:130|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"; 18 in profile, 12 in trace
        trace_test.go:187: execution trace did not include all CPU profile samples for stack "main.cpuHog0:125|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"; 4 in profile, 3 in trace
        trace_test.go:195: CPU profile included 1 samples at stack "runtime/internal/atomic.(*Uint32).CompareAndSwap:236|runtime.casgstatus:1135|runtime.execute:3051|runtime.schedule:3862|runtime.goschedImpl:4009|runtime.gosched_m:4014|runtime.mcall:192" not present in execution trace
        trace_test.go:202: execution trace CPU samples:
        trace_test.go:204: 1: "runtime.traceLocker.GoStop:425|runtime.traceLocker.GoSched:415|runtime.goschedImpl:3995|runtime.gosched_m:4014|runtime.mcall:192"
        trace_test.go:204: 1: "runtime.duffzero:8|runtime.traceStack:40|runtime.traceLocker.stack:166|runtime.traceLocker.GoStop:425|runtime.traceLocker.GoSched:415|runtime.goschedImpl:3995|runtime.gosched_m:4014|runtime.mcall:192"
        trace_test.go:204: 1: "runtime/internal/atomic.(*Uint32).Load:194|runtime.(*traceSchedResourceState).statusWasTraced:191|runtime.traceLocker.eventWriter:113|runtime.traceLocker.GoStop:425|runtime.traceLocker.GoSched:415|runtime.goschedImpl:3995|runtime.gosched_m:4014|runtime.mcall:192"
...
            GoStop dt=89 reason_string=20 stack=13
            GoStart dt=47 g=1 g_seq=13463
            GoStop dt=87 reason_string=20 stack=13
            GoStart dt=48 g=1 g_seq=13464
            GoStop dt=87 reason_string=20 stack=13
            GoStart dt=45 g=1 g_seq=13465
            GoStop dt=88 reason_string=20 stack=13
            GoStart dt=48 g=

... log truncated ...

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|fatal error:|panic:|WARNING: DATA RACE|go tool dist: |(Test .* ran over .* limit)|([^ ]*make.bash: .*error))`)

2023-11-15 20:08 freebsd-arm64-dmgk go@2380862c TestTraceCPUProfile (log)

--- FAIL: TestTraceCPUProfile (4.27s)
    --- FAIL: TestTraceCPUProfile/Default (4.23s)
        trace_test.go:177: execution trace did not include all CPU profile samples; 28 in profile, 14 in trace
        trace_test.go:187: execution trace did not include all CPU profile samples for stack "main.cpuHog0:126|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"; 2 in profile, 1 in trace
        trace_test.go:187: execution trace did not include all CPU profile samples for stack "main.cpuHog0:130|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"; 20 in profile, 9 in trace
        trace_test.go:195: CPU profile included 1 samples at stack "aeshashbody:711|runtime.(*traceMap).put:62|runtime.(*traceStackTable).put:95|runtime.traceStack:79|runtime.traceLocker.stack:166|runtime.traceLocker.GoStop:425|runtime.traceLocker.GoSched:415|runtime.goschedImpl:4051|runtime.gosched_m:4070|runtime.mcall:192" not present in execution trace
        trace_test.go:195: CPU profile included 1 samples at stack "runtime.traceStack:70|runtime.traceLocker.stack:166|runtime.traceLocker.GoStop:425|runtime.traceLocker.GoSched:415|runtime.goschedImpl:4051|runtime.gosched_m:4070|runtime.mcall:192" not present in execution trace
        trace_test.go:202: execution trace CPU samples:
        trace_test.go:204: 9: "main.cpuHog0:130|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"
        trace_test.go:204: 1: "main.cpuHog0:126|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"
...
            GoStart dt=50 g=1 g_seq=13593
            GoStop dt=89 reason_string=20 stack=12
            GoStart dt=50 g=1 g_seq=13594
            GoStop dt=89 reason_string=20 stack=12
            GoStart dt=49 g=1 g_seq=13595
            GoStop dt=92 reason_string=20 stack=12
            GoStart dt=50 g=1 g_seq=13596
            GoStop

... log truncated ...

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|fatal error:|panic:|WARNING: DATA RACE|go tool dist: |(Test .* ran over .* limit)|([^ ]*make.bash: .*(error|fault)))`)

2023-11-17 23:00 linux-arm64-longtest go@3cdc2a13 TestTraceStacks (log)

--- FAIL: TestTraceStacks (2.53s)
    --- FAIL: TestTraceStacks/Stress (0.68s)
        reader_test.go:65: unexpected error while reading the trace: inconsistent status for proc 1: old Syscall vs. new Running
        trace_test.go:569: found bad trace; dumping to test log...
        trace_test.go:570: Trace Go1.22
            EventBatch gen=1 m=18446744073709551615 time=8166829984 size=5
            Frequency freq=15625000
            EventBatch gen=1 m=117706 time=8166829111 size=68
            ProcStart dt=61 p=2 p_seq=1
            GoStart dt=86 g=21 g_seq=1
...
            EventBatch gen=273 m=117705 time=8167201385 size=28
            ProcStatus dt=1 p=1 pstatus=2
            ProcStart dt=1 p=1 p_seq=1
            ProcStop dt=33
            ProcStart dt=949 p=4 p_seq=8
            GoUnblock dt=9 g=21 g_seq=4 stack=0
            GoStart dt=48 g=21 g_seq=5


... log truncated ...

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|fatal error:|panic:|WARNING: DATA RACE|go tool dist: |(Test .* ran over .* limit)|([^ ]*make.bash: .*(error|fault)))`)

2023-11-21 22:23 freebsd-arm64-dmgk go@ff05cdbd TestTraceCPUProfile (log)

--- FAIL: TestTraceCPUProfile (2.16s)
    --- FAIL: TestTraceCPUProfile/Default (2.13s)
        trace_test.go:177: execution trace did not include all CPU profile samples; 25 in profile, 4 in trace
        trace_test.go:187: execution trace did not include all CPU profile samples for stack "main.cpuHog0:130|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"; 19 in profile, 1 in trace
        trace_test.go:187: execution trace did not include all CPU profile samples for stack "main.cpuHog0:125|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"; 4 in profile, 1 in trace
        trace_test.go:202: execution trace CPU samples:
        trace_test.go:204: 1: "main.cpuHog0:130|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"
        trace_test.go:204: 1: "main.cpuHog0:126|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"
        trace_test.go:204: 1: "main.cpuHog0:125|main.cpuHog1:120|main.cpuHogger:107|main.main.func1.1:44|runtime/pprof.Do:51|main.main.func1:43|main.main:50|runtime.main:271"
        trace_test.go:204: 1: "runtime.releasem:586|runtime.traceRelease:238|runtime.execute:3133|runtime.schedule:3922|runtime.goschedImpl:4069|runtime.gosched_m:4074|runtime.mcall:192"
...
            GoStop dt=73 reason_string=20 stack=14
            GoStart dt=7 g=1 g_seq=13726
            GoStop dt=73 reason_string=20 stack=14
            GoStart dt=7 g=1 g_seq=13727
            GoStop dt=74 reason_string=20 stack=14
            GoStart dt=8 g=1 g_seq=13728
            GoStop dt=74 reason_string=20 stack=14
            GoStart dt=6 g=1 g_seq=13729

... log truncated ...

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "go" && !(log ~ `^(All tests passed|FAIL|build failed|fatal error:|panic:|WARNING: DATA RACE|go tool dist: |(Test .* ran over .* limit)|([^ ]*make.bash: .*(error|fault)))`)

2023-11-21 19:04 plan9-arm go@4e6bbbe6 (log)

---
Installed Go for plan9/arm in /boot/workdir/go
Installed commands in /boot/workdir/go/bin
*** You need to bind /boot/workdir/go/bin before /bin.

— watchflakes