runtime: lock ranking failure between sysmon and gscan

[heschi]

2022-08-18T18:18:45-1aa5730/linux-amd64-staticlockranking
2022-08-18T17:26:10-b23d469/linux-amd64-staticlockranking
2022-08-18T17:09:37-c82bbc0/linux-amd64-staticlockranking
2022-08-18T16:35:13-b2c0417/linux-amd64-staticlockranking
2022-08-18T16:15:06-c7c78f1/linux-amd64-staticlockranking
2022-08-18T16:12:31-b8b9e83/linux-amd64-staticlockranking
2022-08-18T14:53:01-741ab7e/linux-amd64-staticlockranking
2022-08-18T05:32:36-a2c2f06/linux-amd64-staticlockranking
2022-08-18T02:51:24-04d3d6b/linux-amd64-staticlockranking
2022-08-18T00:30:19-b30ba3d/linux-amd64-staticlockranking
2022-08-17T23:18:24-9709d92/linux-amd64-staticlockranking
2022-08-17T18:47:33-57d0551/linux-amd64-staticlockranking
2022-08-17T17:40:05-dea67a9/linux-amd64-staticlockranking
2022-08-17T16:48:18-b11b4b4/linux-amd64-staticlockranking
2022-08-17T16:26:22-5b0ce94/linux-amd64-staticlockranking
2022-08-17T04:02:17-c411886/linux-amd64-staticlockranking
2022-08-17T03:24:24-c6be710/linux-amd64-staticlockranking

cc @golang/runtime

[prattmic]

207493  ======
0 : sysmon 1 0xd66998
1 : traceBuf 23 0xd7df98
2 : traceStackTab 42 0xd6def8
3 : gscan 34 0x0
fatal error: lock ordering problem

[aclements]

The failure is almost always in testing cmd/trace (occasionally runtime/trace). It's relatively easy to reproduce:

$ git checkout c6be710
$ GOEXPERIMENT=staticlockranking ./make.bash && stress2 -max-runs 1000 -- go test -count 1 -short cmd/trace
...
926 passes, 74 fails, avg 2s

Presumably I introduced this when I lowered the trace locks to be near the bottom of the lock ranking, and below stack growth in particular. I made sure that trace.lock was only acquired on the system stack, but I think I missed traceStackTable.lock.

[aclements]

Confirmed that I was able to reproduce at cc8bac8, but not at cc8bac8~. I'll send a fix shortly.

[gopherbot]

Change https://go.dev/cl/425097 mentions this issue: runtime: move traceStackTable.lock to the system stack

[bcmills]

The failure is almost always in testing cmd/trace (occasionally runtime/trace).

Hmm, does that also explain the timeouts reported in #54594?

[aclements]

Hmm, does that also explain the timeouts reported in #54594?

The first instance of that issue happened before my change, so it doesn't directly relate to this issue. But I'll comment on that issue.