Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: lock ranking failure between sysmon and gscan #54553

Closed
heschi opened this issue Aug 19, 2022 · 6 comments
Closed

runtime: lock ranking failure between sysmon and gscan #54553

heschi opened this issue Aug 19, 2022 · 6 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@heschi heschi added the NeedsFix The path to resolution is known, but the work has not been done. label Aug 19, 2022
@heschi heschi added this to the Go1.20 milestone Aug 19, 2022
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Aug 19, 2022
@prattmic
Copy link
Member

prattmic commented Aug 19, 2022

207493  ======
0 : sysmon 1 0xd66998
1 : traceBuf 23 0xd7df98
2 : traceStackTab 42 0xd6def8
3 : gscan 34 0x0
fatal error: lock ordering problem

@aclements
Copy link
Member

aclements commented Aug 22, 2022

The failure is almost always in testing cmd/trace (occasionally runtime/trace). It's relatively easy to reproduce:

$ git checkout c6be710
$ GOEXPERIMENT=staticlockranking ./make.bash && stress2 -max-runs 1000 -- go test -count 1 -short cmd/trace
...
926 passes, 74 fails, avg 2s

Presumably I introduced this when I lowered the trace locks to be near the bottom of the lock ranking, and below stack growth in particular. I made sure that trace.lock was only acquired on the system stack, but I think I missed traceStackTable.lock.

@aclements
Copy link
Member

aclements commented Aug 22, 2022

Confirmed that I was able to reproduce at cc8bac8, but not at cc8bac8~. I'll send a fix shortly.

@gopherbot
Copy link

gopherbot commented Aug 22, 2022

Change https://go.dev/cl/425097 mentions this issue: runtime: move traceStackTable.lock to the system stack

@bcmills
Copy link
Member

bcmills commented Aug 23, 2022

The failure is almost always in testing cmd/trace (occasionally runtime/trace).

Hmm, does that also explain the timeouts reported in #54594?

@aclements
Copy link
Member

aclements commented Aug 23, 2022

Hmm, does that also explain the timeouts reported in #54594?

The first instance of that issue happened before my change, so it doesn't directly relate to this issue. But I'll comment on that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsFix The path to resolution is known, but the work has not been done.
Projects
Status: Done
Development

No branches or pull requests

5 participants