-
-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Goroutine stack inspector to admin/monitor #19207
Conversation
Now it would be probably helpful to add a vuejs component around this that would allow easy filtering but I think this might represent a good starting point. |
89a3f65
to
66b86cb
Compare
Using the pprof labels we can now add a stacktrace which can show these labels and name go-routines by the labels if there is a description set. Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
a110aa5
to
1804e52
Compare
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
I've made substantial changes to this to hopefully make the stack trace more friendly |
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After reading the code twice, I'm not sure if I'm actually understanding what's going on. I have a general idea, but the specifics are 🤷🏽.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally LGTM. But it much larger than I thought ..... +1,424 −571 about 900 lines.
As an internal framework level mechanism, personally I think sometimes enough (simple) is better than perfect (complex) 😊
Most of the code is gorutine and context wrapping labelling outside of modules/process. The merging and handling of the profile.Sample is unfortunate but at least means that the object tree makes some sense. We have to do that if we want to merge in the process state in to the goroutine stacks, and we need to do that merging with the process manager locked otherwise the results are racey and you'll end up not knowing if you're seeing a goroutine leak or not. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@zeripath ☝️ one of the latest changes did confict with |
# Conflicts: # modules/process/manager.go
77bc0c5
to
4daf8ca
Compare
I did the merge again and resolved the conflicts. |
🚀 |
* giteaoffical/main: Fix broken of team create (go-gitea#19288) Remove `git.Command.Run` and `git.Command.RunInDir*` (go-gitea#19280) Performance improvement for add team user when org has more than 1000 repositories (go-gitea#19227) [skip ci] Updated translations via Crowdin Update JS dependencies (go-gitea#19281) Fix container download counter (go-gitea#19287) go.mod: update kevinburke/ssh_config to v1.2.0 (go-gitea#19286) Fix global packages enabled avaiable (go-gitea#19276) Add Goroutine stack inspector to admin/monitor (go-gitea#19207) Move checks for pulls before merge into own function (go-gitea#19271) Restore user autoregistration with email addresses (go-gitea#19261) Improve sync performance for pull-mirrors (go-gitea#19125) Refactor `git.Command.Run*`, introduce `RunWithContextString` and `RunWithContextBytes` (go-gitea#19266) Move reaction to models/issues/ (go-gitea#19264)
Continues on from #19202.
Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them.
This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile.
The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines.
If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process.
The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound.
In this way we should be able to more easily examine goroutines that have been stuck.
A manager command
gitea manager processes
is also provided that can export the processes (with or without stacktraces) to the command line.Screenshots