Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Probe goroutine create and exit for request time #49

Merged
merged 12 commits into from
Apr 13, 2023

Conversation

grcevski
Copy link
Contributor

@grcevski grcevski commented Apr 7, 2023

With this PR we add two new uprobes, one at runtime.newproc1 and another at runtime.goexit1. When we capture the time at runtime.newproc1, we can tell the total request time, not just the service time.

I added nested spans in the traces to capture this information. When testing with our 10,000 QPS load driver, I was able to see pathological cases where the request time was dominated by the time the request waits in queue.

This PR is based on #46. It needs to be merged first.

@grcevski grcevski added the enhancement New feature or request label Apr 7, 2023
@codecov-commenter
Copy link

codecov-commenter commented Apr 7, 2023

Codecov Report

Merging #49 (df56dbc) into main (d41cb3e) will increase coverage by 0.10%.
The diff coverage is 44.57%.

@@            Coverage Diff             @@
##             main      #49      +/-   ##
==========================================
+ Coverage   51.13%   51.23%   +0.10%     
==========================================
  Files          14       14              
  Lines         927      968      +41     
==========================================
+ Hits          474      496      +22     
- Misses        418      435      +17     
- Partials       35       37       +2     
Flag Coverage Δ
unittests 51.23% <44.57%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pkg/ebpf/nethttp/nethttp.go 0.00% <0.00%> (ø)
pkg/export/debug/debug.go 15.38% <0.00%> (-0.62%) ⬇️
pkg/pipe/config.go 53.33% <ø> (ø)
pkg/export/otel/metrics.go 80.68% <100.00%> (ø)
pkg/export/otel/traces.go 80.76% <100.00%> (+4.90%) ⬆️
pkg/pipe/instrumenter.go 83.33% <100.00%> (+0.72%) ⬆️
pkg/transform/spanner.go 81.33% <100.00%> (+0.51%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@grcevski grcevski requested a review from mariomac April 12, 2023 17:47
@grcevski grcevski mentioned this pull request Apr 12, 2023
1 task
@grcevski grcevski merged commit 6ef1edc into grafana:main Apr 13, 2023
2 checks passed
@grcevski grcevski deleted the goroutine_probe branch April 13, 2023 14:39
@grcevski
Copy link
Contributor Author

Thanks Mario!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants