Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

usdt: Use new usdt_addsem_probe* helpers #1374

Merged
merged 1 commit into from
Jun 18, 2020
Merged

Conversation

danobi
Copy link
Member

@danobi danobi commented Jun 5, 2020

While debugging a high memory consumption issue, I noticed that a
USDT::Context object can take ~10M per instance [0]. Along with the new
--usdt-file-activation, bpftrace can potentially hold onto many dozens
of USDT:Context instances, causing memory issues.

This patch makes use of the new bcc usdt semaphore helpers (
iovisor/bcc#2953 ) to avoid holding onto
USDT::Context instances for the lifetime of the tracing session.

The new algorithm is:

  1. create a USDT::Context
  2. increment the semaphore count for the probe we care about
  3. destroy the USDT::Context
  4. repeat 1-3 for all probes we want to attach to
  5. do our tracing
  6. create a USDT::Context for the probe we care about
  7. decrement the semaphore count
  8. destroy the USDT::Context
  9. repeat 6-8 for all the probes we're attached to

This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.

[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.

Checklist
  • Language changes are updated in docs/reference_guide.md
  • User-visible and non-trivial changes updated in CHANGELOG.md

@danobi danobi marked this pull request as draft June 5, 2020 23:30
@danobi
Copy link
Member Author

danobi commented Jun 5, 2020

Relies on #1370 and iovisor/bcc#2954 .

@danobi danobi marked this pull request as ready for review June 15, 2020 19:28
While debugging a high memory consumption issue, I noticed that a
USDT::Context object can take ~10M per instance [0]. Along with the new
--usdt-file-activation, bpftrace can potentially hold onto many dozens
of USDT:Context instances, causing memory issues.

This patch makes use of the new bcc usdt semaphore helpers (
iovisor/bcc#2953 ) to avoid holding onto
USDT::Context instances for the lifetime of the tracing session.

The new algorithm is:

1. create a USDT::Context
2. increment the semaphore count for the probe we care about
3. destroy the USDT::Context
4. repeat 1-3 for all probes we want to attach to
5. do our tracing
6. create a USDT::Context for the probe we care about
7. decrement the semaphore count
8. destroy the USDT::Context
9. repeat 6-8 for all the probes we're attached to

This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.

[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.
@danobi
Copy link
Member Author

danobi commented Jun 15, 2020

Ready for review

Copy link
Contributor

@fbs fbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Should we add a tracking ticket for CI testing this? This is (another) feature that gets no testing, might be good to start tracking them

@danobi
Copy link
Member Author

danobi commented Jun 18, 2020

Should we add a tracking ticket for CI testing this?

I think this'll get testing when the new bcc version is released and the CI starts building against the new bcc api. I'm not too sure it'd be worth it to set up a new CI step just to build/test custom versions of bcc.

@danobi danobi merged commit 60d77c2 into bpftrace:master Jun 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants