Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bpftrace to pbench-agent-tools #1108

Merged

Conversation

sourabhtk37
Copy link
Collaborator

@sourabhtk37 sourabhtk37 commented Feb 28, 2019

Todo

  • find where bpftrace is installed: /usr/bin/ or /usr/local/bin
  • Check for required packages and kernel headers for bpftrace to run

This is for tracking purpose. I used the existing jmap tool to base this. Need your inputs/reviews on implementing the post-processing and other features needed for pbench.

Here's the sample stdout from a run, which uses a basic bpftrace program, syscount.bt:

# cat /var/lib/pbench-agent/iozone__2019.02.26T16.23.27/1/reference-result/tools-default/localhost/bpftrace/bpftrace-stdout.txt
Attaching 3 probes...
Counting syscalls... Hit Ctrl-C to end.
 
Top 10 syscalls IDs:
@syscall[202]: 82215
@syscall[186]: 157832
@syscall[232]: 176516
@syscall[38]: 183350
@syscall[7]: 228544
@syscall[20]: 253303
@syscall[8]: 284114
@syscall[1]: 311635
@syscall[0]: 465967
@syscall[47]: 682795
 
Top 10 processes:
@process[grep]: 72991
@process[cut]: 75418
@process[Chrome_IOThread]: 89515
@process[Chrome_ChildIOT]: 100151
@process[code]: 167427
@process[sh]: 358588
@process[chromium-browse]: 365207
@process[xprop]: 622420
@process[Xorg]: 669920
@process[iozone]: 921651

Signed-off-by: T K Sourab sourabhtk37@gmail.com

@sourabhtk37 sourabhtk37 added help wanted DO NOT MERGE!!! tools Of and related to the operation and behavior of various tools (iostat, sar, etc.) labels Feb 28, 2019
@atheurer
Copy link
Contributor

  1. Can you provide examples of how the bpftrace tool was registered and how you started/stopped the tool (using a pbench benchmark or running pbench-start-tools or pbench-stop-tools)?

  2. Does syscount.bt require compiling before using? If so, please see how this is handled for systemtap for pbench (the stap code is compiled during --install, which what pbench-register-tool calls), as bpf might need to do this in a similar way.

  3. The --install will need to check for required packages (not cmds) and install if not found (using check_install_rpm)

Copy link
Member

@portante portante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great start. This is really cool.

Can you be sure to follow the model of the systemtap tool as well?

agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
@sourabhtk37
Copy link
Collaborator Author

sourabhtk37 commented Mar 1, 2019

@atheurer 2. The syscount.bt doesn't need compiling into the kernel, bpftrace using libbpf dynamically loads it into the kernel.
3. Yes, I'm working on that. Need to make sure kernel configs required to run is enabled and other packages required for running are also installed. Thanks.

@sourabhtk37 sourabhtk37 changed the title [WIP] Add bpftrace tool to pbench-agent-tools [WIP] Add bpftrace to pbench-agent-tools Mar 1, 2019
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
@atheurer
Copy link
Contributor

atheurer commented Mar 5, 2019

@atheurer 2. The syscount.bt doesn't need compiling into the kernel, bpftrace using libbpf dynamically loads it into the kernel.

Yes, I understand it can do this, but my concern is that libbpf doing this at tool-start time might require a lot of [cpu] resources, or might require a little bit of time to do this. As a comparison, for systemtap, this is why we "pre" compile the script when we register the tool, instead of just doing it automatically at start-tools. Maybe this is not possible with BPF, but I guess we can see how it behaves with the way you have it now, and then see later if it becomes an issue.

@portante portante changed the title [WIP] Add bpftrace to pbench-agent-tools Add bpftrace to pbench-agent-tools Mar 7, 2019
Copy link
Member

@portante portante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost, just a few things to clean up.

agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
@portante portante modified the milestones: v0.57, v0.58 Mar 8, 2019
Copy link
Member

@portante portante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there, looking good.

agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Outdated Show resolved Hide resolved
agent/tool-scripts/bpftrace Show resolved Hide resolved
@portante
Copy link
Member

Can you also rebase and squash this commits?

Signed-off-by: T K Sourab <sourabhtk37@gmail.com>

: Base bpftrace tool like systemtap tool
: add command to tool_cmd_file

Signed-off-by: T K Sourab <sourabhtk37@gmail.com>

Clean up bpftrace tool-script

Signed-off-by: T K Sourab <sourabhtk37@gmail.com>

Added checks for finding location of bpftrace binary

Signed-off-by: T K Sourab <sourabhtk37@gmail.com>

Cleanup:
- Add check_install_rpm function
- Required argument checking (--script)
- Use default rpm install location

Signed-off-by: T K Sourab <sourabhtk37@gmail.com>

Add comments to example usage

Signed-off-by: T K Sourab <sourabhtk37@gmail.com>

Cleanup:
- check non-empty $script
- check for directory creation success
- typos

Signed-off-by: T K Sourab <sourabhtk37@gmail.com>

Some checking to add to tool script
Copy link
Member

@portante portante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, tried it out with a mock of /usr/bin/bpftrace and it seemed to work fine.

@portante portante merged commit 5699d0a into distributed-system-analysis:master Mar 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Agent enhancement tools Of and related to the operation and behavior of various tools (iostat, sar, etc.)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants