New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pmda to support cpu flame graphs #74

Open
spiermar opened this Issue Jul 30, 2015 · 18 comments

Comments

Projects
None yet
7 participants
@spiermar
Contributor

spiermar commented Jul 30, 2015

New PCP PMDA that would be able to consume perf-events directly and respond to Vector, preferably via JSON. Will be required to support Flame Graphs.

@fche

This comment has been minimized.

Show comment
Hide comment
@fche

fche Jul 31, 2015

Contributor

PCP already has two pmdas related to performance counters: papi and perfevent. The former is more automated out-of-the-box, except that it currently requires privileged access. The latter needs hand-configuration.

Contributor

fche commented Jul 31, 2015

PCP already has two pmdas related to performance counters: papi and perfevent. The former is more automated out-of-the-box, except that it currently requires privileged access. The latter needs hand-configuration.

@spiermar spiermar changed the title from perf-even pmda to perf-event pmda Jul 31, 2015

@spiermar spiermar removed the enhancement label Jul 31, 2015

@spiermar spiermar modified the milestone: r1.0.4 Jul 31, 2015

@spiermar spiermar modified the milestones: r1.0.4, r.1.0.5 Nov 18, 2015

@spiermar spiermar changed the title from perf-event pmda to perf_event integration Apr 22, 2016

@vavrusa

This comment has been minimized.

Show comment
Hide comment
@vavrusa

vavrusa Sep 3, 2016

Hi! Is this being worked? Today I tried to set up on-cpu flame graphs with Vector with not much success. Digging through the mailing lists it looks like events type is not even supported in pmwebd, is that something that needs to be done prior to support in Vector?

vavrusa commented Sep 3, 2016

Hi! Is this being worked? Today I tried to set up on-cpu flame graphs with Vector with not much success. Digging through the mailing lists it looks like events type is not even supported in pmwebd, is that something that needs to be done prior to support in Vector?

@fche

This comment has been minimized.

Show comment
Hide comment
@fche

fche Sep 3, 2016

Contributor

The perfevents pmda (like the papi pmda) supplies normal scalar numeric sorts of metrics, which pmwebd happily relays. PCP events are a different sort of thing.

Contributor

fche commented Sep 3, 2016

The perfevents pmda (like the papi pmda) supplies normal scalar numeric sorts of metrics, which pmwebd happily relays. PCP events are a different sort of thing.

@vavrusa

This comment has been minimized.

Show comment
Hide comment
@vavrusa

vavrusa Sep 3, 2016

Ah, so perfevents does perf_type_hardware only and doesn't support complex sample types like callchain?

vavrusa commented Sep 3, 2016

Ah, so perfevents does perf_type_hardware only and doesn't support complex sample types like callchain?

@fche

This comment has been minimized.

Show comment
Hide comment
@fche

fche Sep 3, 2016

Contributor

Correct.

Contributor

fche commented Sep 3, 2016

Correct.

@spiermar

This comment has been minimized.

Show comment
Hide comment
@spiermar

spiermar Sep 4, 2016

Contributor

At this point, there's a very rudimentary PMDA that allows users to generate flame graphs from Vector. In the future, I hope to have a more flexible PMDA that allows users not only to generate flame graphs, but also visualize more metrics from perf.

Contributor

spiermar commented Sep 4, 2016

At this point, there's a very rudimentary PMDA that allows users to generate flame graphs from Vector. In the future, I hope to have a more flexible PMDA that allows users not only to generate flame graphs, but also visualize more metrics from perf.

@vavrusa

This comment has been minimized.

Show comment
Hide comment
@vavrusa

vavrusa Sep 4, 2016

I see, that's what I am after. Since there's not much infrastructure built yet, I'll rather pursue getting BPF probes output (maps and collapsed+aggregated callstacks) to Vector instead. If I understand it right there's already a d3-based widget to display flamegraphs and heatmaps and having pseudo real-time data from probes is much more compelling.

vavrusa commented Sep 4, 2016

I see, that's what I am after. Since there's not much infrastructure built yet, I'll rather pursue getting BPF probes output (maps and collapsed+aggregated callstacks) to Vector instead. If I understand it right there's already a d3-based widget to display flamegraphs and heatmaps and having pseudo real-time data from probes is much more compelling.

@spiermar

This comment has been minimized.

Show comment
Hide comment
@spiermar

spiermar Sep 4, 2016

Contributor

Yes, I already have the d3 plugin ready and should be able to come up with a Vector widget pretty quickly, assuming I can get the right data format out of PCP. BPF probes would probably be the ideal solution at this point. If you're looking to pursue this route, I'm in. @brendangregg would probably be interested in this too.

Contributor

spiermar commented Sep 4, 2016

Yes, I already have the d3 plugin ready and should be able to come up with a Vector widget pretty quickly, assuming I can get the right data format out of PCP. BPF probes would probably be the ideal solution at this point. If you're looking to pursue this route, I'm in. @brendangregg would probably be interested in this too.

@brendangregg

This comment has been minimized.

Show comment
Hide comment
@brendangregg

brendangregg Sep 8, 2016

Contributor

@vavrusa Yes, I was looking into perf_events, but couldn't help but think it might be wasted effort when we'd prefer to do this with BPF. I'll (probably) start by updating the BPF/bcc profiler (@4ast's profiling support has now made it to net-next), so we have an example to borrow from.

Edit: we'll still want perf_events access for everything else, like PMCs, in Vector...

Contributor

brendangregg commented Sep 8, 2016

@vavrusa Yes, I was looking into perf_events, but couldn't help but think it might be wasted effort when we'd prefer to do this with BPF. I'll (probably) start by updating the BPF/bcc profiler (@4ast's profiling support has now made it to net-next), so we have an example to borrow from.

Edit: we'll still want perf_events access for everything else, like PMCs, in Vector...

@vavrusa

This comment has been minimized.

Show comment
Hide comment
@vavrusa

vavrusa Sep 8, 2016

@brendangregg nice, ultimately what I want is the output from BPF probes (and profiler). It wasn't clear for me from the beginning if there's any support for that in pmc/pmwebd. I think, in the end, it might work better for me to spec what kind of data could Vector consume (tabular data from BPF map, heatmap, callstacks), and generate this data from edge applications directly/through proxy (e.g. NGINX/OpenResty reusing lj-bpf to monitor itself/its sockets) instead of having special daemon(s). I'm not sure how do you secure access to pmcd/pmwebd's on edge nodes at Netflix, if you could share some insight on that, that would fantastic. I think a gateway + OAuth + gw-browser encryption is bare minimum. Anyhow, I'll be on PTO for a few weeks, so that'd give me some time to think it through.

vavrusa commented Sep 8, 2016

@brendangregg nice, ultimately what I want is the output from BPF probes (and profiler). It wasn't clear for me from the beginning if there's any support for that in pmc/pmwebd. I think, in the end, it might work better for me to spec what kind of data could Vector consume (tabular data from BPF map, heatmap, callstacks), and generate this data from edge applications directly/through proxy (e.g. NGINX/OpenResty reusing lj-bpf to monitor itself/its sockets) instead of having special daemon(s). I'm not sure how do you secure access to pmcd/pmwebd's on edge nodes at Netflix, if you could share some insight on that, that would fantastic. I think a gateway + OAuth + gw-browser encryption is bare minimum. Anyhow, I'll be on PTO for a few weeks, so that'd give me some time to think it through.

@spiermar

This comment has been minimized.

Show comment
Hide comment
@spiermar

spiermar Sep 8, 2016

Contributor

The format consumed by the flame graph d3 plugin is roughly this:
https://github.com/spiermar/d3-flame-graph/blob/master/example/stacks.json

I also have a tool to convert the folded stacks into JSON, so Vector could consume that too.

For the other visualizations (heatmaps, etc) it depends.

Contributor

spiermar commented Sep 8, 2016

The format consumed by the flame graph d3 plugin is roughly this:
https://github.com/spiermar/d3-flame-graph/blob/master/example/stacks.json

I also have a tool to convert the folded stacks into JSON, so Vector could consume that too.

For the other visualizations (heatmaps, etc) it depends.

@spiermar spiermar changed the title from perf_event integration to pmda to support cpu flame graphs Sep 8, 2016

@tuxology

This comment has been minimized.

Show comment
Hide comment
@tuxology

tuxology Sep 17, 2016

I tried out Vector on my servers today. Pretty cool! @vavrusa I am also looking for a way to have BCC integration so that we could write custom widgets based on BPF maps data. Can you point me to the spec you are going to maintain? Maybe I can add some stuff to it too as I discover Vector more..

tuxology commented Sep 17, 2016

I tried out Vector on my servers today. Pretty cool! @vavrusa I am also looking for a way to have BCC integration so that we could write custom widgets based on BPF maps data. Can you point me to the spec you are going to maintain? Maybe I can add some stuff to it too as I discover Vector more..

@mogeb

This comment has been minimized.

Show comment
Hide comment
@mogeb

mogeb Oct 27, 2016

I'm trying to write a PMDA that interfaces with BPF. Has anyone else started working on this? I'm writing a PMDA as a python "library", which would allow me to use BCC directly (at least that's my idea). At the moment I'm stuck since all PMDAs run under the pcp user, which isn't allowed to hook onto kprobes. Although I could probably solve this with a workaround, I'm wondering if this is even the right approach. Any thoughts?

mogeb commented Oct 27, 2016

I'm trying to write a PMDA that interfaces with BPF. Has anyone else started working on this? I'm writing a PMDA as a python "library", which would allow me to use BCC directly (at least that's my idea). At the moment I'm stuck since all PMDAs run under the pcp user, which isn't allowed to hook onto kprobes. Although I could probably solve this with a workaround, I'm wondering if this is even the right approach. Any thoughts?

@spiermar

This comment has been minimized.

Show comment
Hide comment
@spiermar
Contributor

spiermar commented Oct 27, 2016

@vavrusa

This comment has been minimized.

Show comment
Hide comment
@vavrusa

vavrusa Oct 27, 2016

@tuxology AFAIK there's only a spec for callstacks as mentioned by @spiermar above. I didn't have much time to work on that further, mapping linear array to JSON array is pretty obvious, so is hashmap (with the exception of values). The call stack map would definitely deserve a schema.

My original idea was to have a library like Prometheus to serve contents of BPF maps on demand (as JSON), this way we can serve information both from non-privileged processes (e.g. observing what goes through sockets), and privileged processes (e.g. callstacks). I've kicked the PCP tires, but it's hard for me to justify managing another infrastructure, when we've already figured out how to get metrics through HTTP endpoints.

vavrusa commented Oct 27, 2016

@tuxology AFAIK there's only a spec for callstacks as mentioned by @spiermar above. I didn't have much time to work on that further, mapping linear array to JSON array is pretty obvious, so is hashmap (with the exception of values). The call stack map would definitely deserve a schema.

My original idea was to have a library like Prometheus to serve contents of BPF maps on demand (as JSON), this way we can serve information both from non-privileged processes (e.g. observing what goes through sockets), and privileged processes (e.g. callstacks). I've kicked the PCP tires, but it's hard for me to justify managing another infrastructure, when we've already figured out how to get metrics through HTTP endpoints.

@natoscott

This comment has been minimized.

Show comment
Hide comment
@natoscott

natoscott Oct 27, 2016

Contributor

@mogeb PMDAs run as root by default, although many choose to switch to pcp user (voluntarily) via the set_user call (in python). If you drop that call from your script, you should find it runs as root.

Contributor

natoscott commented Oct 27, 2016

@mogeb PMDAs run as root by default, although many choose to switch to pcp user (voluntarily) via the set_user call (in python). If you drop that call from your script, you should find it runs as root.

@mogeb

This comment has been minimized.

Show comment
Hide comment
@mogeb

mogeb Oct 28, 2016

@natoscott thanks! I hadn't even notice that.

mogeb commented Oct 28, 2016

@natoscott thanks! I hadn't even notice that.

@spiermar spiermar modified the milestones: v1.3, v1.2 Jun 22, 2017

@spiermar spiermar added this to To Do in V1.2 Jun 22, 2017

@spiermar spiermar moved this from To Do to Doing in V1.2 Jun 22, 2017

@spiermar spiermar removed this from Doing in V1.2 Dec 18, 2017

@spiermar spiermar modified the milestones: v1.2, v1.3, v1.4 Dec 18, 2017

@spiermar spiermar removed this from the v1.4 milestone Aug 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment