New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Web UI should support flame graphs #166

Closed
rakyll opened this Issue Jul 19, 2017 · 23 comments

Comments

Projects
None yet
@rakyll
Member

rakyll commented Jul 19, 2017

image

Flame graphs allow you to move in a specific ancestry path from a very compact UI and lets the users to zoom in/out specific paths easily. Flame graphs became one of the defato representation of profiling data in the industry in the recent years, e.g. go-torch is a highly successful tool and often user's entry point to visualize pprof data in Go.

Consider supporting flame graphs in addition to the dot visualization on the new web UI, so we can provide this representation out-of-the-box with minimal setup and external dependencies.

(If you need this feature as well, you can 👍 so the team knows about your interest.)

@rakyll rakyll changed the title from Web UI should support the flame graphs to Web UI should support flame graphs Jul 19, 2017

@nathanleclaire

This comment has been minimized.

Show comment
Hide comment
@nathanleclaire

nathanleclaire Jul 20, 2017

This would be amazing! Thanks for putting together the proposal. One thing I especially love from go-torch / Brendan's script is that you can search and highlight based on keyword, e.g., reflect to show very quickly each candle that touches reflect package. Would love to see that included in a Golang official version as well.

goflamegraph

nathanleclaire commented Jul 20, 2017

This would be amazing! Thanks for putting together the proposal. One thing I especially love from go-torch / Brendan's script is that you can search and highlight based on keyword, e.g., reflect to show very quickly each candle that touches reflect package. Would love to see that included in a Golang official version as well.

goflamegraph

@nathanleclaire

This comment has been minimized.

Show comment
Hide comment
@nathanleclaire

nathanleclaire Jul 20, 2017

I bet you could do fun things with color too, like give each package its own unique tint, so that one could quickly scan the graph for mis-behaving packages.

nathanleclaire commented Jul 20, 2017

I bet you could do fun things with color too, like give each package its own unique tint, so that one could quickly scan the graph for mis-behaving packages.

@nathanleclaire

This comment has been minimized.

Show comment
Hide comment
@nathanleclaire

nathanleclaire Jul 20, 2017

If I recall as well, I also had a pretty hard time getting a useful flame graph for anything other than CPU usage (e.g., memory) from pprof. That's something to consider, I'd love to see some Golang memory usage flame graphs.

nathanleclaire commented Jul 20, 2017

If I recall as well, I also had a pretty hard time getting a useful flame graph for anything other than CPU usage (e.g., memory) from pprof. That's something to consider, I'd love to see some Golang memory usage flame graphs.

@chenziliang

This comment has been minimized.

Show comment
Hide comment
@chenziliang

chenziliang Jul 20, 2017

Would really love to have this stuff built in

chenziliang commented Jul 20, 2017

Would really love to have this stuff built in

@huydx

This comment has been minimized.

Show comment
Hide comment
@huydx

huydx Jul 20, 2017

flamegraph has been my default development stack for a while now, love it is a default part of pprof

huydx commented Jul 20, 2017

flamegraph has been my default development stack for a while now, love it is a default part of pprof

@pierods

This comment has been minimized.

Show comment
Hide comment
@pierods

pierods Jul 20, 2017

Yes please

pierods commented Jul 20, 2017

Yes please

@mremond

This comment has been minimized.

Show comment
Hide comment
@mremond

mremond Jul 20, 2017

I think it would be very useful as well.
👍

mremond commented Jul 20, 2017

I think it would be very useful as well.
👍

@campoy

This comment has been minimized.

Show comment
Hide comment
@campoy

campoy Jul 20, 2017

As someone that has written a workshop on the topic (github.com/campoy/go-tooling-workshop) I can assure you that flame graphs are an amazing tool for beginners, information is represented in a far clearer way, and removing the setup friction would be great.

👍

campoy commented Jul 20, 2017

As someone that has written a workshop on the topic (github.com/campoy/go-tooling-workshop) I can assure you that flame graphs are an amazing tool for beginners, information is represented in a far clearer way, and removing the setup friction would be great.

👍

@ipfans

This comment has been minimized.

Show comment
Hide comment
@ipfans

ipfans Jul 20, 2017

And hope it happen in Go 1.10

ipfans commented Jul 20, 2017

And hope it happen in Go 1.10

@jhayotte

This comment has been minimized.

Show comment
Hide comment
@jhayotte

jhayotte Jul 21, 2017

Yes please! 👍🏻

jhayotte commented Jul 21, 2017

Yes please! 👍🏻

@anazmy

This comment has been minimized.

Show comment
Hide comment
@anazmy

anazmy Jul 21, 2017

Yes please

anazmy commented Jul 21, 2017

Yes please

@brendangregg

This comment has been minimized.

Show comment
Hide comment
@brendangregg

brendangregg Jul 21, 2017

FWIW... There are multiple implementations of flame graphs, and you could also just write a new one in go. :-)

My original https://github.com/brendangregg/FlameGraph is Perl, and evolved, and has become complex (if you roll back the git history you'll see how simple it once was). It would be cleaner written from scratch, now that I know that it should support differentials, annotations, etc.

Netflix has been (and will be) developing https://github.com/spiermar/d3-flame-graph , as a new d3 version. This should allow us to do more interactivity. Eg, flipping the merge order from bottom-up to top-down in d3, rather than the Perl version where you have to rerun it with --reverse --inverted.

FWIW, if I were advising someone on a go rewrite based on what I've learned, I'd encourage them to do the following :

  • Mouse-overs for more info.
  • Click to zoom (you could ask Adrien Mahieux if you can just use his JavaScript zoom code from flamegraph.pl)
  • Search. A key to this being useful is that it shows the percent time in the bottom right, excluding vertical overlap.
  • A few different palettes. Apart from random warm colors, highlighting code in different ways. Eg, my java palette.
  • An architecture so you can add differentials later (however you want to implement them).
  • A way to save self-contained interactive SVGs, like those my Perl program emit. They've been handy to store, email, and refer to later on when they still work, despite the flame graph code changing.

Now, if it's consuming profiled stack traces that are time-stamped (which isn't necessary, the x-axis just sorts alphabetically), then you could have a toggle that flips between "flame charts" (time-ordered, as seen in Google chrome dev tools) and "flame graphs" (alphabet ordered, to maximize merging). Both can help solve different kinds of issues.

brendangregg commented Jul 21, 2017

FWIW... There are multiple implementations of flame graphs, and you could also just write a new one in go. :-)

My original https://github.com/brendangregg/FlameGraph is Perl, and evolved, and has become complex (if you roll back the git history you'll see how simple it once was). It would be cleaner written from scratch, now that I know that it should support differentials, annotations, etc.

Netflix has been (and will be) developing https://github.com/spiermar/d3-flame-graph , as a new d3 version. This should allow us to do more interactivity. Eg, flipping the merge order from bottom-up to top-down in d3, rather than the Perl version where you have to rerun it with --reverse --inverted.

FWIW, if I were advising someone on a go rewrite based on what I've learned, I'd encourage them to do the following :

  • Mouse-overs for more info.
  • Click to zoom (you could ask Adrien Mahieux if you can just use his JavaScript zoom code from flamegraph.pl)
  • Search. A key to this being useful is that it shows the percent time in the bottom right, excluding vertical overlap.
  • A few different palettes. Apart from random warm colors, highlighting code in different ways. Eg, my java palette.
  • An architecture so you can add differentials later (however you want to implement them).
  • A way to save self-contained interactive SVGs, like those my Perl program emit. They've been handy to store, email, and refer to later on when they still work, despite the flame graph code changing.

Now, if it's consuming profiled stack traces that are time-stamped (which isn't necessary, the x-axis just sorts alphabetically), then you could have a toggle that flips between "flame charts" (time-ordered, as seen in Google chrome dev tools) and "flame graphs" (alphabet ordered, to maximize merging). Both can help solve different kinds of issues.

@Dieterbe

This comment has been minimized.

Show comment
Hide comment
@Dieterbe

Dieterbe Jul 25, 2017

This proposal is great. personally i've found the current state of affairs requires a bit too many hurdles (specifically getting perl and the perl scripts). would it make most sense to collaborate on d3-flame-graph and reuse it as much as possible, and just putting the connecting logic in go tool pprof ? we could then have in addition to web a command like flame and it would generate a html page that uses d3-flame-graph ?

Dieterbe commented Jul 25, 2017

This proposal is great. personally i've found the current state of affairs requires a bit too many hurdles (specifically getting perl and the perl scripts). would it make most sense to collaborate on d3-flame-graph and reuse it as much as possible, and just putting the connecting logic in go tool pprof ? we could then have in addition to web a command like flame and it would generate a html page that uses d3-flame-graph ?

@spiermar

This comment has been minimized.

Show comment
Hide comment
@spiermar

spiermar Jul 25, 2017

Contributor

If anyone is looking into implementing this, specially the d3-flame-graph version, let me know. I'm hoping to get a bit of time next week to look into this. Creating a self-contained HTML file and having the flame graph data structure inline should be fairly easy to get done. The plugin takes a simple JSON file as input and the stack-convert tool has a good example of how to convert the data. As @brendangregg mentioned, we're looking to improve and add more features to the D3 plugin in the future.

Contributor

spiermar commented Jul 25, 2017

If anyone is looking into implementing this, specially the d3-flame-graph version, let me know. I'm hoping to get a bit of time next week to look into this. Creating a self-contained HTML file and having the flame graph data structure inline should be fairly easy to get done. The plugin takes a simple JSON file as input and the stack-convert tool has a good example of how to convert the data. As @brendangregg mentioned, we're looking to improve and add more features to the D3 plugin in the future.

@ajstarks

This comment has been minimized.

Show comment
Hide comment
@ajstarks

ajstarks Jul 28, 2017

I've considered doing a flame graph implementation with SVGo. Happy to contribute.

ajstarks commented Jul 28, 2017

I've considered doing a flame graph implementation with SVGo. Happy to contribute.

@rakyll

This comment has been minimized.

Show comment
Hide comment
@rakyll

rakyll Jul 31, 2017

Member

We have an existing front-end implementation of Flame Graphs at Google and we are considering about publicizing it. It has no external dependencies and covering some of the flavors @brendangregg suggested above. I am willing to export it and then we can iterate from there. My plan is to get this work done before Go 1.10 freeze, so we can have 1.10 shipped with flame graphs.

Member

rakyll commented Jul 31, 2017

We have an existing front-end implementation of Flame Graphs at Google and we are considering about publicizing it. It has no external dependencies and covering some of the flavors @brendangregg suggested above. I am willing to export it and then we can iterate from there. My plan is to get this work done before Go 1.10 freeze, so we can have 1.10 shipped with flame graphs.

@spiermar

This comment has been minimized.

Show comment
Hide comment
@spiermar

spiermar Aug 10, 2017

Contributor

Submitted #188

It's using the D3 version of flame graphs and implements all features mentioned by @brendangregg, except except different palettes, which is under development and requires changes in the D3 plugin. More features are also underway on d3-flame-graph and would just require a version bump on pprof.
Tested with both CPU profiles and Memory profiles (as suggested by @nathanleclaire).

A static demo can be found in pprof_flame.html.

Contributor

spiermar commented Aug 10, 2017

Submitted #188

It's using the D3 version of flame graphs and implements all features mentioned by @brendangregg, except except different palettes, which is under development and requires changes in the D3 plugin. More features are also underway on d3-flame-graph and would just require a version bump on pprof.
Tested with both CPU profiles and Memory profiles (as suggested by @nathanleclaire).

A static demo can be found in pprof_flame.html.

aalexand added a commit that referenced this issue Nov 21, 2017

@aalexand

This comment has been minimized.

Show comment
Hide comment
@aalexand

aalexand Nov 21, 2017

Collaborator

This is available in master now, closing.

Collaborator

aalexand commented Nov 21, 2017

This is available in master now, closing.

@aalexand aalexand closed this Nov 21, 2017

@KISSMonX

This comment has been minimized.

Show comment
Hide comment
@KISSMonX

KISSMonX commented Dec 22, 2017

Good Job!

@tylfin

This comment has been minimized.

Show comment
Hide comment
@tylfin

tylfin Aug 24, 2018

Doesn't look like this is available in Go 1.10.3? Is it planned for a 1.11 release?

tylfin commented Aug 24, 2018

Doesn't look like this is available in Go 1.10.3? Is it planned for a 1.11 release?

@nolanmar511

This comment has been minimized.

Show comment
Hide comment
@nolanmar511

nolanmar511 Aug 24, 2018

Contributor

I think the flame graph is available as part of go 1.10, though the flame graph's location may not be clear enough.
It should be part of the web ui. So, if you run go tool pprof -http=: /path/to/profile, a new window should open in your browser, and then you should be able to switch the view from the graph view to the flame view using the view drop-down menu.

image

Contributor

nolanmar511 commented Aug 24, 2018

I think the flame graph is available as part of go 1.10, though the flame graph's location may not be clear enough.
It should be part of the web ui. So, if you run go tool pprof -http=: /path/to/profile, a new window should open in your browser, and then you should be able to switch the view from the graph view to the flame view using the view drop-down menu.

image

@aalexand

This comment has been minimized.

Show comment
Hide comment
@aalexand

aalexand Aug 24, 2018

Collaborator

Note that the fix to auto-assign a random port is not in Go 1.10 (see #413), so the command you want is something like go tool pprof -http=:8080 /path/to/profile. And looks like flame graph is not in pprof that is in Go 1.10.3, as https://github.com/golang/go/tree/go1.10.3/src/cmd/vendor/github.com/google/pprof/third_party doesn't have the flame graph code. Go 1.11 should have it - https://github.com/golang/go/tree/go1.11rc2/src/cmd/vendor/github.com/google/pprof/third_party has the flame graph in.

Collaborator

aalexand commented Aug 24, 2018

Note that the fix to auto-assign a random port is not in Go 1.10 (see #413), so the command you want is something like go tool pprof -http=:8080 /path/to/profile. And looks like flame graph is not in pprof that is in Go 1.10.3, as https://github.com/golang/go/tree/go1.10.3/src/cmd/vendor/github.com/google/pprof/third_party doesn't have the flame graph code. Go 1.11 should have it - https://github.com/golang/go/tree/go1.11rc2/src/cmd/vendor/github.com/google/pprof/third_party has the flame graph in.

@rakyll

This comment has been minimized.

Show comment
Hide comment
@rakyll

rakyll Aug 24, 2018

Member

Flame graph is available in 1.11.

Member

rakyll commented Aug 24, 2018

Flame graph is available in 1.11.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment