cmd/trace: support for perfetto

[felixge]

The current catapult trace viewer is bit rotting and struggles with large traces. Perfetto UI could offer better performance and UX.

This was already proposed in #57175 but no dedicated issue existed yet.

Also see #57159 which could be a follow-up enhancement.

[gopherbot]

Change https://go.dev/cl/457716 mentions this issue: cmd/trace: experimental support for perfetto

[felixge]

The CL linked above implements a prototype for this. In practice it looks like this:

• Small Trace: Perfetto vs Catapult
• 280MB Trace from busy 96 core machine: Perfetto vs Catapult

Notes on the big trace:

• The catapult UI frequently freezes up completely while trying to view it. Perfetto seems to handle it without issues. I'm not showing it in the video because I don't want to share too much details from the big trace.
• The perfetto integration currently requires two clicks. This perfetto-dev thread explains the underlaying browser popup blocking issue.

[ianlancetaylor]

CC @golang/runtime

[prattmic]

Thanks for filing this! We certainly need to replace Catapult and Perfetto is the best alternative that I have seen, so overall I am very supportive of this.

Regarding #57315 (comment), until recently I hadn't really considered adding Perfetto support without implementing the trace proto format (#57159), as previously Perfetto didn't handle the JSON very well. But I tried again a few days ago, and it seems to work surprisingly well.

The Perfetto format looks daunting due to a huge number of custom event types for Chrome/Android [1], but the core ones (TracePacket, TrackEvent) seem straightforward, so I don't think that implementing proto support would be too difficult.

Perhaps we could add "experimental" links like in https://go.dev/cl/457716, and also work on the proto format. If both end up in the same release, great. If not, JSON is an OK fallback.

[1] These make me wonder if they would be open to adding Go-specific types for Ps and Gs rather than hacking them into existing constructs.

[prattmic]

One issue I did see was that some (but not all) flow events displayed incorrectly.

For example, these events somehow became a loop in Perfetto?

Catapult:

Perfertto:

[felixge]

Perhaps we could add "experimental" links like in https://go.dev/cl/457716, and also work on the proto format. If both end up in the same release, great. If not, JSON is an OK fallback.

Sounds good.

My colleague @nsrip-dd is already looking into the proto format. It looks daunting, but as you said, perhaps we just need to worry about a small subset of it.

One issue I did see was that some (but not all) flow events displayed incorrectly.

I could imagine other issues as well. I'm hesitant to spend a lot of time debugging this before we have a prototype for the protobuf format is ready. So maybe let's give that a few more weeks before deciding on the JSON option.

Meanwhile it would be great to know if the deep link integration itself seems okay. IMO the only thing that would have to change is the URLs if we switch to protobuf.

[prattmic]

Meanwhile it would be great to know if the deep link integration itself seems okay. IMO the only thing that would have to change is the URLs if we switch to protobuf.

Of course I'd prefer a direct link, but from https://perfetto.dev/docs/visualization/deep-linking-to-perfetto-ui it seems like this is our only option, so it is what it is.

On the scalability front, it would be even better if Perfetto didn't need to load the entire trace up front and instead loaded only what it needed on demand, which would eliminate the need for us to cut the trace into chunks for the browser.

Perfetto has a tool [1] for this, where a local process parses the trace and the UI makes RPC queries against it for the current view. In an ideal world, go tool trace could implement that RPC interface to get a really streamlined browser experience.

Unfortunately, that RPC interface is fairly large, and IIUC the core of it is a SQL query interface against the trace data. I think implementing this full API in Go and keeping it up to date as Perfetto evolves is probably too high of a maintenance burden.

I don't think we could ship the trace_processor binary either [2], as we don't have the tooling to build a C++ application for all ports during the release process. We could perhaps check PATH, and use that binary as an optimization if users have installed it themselves?

[1] My understanding is that this same processor is compiled to WASM and run in the browser for the typical full-file mode.
[2] Way out idea is to build trace_processor to WASM and run that from Go!

[felixge]

On the scalability front, it would be even better if Perfetto didn't need to load the entire trace up front and instead loaded only what it needed on demand, which would eliminate the need for us to cut the trace into chunks for the browser.

Yeah, that'd be awesome. But I think CL 457716 already gets us a little bit closer. I'm able to bump the split size from 100 to 500MB like shown below, and the Perfetto UI still works really well on the large trace I have.

-	s, c := splittingTraceConsumer(100 << 20) // 100M
+	s, c := splittingTraceConsumer(500 << 20) // 500M

But if I go to 1000MB it fails:

Unfortunately, that RPC interface is fairly large, and IIUC the core of it is a SQL query interface against the trace data. I think implementing this full API in Go and keeping it up to date as Perfetto evolves is probably too high of a maintenance burden.

Agreed.

[1] My understanding is that this same processor is compiled to WASM and run in the browser for the typical full-file mode. > [2] Way out idea is to build trace_processor to WASM and run that from Go!

That would be a fun thing to try out at some point :). But this would still require the Go runtime to ship a WASM runtime such as wazero (Apache 2). Is that realistic?

[prattmic]

Yeah, that'd be awesome. But I think CL 457716 already gets us a little bit closer. I'm able to bump the split size from 100 to 500MB like shown below, and the Perfetto UI still works really well on the large trace I have.

Nice. I imagine that the proto format will be even better here, as it should be more information dense.

That would be a fun thing to try out at some point :). But this would still require the Go runtime to ship a WASM runtime such as wazero (Apache 2). Is that realistic?

I don't know, it was just a fun thought, not particularly serious. :)