[Calltree Comparison (Talos)] Ability to compare multiple profiles

[mstange]

Sometimes you want to measure the performance impact of a change. You'll usually get a profile from before the change and a profile from after the change, and then you want some way of comparing the two.

At the moment, the only way of doing that is to load the profiles in two different tabs and to repeatedly switch between the two. That's not a great experience.

It would be nice to be able to select two profiles to compare, maybe pick one thread from each of them, and then have a few ways of comparing them:

• Difference tree: Let's say the profiling interval is 1ms. In the regular call tree, in order to compute the time for a call tree node, you just sum up all the samples under that call stack, and then you sort by that number, and the callstacks with the highest number show up at the top. In a difference tree, you could count all samples from the new profile as 1ms and all the samples from the old profile as -1ms. Then you still do the same summing of numbers in the call tree, and you end up with call tree nodes that can have negative or positive costs. Call tree nodes with positive numbers are call stacks which are present in the new profile more than in the old profile (i.e. the parts that got slower), and they'll be sorted to show up at the top.
• Matching up timelines: Sometimes it's nice to compare the information represented in the timeline views at the top with each other visually, both the stack graph and the interval markers. So you'd want some way of moving the time lines of the two threads underneath each other until they match up somehow.
• Comparing information contained in markers: If I have a change that affects composite times, I'd like to have some way of comparing the composite markers on the Compositor threads of the two profiles. E.g. compare the average time of a composite taken before and after.

Comparing profiles is a big subject and we'll probably need to experiment with a bunch of different views until we find something that works.

┆Issue is synchronized with this Jira Task

[PaulBone]

For comparing markers, I'd like to see statistical information about the markers reported (probably on a new pane) such as for duration of GCMinor events I'd like to see min, 25%, mean, 75%, 90%, max, for example. When comparing profiles I'd like to see box-and-whisker diagrams for these stats, possibly run t-tests or other statistical tests and so on. I can help with this.

[violasong]

Sounds like we need:

• A way to "import" one profile into another profile tab via URL or local file
• Perhaps also a way to display a history of the last n recorded profiles in the Profiler toolbar panel, which could allow you to select two profiles and click "Compare"?
• A way to show the two profiles' timelines in the header area, lined up with each other
• Ability to select and compare markers in each profile?
• Fine-grained comparison in the call stack (and possibly also other panels?)

I feel this can be made possible by ideas we were having about improving the header timeline by only showing one thread/activity row by default. We'd need to be able to:

• Auto-select the most interesting thread of the profile and display only its events
• There's also a need to automatically display pertinent markers/events from other threads

This feature is kind of intense so let me know if I seem to be misunderstanding anything 😅

[violasong]

Also, re: call stack comparison, would it be helpful/possible to display the two profiles in two columns like a diff tool, connecting/highlighting the parts that are different and using some green success color to indicate the side that's more performant?

[PaulBone]

This should also use #448

[gregtatum]

I feel this can be made possible by ideas we were having about improving the header timeline by only showing one thread/activity row by default.

My comment was for showing one thread for the Timeline panel, not the header. So I think we need a mechanism to select which threads we are interested in comparing.

Auto-select the most interesting thread of the profile and display only its events

I'm not sure how this would work or if it's feasible.

Also, re: call stack comparison, would it be helpful/possible to display the two profiles in two columns like a diff tool, connecting/highlighting the parts that are different and using some green success color to indicate the side that's more performant?

The problem here is that the stacks won't always line up. A top functions view would be much easier to do this with, and would be very nice. The call tree is dependent on the order that code is called. Often performance comparisons are about changing that order to make certain things faster.

Possibly what I would like to do would be to use the call tree and focus in on a part of the call tree, then do that for a second profile. Afterwards I would like to compare timing on top functions. I could see a diff-like approach for that.

@mstange had some thoughts on computing a diff call tree. I'm not sure if that's written down somewhere.

@violasong did I miss anything?

[violasong]

Here are some mockups - let me know if you have any feedback! Also, do any of the other panels need design work for this?

Importing a profile into another profile:

Profile comparison with difference tree:

[clarkbw]

Need to file break out issues for this.

Suggest starting with the Call Tree comparison view and waiting on the "top thing"