Reduce the size of bazel profile

[sluongng]

On our end, we started seeing some larger users with Bazel profiles hitting 200 MB in the gzipped format.

I think this might be a good time to start a discussion about ways to improve the profile format for storage and transfer.

I think an easy win while keeping the existing trace viewer compatibility would be adopting Perfetto protobuf format and best practices https://perfetto.dev/docs/reference/synthetic-track-event#interning-data-for-trace-size-optimization. But I'm open to other ideas as well.

[meisterT]

I am open to changing things in case it helps.

Have you tried to play around with interning (does it actually help over gzip compression?) and protobuf to see how much it reduces trace sizes?

I am little suprised to see such large profiles, I rarely see anything above 50MB. How many actions do you have in these large builds?

[meisterT]

slim vs nonslim is a 10x reduction in size, everything else will only help around the margins

[sluongng]

I think it's hard to tell the size of the build "before" it runs.

So perhaps if we can do --slim_profile=auto, where the events are only compacted after a certain threshold, would be ideal.
That way, for smaller builds, folks still can retain the high resolution of the profile. But for bigger builds, perhaps only the bigger spans in the critical path matter?

[meisterT]

That's a good idea and also something that I thought about before. I will experiment with this.

[meisterT]

You might want to give https://bazel-review.googlesource.com/c/bazel/+/329610 a try

[meisterT]

This has been submitted as 0da18f3 - please give feedback so that we can also select a good new default

[coeuvre]

I understand your main concern is storage, but i have built ztracing to load and render large profiles which you might be interested in. We are embedding this viewer internally in order to handle large profiles (a few GB after decompression). I'm also open to change the format.