Produce msbuild binlog files for all build phases that today produce the triplets of logs / errors / warnings

[trylek]

Initial cost estimate: 2 weeks
Initial contacts: @trylek, @safern, @ViktorHofer

Review our current policy of emitting MsBuild text-based logs and maximize the use of binlog files that contain much more complete information usable for investigating build issues.

• Measure MsBuild performance w.r.t. emitting the text-based vs. binlog files;
• Use the measurements to design which product / test build steps should newly produce binlog files in addition to or instead of the existing text logs without substantial performance degradation;
• Review the design on the @dotnet/runtime-infrastructure alias;
• Implement the design.

[KirillOsenkov]

Note that there's an API to playback the .binlog into a text log with any level of verbosity.
Also it's simple to write a console app that will read all the warnings/errors from the binlog and output them to a text log.

However note that reading a .binlog on Core is 10x slower than on Desktop because of this bug:
#39233

Adding /bl does slow down your build, but not as much as adding a text log with diag verbosity.
My recommendation is to use /bl and no text logs, and generate text logs either via a little tool or on demand.

Also you can view binlogs online using https://live.msbuildlog.com so maybe integrating that into the CI somehow might also be interesting.

Finally, we have a whole bunch of bugs and a plan on how to reduce binlog overhead, but we haven't had the time to act on it. See here for details: https://github.com/KirillOsenkov/MSBuildStructuredLog/wiki/BinLog-Perf

[am11]

https://live.msbuildlog.com

Thanks for sharing this link! 😎

[KirillOsenkov]

Update: we have found a workaround for the slowdown I mentioned when reading .binlogs. Just wrapping the GZipStream with a BufferedStream solves the issue.

Check out https://github.com/vatsan-madhavan/MSBuildStructuredLogSummary for an example of how to read all the errors and warnings from the binlog.

[trylek]

Thanks Kirill for the update - I assume this is basically an extra argument for switching over to binlogs wherever possible.