StructuredLogViewer fails to load large log files

[imag0r]

I have a build which produces 1.5GB log file with StructuredLog logger. When trying to load it to the viewer, I get OutOfMemoryException:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.Text.StringBuilder.ToString() at System.Xml.XmlTextReaderImpl.FinishPartialValue() at System.Xml.XmlTextReaderImpl.get_Value() at System.Xml.Linq.XContainer.ReadContentFrom(XmlReader r) at System.Xml.Linq.XContainer.ReadContentFrom(XmlReader r, LoadOptions o) at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options) at System.Xml.Linq.XDocument.Load(String uri, LoadOptions options) at Microsoft.Build.Logging.StructuredLogger.XmlLogReader.ReadFromXml(String xmlFilePath, Action1 statusUpdate) in C:\MSBuildStructuredLog\src\StructuredLogger\Serialization\XmlLogReader.cs:line 22

This can be easily fixed by unchecking 'Prefer 32bit' option in StructuredLogViewer project build settings, to allow the 64bit JIT for AnyCPU configuration.

[KirillOsenkov]

I need a fast and compact binary serialization format. XML was good for a prototype, but for large projects such as yours it is wasteful and inefficient.

I'll need to look into Protobuf.Net, Microsoft Bond or just roll a custom binary serialization format for the log files in addition to XML.

And yes, setting Prefer 32 bit to false is a cheaper workaround which I shall apply as well.

[KirillOsenkov]

I've made the app 64-bit for now, can you please run it, wait for it to auto-update, then close and run it again?

I'd be curious if setting it to 64-bit helps. Thanks!

[0x53A]

What would you think about using something like Sqlite as the storage format?
Then any consumer could (in theory) use custom sql queries to get the information they want, without you needing to implement a gui for everything.

[KirillOsenkov]

The appeal of StructuredLogger.dll as it is right now is that it's a single .dll without dependencies. This way we can optimize for performance and make sure that the logger doesn't slow down builds too much. Also if we control the serialization format we can optimize it as well.

Whatever we'll end up with as a storage format shouldn't matter since we'll be providing a public API to read that data and then you can use C#/LinqPad/LINQ to query that, which IMHO is a better experience than Sqlite can provide.

[KirillOsenkov]

Hi @imag0r and @0x53A. I've just published a new version that supports a compact binary log format: *.buildlog. When passing the log file name to MSBuild, use the .buildlog extension instead of .xml. The UI will now default to .buildlog.

I'll be closing this issue but let me know if there's anything else I can do to improve it.

I'd be curious to the numbers you get on the log file size for your large builds. How much better is it?