Stream json data to a file to save 30% of the memory.

[rhpvorderman]

• This comment contains a description of changes (with reason)

I noticed that using --no-data-dir saved more than 300 MiB when working with the multiqc/test-data. This is odd. Turns out the internal data is serialized into a string before it is written. Using a streaming method saves the memory.

before:

/usr/bin/time bash -c "multiqc --force ../test-data/ 2>/dev/null"
26.09user 3.37system 0:28.28elapsed 104%CPU (0avgtext+0avgdata 1030072maxresident)k
320inputs+558480outputs (5major+327488minor)pagefaults 0swaps

After:

/usr/bin/time bash -c "multiqc --force ../test-data/ 2>/dev/null"
25.82user 3.61system 0:28.29elapsed 104%CPU (0avgtext+0avgdata 674052maxresident)k
464inputs+558440outputs (5major+356765minor)pagefaults 0swaps

I would love to reduce the memory further, but a lot of the memory profile shows stuff that has already been touched by other PRs. As stated before the LZ compression is a major culprit, using 200 MiB. If it could be replaced with gzip that usage would simply vanish. Another culprit is reading and storing lines from files, but that is already touched upon by #2505 and I suspect that that will have much better memory use as lines are not saved permanently. I added the profile report.
memray-flamegraph-multiqc.9194.html.zip

This change is pretty small. While I do not like that it can dump to file (returning None) and dump to string (returning str) it is good that the replace_nan function is still used.

[vladsavelyev]

That's awesome! I changed a few more places where JSON can be written directly to file.

[rhpvorderman]

Awesome. Thanks!