db_dump is inefficient

[romw]

Reported by Nicolas on 10 Dec 38606274 05:20 UTC
Let's compare.

• Write a potentially big uncompressed XML file (over 2GB for SETI@Home), then use system() to run gzip on it (which reads it '''again''', writes something like 300MB, then deletes the original that took so long to write). Total I/O: 2GB read, 2.3GB write, plus the database reads (which are quite a lot too).
• Use zlib to write the file in compressed form in the first place. I/O: 300MB write, plus database reads.

Which would you choose?

I'll see if I can modify db_dump myself to use zlib, and post the patch. Otherwise, anyone is free to take the task.

Migrated-From: http://boinc.berkeley.edu/trac/ticket/717

[romw]

Commented by Nicolas on 3 Jul 38606279 21:20 UTC
In case it isn't clear: the first is what db_dump currently does :)

[ChristianBeer]

For reference: the db_purge tool already does 'in place' compression using pipes: sched/db_purge.cpp

[denravonska]

I added a proposal for this in #2767. Ideally, it should handle zip the same way but I was unsure if BOINC had an existing libzip dependency.

[AenBleidd]

Fixed in #2767