Some day we should do a prototype of stream compression. Stream blocks to be written can be sent to a compression thread, which compresses the data and sends it to a writer thread.
Hopefully, the compressor can vary its effort according to available I/O bandwidth and CPU cycles.
The Google Snappy library might be interesting here: https://code.google.com/p/snappy/
This would be cool to test out. I added support in our CMakeLists in the snappy branch.
What should we try it out on?
I think it would be cool to adapt it to the serialization streams, since they already do not support seeking.
In the snappy branch I have now enabled stream compression at the stream_accessor level, meaning all streams are compressed with no change in API. However, seeking is not supported except to the beginning of the stream, so I will have to change this solution to something more permanent. In particular, stream reversing and TPIE stacks do not work, so I cannot really test this on a real world application yet.
@Mortal I guess we can close this issue now?