Streaming decompression

[veloman-yunkan]

Partial cluster decompression (#78) can be done if the decompression API provides a means to read and parse the data in a streaming fashion. Such an API will also enable representing in-memory clusters corresponding to compressed on-disk clusters as proposed in #395.

[mgautierfr]

This can be done without a streaming api. As the name imply, a streaming api means that we are reading sequentially the content (no random access). This is the opposite of what we do in libzim.

The buffer interface with a read(offset, size) can decompress the content up to offset+size if needed without changing the api.

[veloman-yunkan]

I target the compression API found in src/compression.h:

template<typename INFO>
std::unique_ptr<char[]> uncompress(const zim::Reader* reader, zim::offset_t startOffset, zim::zsize_t* dest_size);

which can only be used to decompress all of data hiding behind a reader. By streaming API I mean a use model where I can create a decompressor object functionally equivalent to a C++ istream and pull decompressed data from it as needed.

Pseudocode:

Decompressor d(compressed_data_reader);
int n = d.read<int>();
while (n--) {
  uint64_t size = d.read<uint64_t>();
  std::unique_ptr<char[]> blob = d.read_blob(size);
  ...
}

[mgautierfr]

We have this in our git history (#44)