pass around metadata

[ThomasWaldmann]

in borg, there is a flow of data through the different stages / layers, but metadata of files / of content is missing mostly. we could use a metadata dict and pass it around with the data, e.g. as a tuple (meta, data).

e.g. currently, the compression component is statically set up, you choose it via commandline param.
it could be that an entry in the meta dict determines the compression that will be used.
the entry could default to whatever commandline says, but it could also be changed dynamically (e.g. if the file reader knows it is .mp3 and can't be compressed, so it sets meta["compression"]='none' for that data).

e.g. it could be also used for sparse files, so hole=True/False can get passed around.

[ThomasWaldmann]

related to #14

[ThomasWaldmann]

The infrastructure added in #934 was removed again in #2364. Trying to find out why...

[ThomasWaldmann]

Can't find out why.

I guess we need this for sparse handling (and maybe also for other metadata later).

@enkore do you remember?

[ThomasWaldmann]

Anyway, in #5620 I re-added a little bit of it to support communicating between chunker and hasher.

After the hasher, everything is still as it was - the compressor will not get any metadata yet.

[ThomasWaldmann]

Considering the metadata (about sparseness) produced by the chunker (mostly by the fixed chunker, a bit less by the buzhash chunker):

• if it is a piece of DATA: metadata not useful to compressor, the compressor will need to compress the data anyway
• ZEROS / SPARSE HOLE: the compressor will compress the each first <size> block of zeros and the chunk will be stored in the repo. any other all zeros chunk of same size will be deduplicated, so we do not need to pass this metadata to the compressor either.