borgstore recently got defrag, that can transfer blocks from source files to a target file on the server side, to avoid that the client has to download all the stuff and upload the new stuff again.
to work with that, the client has to submit a list of blocks (source_name, offset, length) to transfer from the source items to the target item. it can compute a new storage name for the target to support sha256 content hash storage names, but it can't do other computations (e.g. update header/trailer contents if there are counts in there).
encryption:
- if encryption is used, the blocks need to be independently encrypted.
- we don't want to encrypt each hashtable entry independently, that would add too much overhead due to the nonce and the tag and the usually relatively small size of a HT entry.
- so we need bigger blocks: we could sort the entries by a value tuple element, so multiple index elements group together in a useful way.
rabbit hole:
- to defrag, we need to know what's where (name, offset, length)
- do we need an index for the index? %-)
Is it worth it?
borgstore recently got
defrag, that can transfer blocks from source files to a target file on the server side, to avoid that the client has to download all the stuff and upload the new stuff again.to work with that, the client has to submit a list of blocks
(source_name, offset, length)to transfer from the source items to the target item. it can compute a new storage name for the target to support sha256 content hash storage names, but it can't do other computations (e.g. update header/trailer contents if there are counts in there).encryption:
rabbit hole:
Is it worth it?