Release 3.5

[icristescu]

TODO list for irmin 3.5. The list is not exhaustive yet because more tasks will pop as we go.

Chunked Suffix

The highlight of 3.5 is the suffix chunking. Follows the list of subtasks for it, listed in topological order of dependency:

• Update V3 to V4 virtual migration Expose a gc_commit function #2097 (assigned @icristescu)
• UT migrations Some more migration tests #2113 (assigned @icristescu)
• Implement chunk files (assignee(s) @metanivek, reviewer(s) @Ngoguey42)
• Add chunk-related infos to GC stats
• Update lib_context replay to call split (assigned @Ngoguey42)
• UT the split function edge cases (assigned @icristescu)
• UT a GC that targets the last object of a chunk. That chunk should be deleted.

Implement chunk files's subtasks:

• Reading/writing existing suffix as a chunked suffix
  • Update Layout irmin-pack: switch to V4 layout #2114 (assigned @metanivek)
  • Chunk file abstraction irmin-pack: introduce chunked suffix abstraction #2115 (assigned @metanivek)
  • Add chunk suffix CF fields irmin-pack: add chunked suffix fields to control file #2117 (assigned @metanivek)
• Add split/create chunks (assigned @icristescu) Add split function to create a new chunk  #2118
• Implement chunk-based GC (assigned @metanivek) irmin-pack: update GC to work with chunked suffix #2126
• Test chunk-based GC

Wishlist

Wished features for irmin 3.5:

• Cleanup residual files at open_rw and on failed GCs Cleanup residual files #2095 (assigned @art-w)
• Move the GC commit from the suffix to the prefix Move gc commit from suffix to prefix #2102 (assigned @icristescu)
• Include a checksum in the control file Checksum control file #2119 (assigned @art-w)
• Expose to the user the commit_key of the latest GC Expose a gc_commit function #2097 (assigned @icristescu)
• Create a new GC benchmark based on the lib_context replay Gc stats #2089 (assigned @Ngoguey42):
  • Gc stats #2089
  • Add old file sizes to worker stats #2116
  • irmin-pack: Diffs for gc benchmark #2129
• A GC runtime and memory improvement GC reachability traversal in revorder #2085 (assigned @art-w)
• Sort out the use of read_in_suffix_and_prefix irmin-pack: asynchronous gc #1950 (comment) Move gc commit from suffix to prefix #2102 (assigned @icristescu) (+ @Ngoguey42 doing a refactoring of it)
• Fix the ./tezos-node storage command line for minimal indexing stores Traverse gced pack stores #1919 (assigned @icristescu)
• Snapshot export create a new store with only one commit, the import then becomes just opening the store:
  • Initial implementation Gc-based snapshots #2125 (assigned @icristescu)
  • Make snapshot canonical ?
• Add tests for dispatcher irmin-pack: The dispatcher has no tests #2092
• Catch decoding errors to reraise/return clean exceptions/errors Collection of IO and GC improvements #2039
• Improve crash consistency again irmin-pack.unix: idea to improve GC crash consistency on startup #2082 (assigned @art-w)
• Make conversion between physical offsets and (global) offsets require explicit casting (assigned @clecat)
• Add repo config in control file (to be discussed)
• Log to disk all the important activities on a pack store (to be discussed)

All the above tasks can be done in parallel.

[metanivek]

Closing since 3.5 has shipped. Incomplete items were on the wishlist and either have associated issues or are still tbd.