irmin-pack: add V5 version

[metanivek]

This PR makes changes for the upper control file needed for the lower layer and bumps the irmin-pack version to V5.

• Add volume_num to upper control file.
• Change version of volume control file to V5.
• Refactor checksum code to model valid and invalid payloads and allow checking checksums when upgrading.

I ran into one small issue where I wanted to use Version.t option as a type in the control file (for tracking upgrades) but ran into an error in the tests that seems to be coming out of repr (or maybe the version.ml code -- my brief investigation didn't shed light on what was going on). To get around this, I changed the type to int option and am using Version.to_int V4 when setting the value during an upgrade. It's not the most ideal but not the worst thing either. If anybody wants to try to dig further... 👼

[Firobe]

As far as I can tell, this seems to make sense!
Just to make sure I have a good understanding of the invariants around volume_num though, is it right to say that (in an ideal world when everything is implemented):

• if volume_num > 0, then the store definitely has a lower layer
• if volume_num = 0 after opening it, then the store definitely doesn't have a lower layer during this session (but one may be added at a later open), since even a newly created lower would have volume_num = 1 after migration/creation

?

[metanivek]

@Firobe thanks for the review!

For definitional purposes, a lower layer contains 0 or more volumes, so volume_num = 0 doesn't necessarily mean there is not a lower. You need more information to determine that. Said the other way, having a lower layer does not mean you have any volumes.

I'm currently working on the code to load the lower and it is modeled as an option type on the file manager. If a lower root is configured, it is Some; otherwise, it is None. This will be the primary way of "knowing" if a lower exists or not.

Here is how I am currently thinking about when to create the first volume in a lower (I think this agrees with what you said but open to discussion!):

• When creating a store that has configured a lower root, we could create the first volume at that time. (this is code I am currently working on)
• When opening a store in rw that has a configured lower root, this store needs a migration, so we create the first volume and move the suffix into it. (aside: we should discuss the nuances of the signal to migrate to make sure we handle all scenarios)

All other increments of volume_num will be through an explicit call to add_volume (code I am also working on).

So, to loop around to where we began: I think in practice volume_num > 0 does indicate the presence of a lower layer since we create the first volume when the store is created/opened, but it is not the primary signal. The primary way of knowing is the lower_root configuration parameter. Although, this brings up a question of what we do if lower_root is changed since we are not directly correlating the two values (we don't record on disk any config information, currently). The current expectation would be that if a user changes the lower_root value that they also moved the volumes that were in the previous path; otherwise, the store would not open correctly.

[Firobe]

Thank you for the explanation! That was more or less what I had in mind: in practice yes, but not the primary signal. In what I'm working on I've used a placeholder optional field in file manager as well to track that, so that'll be easy to merge!

[art-w]

It looks good! Regarding the version issue, I'm guessing that either the polymorphic variant don't play too well with repr, or that the encoding changes depending on the number of constructors? (which would explain why status has some reserved T1...T15 constructors) In any case, an integer version makes sense :)