Expectations of ACID properties from the datastore for different error scenarios

[schomatis]

I would like to get an idea of what are the (implicit) expected properties of the datastore being used by IPFS during different error scenarios, e.g., if a system crash occurs it would be acceptable to lose the last inserted blocks, if there is a disk corruption would it be expected for some of the data to survive? (Maybe a size inversely proportional to the corruption magnitude.)

Currently the default flatfs datastore is just a proxy for the file system, it doesn't add much complexity over it, so it can be said that it is as robust as the file system lying underneath. Badger, on the other hand, adds another layer of abstraction to the block storage (mainly through the grouping of keys and values in different files) that could potentially make it more susceptible to the error scenarios described above.

As a simple example consider a corruption in a single disk data block: for flatfs that would only represent (except rare cases) a single IPFS block corruption (which could be easily detect through its hash with ipfs repo verify), but for badger that could have more subtle consequences (e.g., when the corruption modifies an index in the SST) which may not be so easily detectable (or in the other extreme, it could result in a very visible case of many corrupted blocks).

I'm not looking for any type of precise quantification of data loss but just to get an idea of what a user expects from the IPFS storage service (expectations that may have been set from his current use of the flatfs datastore, which he would expect to keep maintaining when transitioning to badger). Badger will never be as robust as flatfs and that is expected (the loss in robustness is more than compensated with its performance gain) but before making the transition it would be nice to have a qualitative idea of what is being traded between both datastores in terms of persistence and consistency, and how (or if) could that trade-off be mitigated.

For more context there is a blog post where Badger discusses the tests performed to be more crash resilient, the paper of the ALICE framework used for those tests and an issue of a corrupted database that may have been due to disk corruption.

[manishrjain]

FYI. Added a Truncate flag to avoid automatically truncating the value log. Instead, Badger by default would return an error if truncation is required.

dgraph-io/badger#452

Badger will never be as robust as flatfs and that is expected

I wouldn't be so sure. A simpler store doesn't necessarily equate to a more robust store when the underlying storage itself is quirkly. File systems are strange beasts, and LSM trees have been proven over time to have dealt with and be a lot more reliable given the quirks of file systems. LevelDB has been the core basis of storage at Google for a very long time. So much so that Google's file system is now built on top of Bigtable, built on LevelDB. We've built Badger with many checks and bounds in place to ensure that we operate at the same level of reliability as other key-value stores.

Any solution which directly talks to the file system needs to be tested thoroughly to deal with various levels of potential corruption -- something that we've done for Badger via ALICE. I'd recommend doing that for flatfs to gain more trust in that store.

[schomatis]

@manishrjain Thanks for adding the flag and also for your input on this subject.