"Blockless" IndexedDB VFS

[rhashimoto]

There are sometimes conflicting goals in writing VFS samples: one is to provide a good starting point for others to write their own, and another is to explore interesting approaches and techniques. I've been concerned that IDBVersionedVFS (originally IndexedDbVFS) is becoming too complicated and intimidating for someone to use as a model. So I thought I might write an additional VFS for IndexedDB where the logic was as simple as I could make it.

The normal way to implement a filesystem is to segment file data into fixed-size blocks. Even though I've done this a number of times, the part I hate writing is the data blocking and unblocking. It's fiddly to get right with lots of opportunities for off-by-one errors. And if writes can be unaligned then that requires read-modify-write, and that makes you want to cache those partially written blocks which is more state and more code... So I was trying to come up with a better way to express that, e.g. better than this and this.

I found a different approach that doesn't mandate fixed-size blocking. It isn't completely general - meaning that it can't handle every possible sequence of reads and writes - but our advantage is that SQLite is the only reader and writer, and its reading and writing patterns are very similar. All we require is that SQLite makes its read and write calls with these constraints:

1. Any overwritten data uses the same write offset and size. That is, if SQLite writes new data to replace old data in a file, it has to use the same write parameters (other than the actual bytes).
2. Every read requests data from only one write. That is, no SQLite read crosses a write boundary.

If these constraints are met, then each write can be stored to one object in IndexedDB, and each read fetches the one object that contains its data. The object data sizes are whatever SQLite passed (which are fixed for database files but not for other files). The last piece of the puzzle is how to look up the object that contains the data for a particular file offset, and the solution to that is to use the negative of the block address as part of the object key. To look up the object containing the data at offset, fetch the first object whose key is equal to or greater than [filename, -offset] (IndexedDB only searches in increasing order which is why the negation is needed).

I implemented this in a new class, IDBMinimalVFS, and it works (choose "IDBMinimal" in the demo)! Here is the much simpler corresponding read and write logic. I don't think it can be simpler than that.

SQLite will violate constraint 1 if you change the page size of the database file in-place, e.g. with:

PRAGMA page_size=8192;
VACUUM;

SQLite will violate constraint 2 if a memory journal is spilled to the VFS and then read with a rollback. I have only seen this when batch atomic writes are enabled, so this might not happen with the IDBMinimalVFS xDeviceCharacteristics configuration.

IDBBatchAtomicVFS is a more complicated example class that handles the above situations.