Skip to content

long-lived IndexedDB transactions #29

rhashimoto started this conversation in Ideas
long-lived IndexedDB transactions #29
Aug 16, 2021 · 0 comments

I modified the sample IndexedDB VFS to try extending the life of IDB transactions to reduce the transaction overhead. This is inspired by some game-changing work from @jlongster on IDB with SQL.js.

Keeping an IDB transaction alive in an asynchronous context is tricky because they automatically close at the end of an event loop when no outstanding requests remain. You have to determine when the transaction might be idle and issue an extra request to bridge to the next VFS call. And not only that, a transaction has an "inactive" state when it won't accept requests - you can only issue requests in an event loop iteration in which it is created or it returns results.

I'm using two strategies for these bridging requests. When a file is locked, an indefinite number of extra requests will be made to keep the transaction alive, so all I/O made under a lock will be in a single IDB transaction. When a file is not locked, a small configurable number (currently 2) of extra requests will be made, and if exhausted then a new transaction will be opened the next time one is needed. In general, SQLite requests a lock for the main database file and not for journals.

This change makes a ballpark 2x to 4x improvement for most tests on the benchmark page on Chrome on my desktop machine with the default journaling, which is also written to IndexedDB. The improvements are even more dramatic with memory journaling, where many tests improve by 15x.

Further improvements are still possible, especially when journaling to IndexedDB (though this should probably be avoided for all but the most RAM-needy applications). My VFS implementation was modified to use a separate IDB database for each SQLite file, which now seems like a mistake because creation/deletion, which happens a lot for journal files, is slow. My motivation for putting files in separate IDB databases was so that testing for file existence (which also happens a lot for journal files) could be done using IDBFactory.databases which doesn't require a transaction, but I didn't realize until later that Firefox doesn't have it yet and the fallback method of opening the database, checking if it was created, and deleting if so is agonizingly slow (fixed).

I also removed the LRU cache from the implementation so now there is no VFS cache at all. There are likely gains possible from adding a more specialized cache back in. Without a cache, a partial or unaligned write requires a block read first so a 1-block cache could help. This occurs a lot with journal files (1-block write cache added).

Replies

0 comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Ideas
Labels
None yet
1 participant