IDBMutableFile / webkitRequestFileSystem

[ghost]

Chrome and firefox each have a non-standard way of storing and retrieving flat files that can be much faster and less overhead than using IDB for file-like data. In the browser I normally use https://github.com/random-access-storage/random-access-web to pull in all the hacks required and there is an IDB fallback if the faster versions aren't detected.

Using the random-access-storage API you'd get all of these backends with less work but that would involve replacing or wrapping the storage layer. I'm doing this for a similar wasm project (a spatial database) written in rust when it runs in the browser.

But all of that would make the project more useful and less absurd.

[jlongster]

That's interesting, thanks for the links!

I glanced at some of the filesystem APIs, but they seemed to vary so much and have contention across browsers I punted digging into it. I also did not see APIs for writing data to arbitrary positions, but clearly I missed that.

My biggest concern is locking. Writing data must properly lock the file, or the database can get corrupted. This must work across tabs. I glanced at random-access-chrome-file and it looks like it just uses a local mutex: https://github.com/random-access-storage/random-access-chrome-file/blob/master/index.js#L228

I think that's problematic if you have the app open in many tabs. While storing data in IDB is absurd, at least we have a way to guarantee write order because we can rely on its transactional semantics (which works everywhere!)

I'd love to unlock better perf & permanent persistence though by using APIs like that. Right now I wanted something that works across all browsers as baseline. I'll check out those projects & APIs though and see what's possible. There's also this new API? https://web.dev/storage-foundation/

Also: how did you find this? :)

[jlongster]

I just took a look at LockedFile: https://developer.mozilla.org/en-US/docs/Web/API/LockedFile

It looks like when you open a file with IDBMutableFile, it just returns a file that is permanently locked? So would I open the file whenever I want to read/write, treating it like a transaction? Instead of opening it once and then using APIs to lock/unlock the file. Is opening a file fast?

[jlongster]

I will play around with Chrome's API at least. I am curious about the perf difference, because I got IDB to not perform too badly.

I looked the the IDB fallback for random access: https://unpkg.com/browse/random-access-idb@1.2.1/index.js. It creates a new transaction for every read, and that's going to have a huge perf hit. One of the neat tricks in my library is the ability to keep an IDB transaction open over many requests, so if you do lots of reads (which sqlite does, it will requests lots of blocks separately) it will reuse a readonly transaction. That improved perf several times over.

[jlongster]

The biggest problem with IDB thought is unfortunately browsers might delete your data. That's a huge drawback, and for that reason alone I'll check out these other APIs.

[jlongster]

@substack Ok I got it working, thanks for your code for reference! https://gist.github.com/jlongster/ec00ddbb47b4b29897ab5939b8e32fbe

It's horribly slow right now, but that's not surprising - I'm not doing all the transactional stuff yet. It's calling entry.file for every single read, and for every write it's creating a new write.

How do you do bulk writes with this API? I can't find anything about it, and there has to be an improvement on this: https://gist.github.com/jlongster/ec00ddbb47b4b29897ab5939b8e32fbe#file-webkitfilesystem-backend-js-L29-L39. That's waiting for the write to complete before moving on, which is unsurprisingly slow.

Is there also any way to speed up the reads? Running the query that counts 1 million items is taking ~30 seconds, while the IDB backend only takes ~3.5 seconds. There's no writes there , so hopefully there's also a way to speed up the reads?

[jlongster]

Closing this as there's lack of interest. Will put my gist somewhere in hopes that someone can optimize it. If it can't beat IDB it's not worth including. (hoping newer storage APIs will solve it better)