Caching the requests persistently?

[yaqwsx]

Hello,
my issue is rather a question than an issue, however, your repository does not seem to have discussions enabled.

I am running a catalog of electronic components that is intended to be served from GH pages with no backend (https://github.com/yaqwsx/jlcparts). I stumbled upon your project and it seems it might be a suitable upgrade for my project.

I studied the project briefly and I see that you cache the requests in RAM. However, I was wondering if it would be possible to cache them either via Web Storage API or Cache API persistently, so the cache survives page reload (and have an external mechanism of purging the cache). If so, could you point me to where to look in the sources to add persistent caching support?

[phiresky]

my issue is rather a question
That's fine.

if it would be possible to cache them either via Web Storage API or Cache API persistently

Great idea!

Currently, all the cached data is stored in a sparse array here: https://github.com/phiresky/sql.js-httpvfs/blob/master/src/lazyFile.ts#L36 So to generalize that you could just replace that with a simple interface Cache {get(i: number): Uint8Array, set(i: number, chunk: Uint8Array)}.

The main problem you'll probably encounter is that the cache has to be synchronous to work with my code, since SQLite is synchronous, so e.g. IndexedDB won't work directly.

As far as I know, the localStorage API is pretty limited, both in amount of data that can be saved as well as the structure, but it should work. The Cache API probably won't work because ServiceWorkers can't respond to synchronous XHR.

So for a simple solution, I'd try using localStorage. You can also use IndexedDB (or any async storage) if you save to the cache in the background and load everything into memory on page load.

For a better solution, you probably need to make the whole thing async. The easiest way here is probably to base it on wa-sqlite/IndexedDbVFS instead of my project, and reimplement my lazyFile on top of that. The reason is that wa-sqlite provides an API for async VFSes based on emscripten asyncify.

[Tails]

You can check out how https://github.com/jlongster/absurd-sql does it. Ideally these projects would be merged!

[make-github-pseudonymous-again]

The main problem you'll probably encounter is that the cache has to be synchronous to work with my code, since SQLite is synchronous, so e.g. IndexedDB won't work directly.

Can you elaborate? The API of sql.js-httpvfs is asynchronous. Why does it matter that the SQLite is synchronous?

[phiresky]

All the VFS calls are called from within SQLite (like callbacks). Since SQLite calls them in a synchronous manner the VFS JS functions also have to be synchronous.

[make-github-pseudonymous-again]

@phiresky So you mean that in addition to the patch you have already made to sql.js to support byte-range queries, you would need to write another patch to make sql.js asynchronous?

[phiresky]

If you want to use sql.js specifically, yeah. But I wouldn't base new work on sql.js in any case

[phiresky]

Closing because I have no plans on continuing adding new features to sql.js-httpvfs, since the base is pretty hacky and reimplementing on top of a cleaner base would be preferrable. If I would do it again I would try using wa-sqlite which is better suited for this kind of thing.