OPFS asyncer: in waitLoop(): Error: No waitLoop handler for whichOp # 0

[VegarRingdalAibel]

Might be something Im doing wrong.
Atm it feels more random if it works or not, and I think its because I have 5 workers/5 databases

Code running

Also tried this one console.log(this.db.affirmOpen()?.pointer, this.db.affirmOpen()?.filename)

It it weird all 5 show same pointer?
I do not know enough to say if this is to be expected....

[tomayac]

@VegarRingdalAibel As per the README, I'm closing this Issue here. @sgbeal, this might be a worthwhile problem to look into on your side, though.

[VegarRingdalAibel]

@tomayac
sorry missed that one. My bad.

@sgbeal tell me if you want me to create issue under sqllite / or you create a issue. Or you know what Im doing wrong.

[sgbeal]

It it weird all 5 show same pointer?

Not weird at all: each worker has its own independent virtual machine and memory address space. If all 5 start up identically, it's unsurprising that they have the same virtual addresses.

As to the error you're seeing, i have no clue as to the cause. It's never come up in 9ish months of testing. Where it's happening is clear, but why it happens is currently inconceivable to me. It seems very possible that the number of workers is the problem. Frankly, i consider 5 to be expecting too much from a browser environment, given the high memory requirements of workers and limited file-sharing capabilities of OPFS. Some degree of moderation is required.

(please pardon brevity - currently living on a boat and limited to a phone for internet access.)

[VegarRingdalAibel]

@sgbeal
Thank you for taking the time to answer.

Frankly, i consider 5 to be expecting too much from a browser environment, given the high memory requirements of workers and limited file-sharing capabilities of OPFS. Some degree of moderation is required.

Every worker have own file/db, so its not really sharing the file with anyone else. But yes it might be a bug in OPFS.
Maybe its because I close/open it alot?

   openDB() {
        this.db = new this.sqlite3.oo1.OpfsDb(`/${this.name}.sqlite3`);
    }

    closeDB() {
        this.db.close();
    }

Btw, only seen this issue when Ive created indexes or delete a lot.
Ill see if I can create very simple solution to reproduce the error with as little possible code during the week, so its easier for you to check out.

Hopefully its a simple fix somewhere 😄

[sgbeal]

Btw, only seen this issue when Ive created indexes or delete a lot.

That is an interesting hint. A few weeks ago we fixed a JS-side sync-related bug for journal files which could hypothetically be a factor in this problem. IIRC that fix went into 3.42.x but i will dig that out the next time i have access to an AC power outlet (in the next couple of days).

In the meantime, if you can manage to reproduce it reliably and post a tree which does it (public or direct to me, as you prefer: stephan at wanderinghorse.net), that would be a tremendous help.

Every worker have own file/db, so its not really sharing the file with anyone else. But yes it might be a bug in OPFS.

In that case there "shouldn't" be any contention issues with OPFS directly, but it is hypothetically feasible that that number of high-I/O workers is causing the browser to hiccup. Can you try it in Firefox v111 or higher, to hopefully rule out (or in!) a browser-specific quirk?

[VegarRingdalAibel]

Ill make a solution during the week so its easy for you guys to just clone/run to see the issue.

btw, did a little more testing now.
The issue goes away if I never do delete/recreate index in all worker at once, just do those in sequence (let one worker finish before asking the next one to do the same)

So its like every worker is fighting for resources and maybe timeout to quickly?
I dont know the limits of OPFS...
But Ill try and see if same issue is in firefox later too.

[sgbeal]

So its like every worker is fighting for resources and maybe timeout to quickly?

That sounds plausible, but the error you're reporting seems to indicate that either the browser is misbehaving (posting an Atmics-related message when it shouldn't) or we have overlooked a finer detail of the Atomics API and are improperly handling some as-yet-unknown error case.

I dont know the limits of OPFS...

As far as timing goes, those details are all "implementation-defined". The storage limits are documented fairly well but are browser- and device-dependent. Concurrency is an uphill battle with OPFS, but if your workers all use separate databases then there shouldn't be any concurrency issues, barring browser quirks.

But Ill try and see if same issue is in firefox later too.

Please do. My apologies for being unable to be more proactive, but living (temporarily) on a boat drastically limits my computing ability.

[sgbeal]

(Back on dry land for a couple of hours and have looked more closely at the JS code...)

There's a potential change which might resolve the "no waitLoop handler" error but it risks other fallout. Since i cannot currently reproduce the problem i'm hesitant to check in this change, but if you can try it locally, we can probably figure out whether this breaks anything in the attempt to resolve something else.

In sqlite3-opfs-async-proxy.js is a line which looks like:

        if('timed-out'===Atomics.wait(
          state.sabOPView, state.opIds.whichOp, 0, state.asyncIdleWaitTime
        )){

We can try changing that top-most bit to:

        if('not-equal'!==Atomics.wait(

that will (i believe) effectively squelch the "No waitLoop handler" error but it will further hide whatever is causing that errant 0 value to be stored in the "whichOp" slot of the SharedArrayBuffer from which the SQLite driver and the async OPFS part coordinate the sync-vs-async communication.

Another option is to replace this part (a few lines down from the above):

        const hnd = opHandlers[opId] ?? toss("No waitLoop handler for whichOp #",opId);

with something like:

        const hnd = opHandlers[opId];
        if(!hnd) continue;

similarly, that will squelch the error message but further hide the underlying issue.

Regardless of whether we do that or not, the loop in which the exception is thrown continues to loop, so it "should" recover either way. From that perspective, the above doesn't "fix" anything, so might not be worth the effort of trying out :/.

[sgbeal]

After some testing, i've gone ahead and made the above change, squelching (i believe) that particular instance of that particular console error. There appears to be a potential window for an exception to be thrown which, in turn, leads to the side effect of the "No waitLoop handler for whichOp # 0", in which case that report is harmless but unduly scary. That change will be in v3.43. The prerelease snapshot has been updated with this change.

@VegarRingdalAibel please be sure to try the "unlock-asap" mode to see if that helps as well:

https://sqlite.org/wasm/doc/trunk/persistence.md#opfs-unlock-asap

i don't believe it will for your particular setup, but it can't hurt to try. It should have noticeable effects on concurrent access to the same db file, though.

[VegarRingdalAibel]

👍
Ill give a try later today.

I was thinking if its getting timeout/fighting for resources we might need to up the asyncIdleWaitTime/maxTries ?
Is that a option I can pass inn to it?

[sgbeal]

Is that a option I can pass inn to it?

Those are not currently configurable. The asyncIdleWaitTime is necessarily global across all connections because of how the cross-thread communication works. We could possibly make the max retry count configurable per connection. The next time i'm back on land i'll explore that. It's not a magic bullet for improving concurrency but it is a value which would ideally be tweakable.

[vegarringdal]

Made a small weird sample here
https://github.com/vegarringdal/sqllitedebug

Not tried to improve anything, needed something that broke first..
Important to have dev tools closed, will break faster.

Sometimes take more time to break it... really depends on operations usually taking more time, like making index/large deletes

[Matt-TOTW]

Hello, I'm experiencing this issue, or one very similar to it.
Like vegarringdalI am also running multiple workers, where each worker is connecting to its own db/opfs file so I don't see how concurrency could be an issue. At no point are separate threads connecting to the same db. Here are the console warnings and errors

It's probably worth mentioning that I often don't see that first error

When I slow things down so that the connections don't occur simultaneously, all is well. Also, it seems much more likely to happen with longer running queries.
Each db is quite large ~1.5Gb.

There is one difference that I am seeing compared to vegarringdalI. When I revert the suggested changes ie back to
if('timed-out'===Atomics.wait(
I am not seeing the No waitLoop handler for whichOp error - so I'm not certain where exactly the problem is happening.
Scratch that - I have now seen that error too, although it's far from every time.

@vegarringdal, did the above solve your problem?

[sgbeal]

When I slow things down so that the connections don't occur simultaneously, all is well. Also, it seems much more likely to happen with longer running queries.

This is the first time in more than a year the problem has been reported and we unfortunately know nothing about it beyond the other comments in this thread. In short, it appears to be browser-specific, only happens under specific severe loads, and manifests as an "impossible" result code.

If you do not require any concurrency, i'd recommend trying the other OPFS VFS, which does not require the Atomics API and is much faster (the main tradeoff being a complete lack of concurrency):

https://sqlite.org/wasm/doc/trunk/persistence.md#vfs-opfs-sahpool