Should the test suite also run on Firefox?

[lalexdotcom]

While working on #372 I ran the whole suite on Firefox too (Firefox 153 through Playwright 1.62.1), and I wonder whether CI should do the same.

Why Firefox. It takes paths Chromium never does. Without readwrite-unsafe access handles ('mode' in FileSystemSyncAccessHandle.prototype is false there, as on Safari), OPFSAdaptiveVFS and OPFSWriteAheadVFS keep exclusive handles, and several recent fixes were Firefox-only or much worse there:

• OPFSAnyContextVFS: publish writes where SQLite ends them #363: the read after a VACUUM failed with disk I/O error on Firefox only;
• OPFSAdaptiveVFS: release the open lock when an open fails #369: the open lock leak never shows on Chromium;
• OPFSWriteAheadVFS: close the write-ahead file acquired beside one that failed #367: the leaked write-ahead handle costs one handle on Chromium, but blocks every reopen on Firefox.

Firefox supports JSPI now, so all three builds run. And Playwright's WebKit on Linux has no OPFS, so Firefox is the practical way to cover the no-readwrite-unsafe side in CI.

What it would take. Everything passes on Firefox except two things, both on master, both in the tests rather than in a VFS:

• OPFSWriteAheadVFS's multi-connection tests (vfs_read_freshness, and sql_0005 in sql.test.js) open a second connection, which fails on Firefox with unable to open database file: without readwrite-unsafe, the VFS keeps its handles for the connection's life. They would need a skip where readwrite-unsafe is missing. vfs_read_freshness is mine (OPFSWriteAheadVFS: read the WAL to its end when a read transaction begins #365), so that assumption is on me.
• That sql_0005 failure then hangs sql.test.js. The worker whose open failed is never destroyed (its cleanup is registered after the open), so it keeps its session directory open. The next context.create() with reset cannot empty OPFS, maybeReset gives up after 10 s without posting anything, and TestContext.create() waits forever. Jasmine runs in random order, so it only hangs when a resetting spec runs after that failure. Registering the destroy before the open, and having create() reject when the worker fails during setup, would turn the hang into an ordinary failure.

Plus the launcher itself: @web/test-runner-playwright with Firefox in the config, and Playwright's Firefox installed in the workflow.

If that sounds worthwhile, I'd be glad to send it as a PR: the launcher, the skip, and the two harness fixes. And if Chromium-only is a deliberate choice, no problem at all.

[rhashimoto]

And if Chromium-only is a deliberate choice, no problem at all.

I'm not opposed to some testing diversity, but is Firefox the best choice, given it's declining usage and precarious finances? Is the OPFS problem with Webkit because Playwright runs it in private mode by default, and if so do we know whether that is painful to override?

[lalexdotcom]

Thanks for the question, it was the right one. You were right about private mode, and it changes my answer: WebKit looks like a better choice than Firefox.

WebKit's OPFS under Playwright. "OPFS is currently not supported in ephemeral WebKit contexts" (Playwright docs), and browser.newContext(), which @web/test-runner-playwright uses by default, is ephemeral. A persistent context fixes it, and it is one option in the config:

playwrightLauncher({
  product: 'webkit',
  createBrowserContext: () =>
    webkit.launchPersistentContext(mkdtempSync(join(tmpdir(), 'wtr-webkit-'))),
}),

It needs Playwright 1.63 or later. On 1.62 WebKit has no navigator.storage in any context, which is what I had seen and wrongly put down to the Linux port; microsoft/playwright#41984 fixed it. With 1.63 (WebKit 26.6), sync access handles, createWritable and Web Locks all work in a worker. There is no readwrite-unsafe, and a second handle on a held file fails at once with InvalidStateError. That is also what Safari 27.0.1 does on a Mac, so the engine covers the same path Firefox would. It also runs the JSPI builds: WebAssembly.Suspending is there, in Safari 27 too.

The suite on that WebKit, current master, one file per job on GitHub Actions and locally:

• Everything passes on every build, except what follows.
• vfs_open_last_error (mine, Record the cause of an OPFSCoopSyncVFS open that failed asynchronously #357) expects NoModificationAllowedError, and WebKit throws InvalidStateError. The test should accept either.
• vfs_read_freshness (mine, OPFSWriteAheadVFS: read the WAL to its end when a read transaction begins #365) needs two connections on OPFSWriteAheadVFS, which needs readwrite-unsafe. It fails as on Firefox and should skip there.
• sql.test.js hangs, the same way as on Firefox: sql_0005 cannot open its second connection, and the harness turns that failure into a hang.
• IDBBatchAtomicVFS.test.js hangs intermittently, and this one is a WebKit bug: a worker terminated while an IndexedDB write is in flight never releases its connection, so the next spec's reset calls deleteDatabase(), which never completes. It is WebKit bug 324094, fixed upstream last month. Safari 27.0.1 still has it (a short page with a worker writing to IndexedDB reproduces it there), and the WebKit in Playwright's next release does not. Closing the VFS in TestContext.destroy() before terminate() avoids it: 0 hangs in 20 runs, against 8 in 20 without.

That last one may be worth knowing beyond the tests: per the WebKit report, a database stranded this way stays unusable for the rest of the browsing session, and in the suite it took no more than terminating a worker without closing its IDBBatchAtomicVFS.

If you'd like WebKit in CI, I'd be glad to send a test-only PR: the launcher in the config and the workflow, plus the adaptations above. Firefox needs the same adaptations, so it could come along in the same PR or stay out, as you prefer.

[rhashimoto]

Running CI tests on all three browsers by default would be great if you're up for that. It would be really nice if testing could easily be restricted to a single browser during development, e.g. with an environment variable.

[lalexdotcom]

Thanks, I'm up for it. Before I start, here are a few choices that are yours to make rather than mine. Each one comes with what I'd do by default.

1. Which Chromium. CI runs Chrome 129 today, through browser-actions/setup-chrome and chromeLauncher. Firefox and WebKit will need Playwright anyway (@web/test-runner-playwright), so Chromium could go through it too, and then the three engines share one launcher and one install step. Playwright can also drive a real Chrome instead of its bundled Chromium: channel: 'chrome' uses the installed stable Chrome, and executablePath uses a specific binary, such as the one setup-chrome installs. Playwright only guarantees its own bundled browsers, so that last option is at our own risk. Which would you prefer?

• Playwright's bundled Chromium
• a real Chrome through Playwright, either stable or pinned to a version
• chromeLauncher kept as it is, with Playwright used only for the other two

Also, is 129 a deliberate pin (for JSPI, perhaps), or just the version that was current when you last bumped it?

2. Restricting to one browser. I'd read a comma-separated list from an environment variable, for example WTR_BROWSERS=webkit yarn test, and run all three when it's unset. Does that shape suit you, and do you have a preference for the name?

3. CI layout. Each CI run tests twice today: once with the checked-in WASM, then again after make. I'd use a matrix with one job per browser and fail-fast: false, each job doing both passes. If that's too much CI time, I'd drop the pass with the checked-in WASM rather than the one after make. A PR can change how the WASM is built, and only the post-make pass tests that change. The pre-build pass could run on Chromium only, or not at all. Which would you prefer?

4. Tests that cannot pass as written without readwrite-unsafe. Neither Firefox nor WebKit has it.

• vfs_read_freshness (mine, from OPFSWriteAheadVFS: read the WAL to its end when a read transaction begins #365) and sql_0005 both need a second connection on OPFSWriteAheadVFS. I'd skip them when FileSystemSyncAccessHandle has no mode, rather than rewrite them. Is that OK with you?
• vfs_open_last_error (mine, from Record the cause of an OPFSCoopSyncVFS open that failed asynchronously #357) expects NoModificationAllowedError, but WebKit throws InvalidStateError for a second handle on a held file. I'd accept both. Is that OK?

5. The harness turning a failure into a hang. When sql_0005's second open fails:

• that worker is never destroyed, because its cleanup is registered only after a successful open;
• the next reset can't empty OPFS, and rejects after 10 s without posting anything;
• TestContext.create() then waits forever.

I'd register the cleanup before the open and make create() reject when the reset fails, so a failure is reported as a failure. Is that fine in the same PR, or would you rather have it as a separate one?

6. WebKit bug 324094. Until Playwright ships a WebKit with the fix (1.64, not released yet), IDBBatchAtomicVFS.test.js hangs intermittently on WebKit. I'd have TestContext.destroy() close the VFS before terminate(), which gave 0 hangs in 20 runs. It's also arguably what a test should do anyway. Or would you rather wait for 1.64 and keep the harness as it is?

7. One PR or two. I'd send a single PR, with the test and harness changes in their own commits ahead of the config and workflow changes. Would you prefer them split?

[rhashimoto]

1. Which Chromium.
I would use the Chromium bundled with Playwright. Chrome 129 was just the current version at the time.

2. Restricting to one browser.
Your proposal is exactly what I had in mind.

3. CI layout.
I think we can drop the pre-build tests.

4. Tests that cannot pass as written without readwrite-unsafe.
Your proposal works for me.

5. The harness turning a failure into a hang.
Your proposal works for me. I prefer a single PR.

6. WebKit bug 324094.
You're right, the VFS should be closed.

7. One PR or two.
I agree, one PR.

@lalexdotcom I really appreciate all the investigations, debugging, and improvements you have contributed to the project! Your persistent and professional efforts in identifying failures, performance, and testing are amazingly valuable. Thank you!

[lalexdotcom]

All clear, I'll start on it and send you the PR soon: Playwright's bundled Chromium, Firefox and WebKit, WTR_BROWSERS to pick a subset, and one CI job per browser running the suite after make. All in one PR, with the test and harness changes in their own commits ahead of the config and workflow.

And thank you for the kind words! It's a great project, and I'm proud to be part of it. It's a win-win: trying to get the most out of wa-sqlite for my own use is also a way to give something back, and I'm really glad about that. Blame my obsessive, stubborn streak, plus a good assistant (Claude) for the parts that are beyond me: together they let me get to the bottom of things, and wa-sqlite gets the benefit. 😇

[lalexdotcom]

A follow-up before I open the PR: while setting up the three browsers I measured a few more things on my fork's Actions, and they changed my proposal. I'd rather run it by you first.

Test what users actually run. The point of running the suite in browsers is to know that wa-sqlite works where it will be used, so the browsers should be the ones users run. Anything else is noise: a failure that no user will ever hit, or a pass that says nothing about a real browser. Playwright's Firefox and WebKit are patched builds, and its WebKit on Linux is not Safari.

The suite runs fine on stock browsers through WebDriver, with @web/test-runner-webdriver alongside the existing setup:

• Linux and Windows: stock Chrome and Firefox.
• macOS: Safari, through safaridriver (the macos-26 and xcode-27 images give Safari 26 and 27).

So I'd drop Playwright entirely, including your point 1 (its bundled Chromium).

During development, yarn test would run Chrome only by default, and WTR_BROWSERS would pick others: chrome,firefox on Linux or Windows, plus safari on macOS. No browser download is needed beyond what is installed.

In CI, one reusable workflow holds the steps, and each platform gets a small workflow of its own: Linux (Chrome, Firefox), Windows (Chrome, Firefox), macOS (Chrome, Firefox, Safari).

• Triggers are yours to choose per platform. For example: every platform on a release, Linux only on pull requests and pushes to master, and any of them by hand (workflow_dispatch).
• The WASM is built once with make, and every job tests that same build.
• Running time: most jobs take a few minutes; Firefox on Windows takes about 20.

Tests that fail because of the browser, and what I'd do:

1. A harness race, not a browser bug. test-worker.js hands the VFS a Comlink.proxy() of the output DataView, so the VFS's setInt32() travels on that proxy's port while the call's result returns on the main port. The HTML spec gives each port its own task source, so the two are not ordered. Safari 26 delivers the result first: 72 to 81 vfs_xOpen/vfs_xClose failures per run. The fix does what the harness already does for jRead: pass a local DataView and copy it back before replying. After the fix: 0 failures, 3 runs out of 3.
   • Where it goes: this could be a PR of its own. But only Safari exposes the race, and the current CI never would, so I'd ship it as its own commit in the test PR, where the Safari job shows it failing without the fix and passing with it.
2. Safari 26 kills the page in sql_0005 on IDBMirrorVFS.
   • What happens: IndexedDB requests stall for tens of seconds, WebKit's internal IPC queue grows to about 265,000 messages, and the page is killed at Safari's 8 GB memory limit.
   • What it is not: I ruled out the retry rate, the proxies and BroadcastChannel.
   • Where it passes: Safari 27, Chrome and Firefox.
   • Proposal: skip that one spec on Safari before 27, with pending() and the reason, so it shows in every run's output.
3. No readwrite-unsafe on Firefox and Safari. As agreed, vfs_read_freshness and sql_0005 on OPFSWriteAheadVFS are skipped where FileSystemSyncAccessHandle has no mode.
4. Windows tooling. WebdriverIO cannot download geckodriver under Yarn PnP on Windows (yarnpkg/berry#6580), so the job uses the runner's geckodriver, and the docs will say to install it for contributors who run the tests on Windows.

Everything else passes on every platform; each case above was measured, and I can share the runs.

The same design also leaves room for mobile browsers:

• iOS Safari. It is WebDriver through safaridriver as well, so an iOS job can follow whatever simulators GitHub's macOS images provide. Today it works with the iOS 26 simulator on macos-26 (Safari 26, with the same sql_0005 skip). The iOS 27 simulator on xcode-27 still refuses WebDriver sessions, so I'd add iOS once that settles, or now with iOS 26 if you'd like it.
• Chrome on Android. I haven't looked into it yet, but it should be possible: chromedriver can drive Chrome in an Android emulator, and Linux runners can run one.

Does this direction work for you?

[rhashimoto]

The point of running the suite in browsers is to know that wa-sqlite works where it will be used, so the browsers should be the ones users run. Anything else is noise: a failure that no user will ever hit, or a pass that says nothing about a real browser. Playwright's Firefox and WebKit are patched builds, and its WebKit on Linux is not Safari.

I'm not convinced that the Playwright browsers aren't sufficient for wa-sqlite unit tests. It doesn't seem that the behavior of storage and Web Locks and BroadcastChannel would be as affected as APIs for display and user events. What kinds of cases are there with significant differences between Chrome and Playwright Chromium, or Safari and Playwright WebKit?

The argument that Playwright is good enough wouldn't matter much if Playwright has no advantages over WebDriver. Are there any such reasons? My very limited understanding is that even the modern incarnation of WebDriver is still slower than Playwright. Is that true, and if yes, how much slower?

[lalexdotcom]

Fair questions. For Chromium I don't have a case: I haven't seen a difference between Playwright's Chromium and Chrome. Going through WebDriver there is about uniformity: one launcher and one configuration for every browser. Chrome and Firefox versions can be pinned there too; only Safari follows macOS.

For the other two, the differences I ran into are precisely in storage and workers:

• WebKit. Playwright's WebKit is built from a recent WebKit trunk, ahead of the Safari people actually run. The IndexedDB hang behind the destroy() change (WebKit bug 324094) is already fixed in the WebKit of Playwright's next release, while Safari 27.0.1 still has it, so the suite would pass there and hang on Safari. The other way round, the DataView race in the test harness shows only on Safari 26, never on Playwright's WebKit. And OPFS itself depends on Playwright: missing before 1.63, and still missing in the ephemeral contexts the launcher creates by default.
• Firefox. Playwright's Firefox is patched so that it can be driven, and that shows in timing: on the same page, awaiting a promise costs several times what it does on a stock Firefox. The suite leans heavily on exactly that.

On speed: I haven't timed the two launchers against each other. With web-test-runner the driver mostly opens pages, while the tests run inside them and report to the server, so the protocol itself should matter little beyond starting a session per file. In practice, the whole suite on stock Chrome through WebDriver takes two to three minutes on GitHub's Linux runners.

[rhashimoto]

With web-test-runner the driver mostly opens pages, while the tests run inside them and report to the server, so the protocol itself should matter little beyond starting a session per file.

Makes sense. Okay, let's see what a WebDriver PR looks like!

For CI triggers, I'm fine with just Linux for now.