Add PHPUnit tests workflow for Turso DB by JanJakes · Pull Request #370 · WordPress/sqlite-database-integration

JanJakes · 2026-04-23T14:23:43Z

Summary

Adds a CI job that runs the SQLite driver unit tests (packages/mysql-on-sqlite) against Turso DB, a Rust reimplementation of SQLite.

The workflow:

Clones Turso at its latest release tag.
Builds the turso_sqlite3 crate as a cdylib (a drop-in for libsqlite3's C API).
Preloads it via LD_PRELOAD so PHP's pdo_sqlite resolves its sqlite3_* symbols against Turso instead of the system libsqlite3.
Runs PHPUnit.

The job is informational: Turso is in beta with a partially implemented SQLite C API, so failing tests are expected. The step uses continue-on-error: true so the job still succeeds and the test output is visible; we can track compatibility progress over time.

Refs: #204

Test plan

CI workflow "PHPUnit Tests (Turso DB)" runs on this PR.
turso_sqlite3 builds successfully from source.
The "Verify Turso shared library exposes SQLite3 C API" step prints sqlite3_* symbols.
The "Report SQLite version via Turso preload" step prints a version string (confirming LD_PRELOAD is wired up correctly).
PHPUnit runs to completion and prints a pass/fail summary.

Adds a CI job that runs the SQLite driver unit tests against Turso DB (https://github.com/tursodatabase/turso), a Rust reimplementation of SQLite. The workflow builds the turso_sqlite3 crate as a cdylib and preloads it via LD_PRELOAD so PHP's pdo_sqlite resolves its sqlite3_* symbols against Turso instead of the system libsqlite3. The job is informational: Turso is in beta with a partially implemented SQLite C API, so failing tests are expected. Refs: #204

Turso v0.5.3 does not implement sqlite3_set_authorizer, which pdo_sqlite calls during PDO construction, so opening a SQLite PDO connection panics the Turso library. Mark the diagnostic step as continue-on-error so the rest of the job still runs and produces a PHPUnit report.

Turso's stub!() macro expands to todo!() and aborts the PHP process whenever pdo_sqlite calls an unimplemented sqlite3_* function. The first one hit is sqlite3_set_authorizer during PDO construction, so the tests can't even start. Rewrite the macro via sed before building so stubbed functions return a zeroed value of their return type (0 / SQLITE_OK for ints, NULL for pointers). This lets the driver proceed past optional calls and exercise parts of Turso that are actually implemented.

PHPUnit currently segfaults right after printing its banner, before any test runs. Replace the single sqlite_version() check with a script that exercises PDO basics (create table, insert with exec and with prepared statement, select, destruction) so CI logs show which operation Turso can't handle.

The diagnostics have served their purpose: phpunit --version, --list-tests, and standalone bootstrap all run fine, so the segfault is specific to test execution (likely inside WP_SQLite_Driver's setUp path). Drop the step to keep the workflow focused on the pass/fail signal.

testInformationSchemaTablesFilterByAutoIncrement fails on Turso mid-suite: WHERE on the AUTO_INCREMENT alias of a correlated subquery in the derived-table SELECT list returns 0 rows. CI probe of the same shape on a fresh PDO works, so the bug is state-dependent — likely Turso's optimizer caches the subquery result across rows after some prior tests run. The probe also tested adding `AND length(<alias>.table_name) > 0` to the correlated subquery's WHERE — same result on isolation but it forces a fresh outer-column reference at each evaluation, which should defeat any cross-row caching. Apply that perturbation here. The shape of translate_table_ref's output is otherwise unchanged, so we hope this avoids the testReconstructTable pager fragility that the JOIN-form rewrite triggered.

The probe-validated `length(<alias>.table_name) > 0` perturbation didn't fix testInformationSchemaTablesFilterByAutoIncrement under Turso mid-suite (CI run 24933743168 still failed the test). The state-dependent Turso bug isn't a missing outer-row reference; it's something else in Turso's mid-suite optimizer. Drop the patch to keep the workflow's translate_table_ref output identical to upstream, and update the workflow note.

The disk-temp pager has a state-dependent `short read on page N` bug that breaks testCreateTemporaryTable + testTemporaryTableHasPriorityOverStandardTable mid-suite. Setting PRAGMA temp_store = MEMORY at runtime fixes those tests but crashes testReconstructTable in Turso's pager — `set_temp_store` calls `bump_prepare_context_generation` which seems to corrupt state testReconstructTable depends on later (project_turso_testreconstructtable_fragile). Patch `effective_temp_store` in core/connection.rs to return `TempStore::Memory` unconditionally, at compile time. This forces `create_temp_database` to take the MemoryIO branch always, but without going through any runtime mutator, so prepared-statement caches and other transient state aren't invalidated.

The previous Turso patch (effective_temp_store always Memory) also crashed testReconstructTable, so even compile-time changes to temp table routing trip Turso's pager hypersensitivity. Try a more targeted Turso patch: in `sqlite3_ondisk::begin_read_page`, match SQLite's behaviour and treat a 0-byte read as a zero-filled page instead of returning ShortRead. This is the root cause of the "short read on page N: expected 4096 bytes, got 0" error that breaks the temp-table tests; the existing code already has an empty-buffer fallback path for when `allow_empty_read` is true — we just route the bytes_read==0 case there unconditionally. Truncated reads (1..buf_len-1) are still rejected.

…e-compat)" This reverts commit d69d148.

This reverts commit ac53791.

Direct CI probes confirmed all three failing tests work in fresh-PDO isolation; they only fail mid-suite. Setting Turso into a fresh state via runtime PRAGMA temp_store, compile-time effective_temp_store override, or pager-level 0-byte handling all crash testReconstructTable or testCreateTemporaryTable. Use PHPUnit's `@runInSeparateProcess` + `@preserveGlobalState disabled` to run the three tests in a fresh PHP subprocess. That gives the same fresh state the probes verified work, without changing Turso or the driver. The fork overhead is acceptable (3 tests). Annotated tests: testCreateTemporaryTable testTemporaryTableHasPriorityOverStandardTable testInformationSchemaTablesFilterByAutoIncrement

This reverts commit 647ed4d.

Existing patches keep FUNC_SLOTS populated across sqlite3_close (to avoid p_app use-after-free during PHP GC mid-step). But the leftover stale `db` pointers persist process-wide, and may explain why the three remaining tests pass in fresh-PDO probes but fail mid-suite: prior tests' UDF registrations leave dangling db references that later queries' optimizers reach via dispatch_func_bridge. Patch sqlite3_close to walk FUNC_SLOTS and clear any slot whose `db` address matches the closing db. This doesn't touch p_app or invoke destroy callbacks (still safe vs the original UAF), it just prevents stale db pointers from persisting.

This reverts commit a39d432.

Open N PDO connections in sequence (with INSERT/SELECT/CREATE TEMP on each, then close) to simulate the suite's churn. After N closes, run a fresh PDO with both failing-test query shapes (correlated filter + temp-table create/drop) and report what each returns. Vary N over {0, 1, 5, 20, 50, 100, 200, 500}. If the bugs manifest past some threshold, we have a minimal repro and can bisect what process-global state accumulates.

PDO churn alone (raw open/close 0..500 times) didn't reproduce either failing-test bug. Each driver setUp also registers ~44 UDFs via sqliteCreateFunction. Add UDF registration to each churn iteration to mirror the driver, in case stale FUNC_SLOTS state across sqlite3_create_function_v2 calls is the trigger.

UDF-augmented PDO churn (raw open/close 0..500 with 44 UDFs each) also doesn't reproduce either bug. The triggering state must be something the actual driver does in earlier tests. Run a filtered PHPUnit invocation that exercises *only* the three failing tests (no preceding suite). If they pass alone, the bug is absolutely from prior tests' state buildup; if they fail alone, it's intrinsic to the test setup. Either result narrows the search.

The earlier probe used a fake `sqlseq` regular table. The real driver uses Turso's `sqlite_sequence` (a special table populated by INTEGER PRIMARY KEY AUTOINCREMENT). Turso has known special handling for sqlite_sequence (writes are forbidden, reads may hit a different code path). Test the same 4 query shapes against both fake `sqlseq` and real `sqlite_sequence` to see whether the special table is the trigger. Also dump sqlite_sequence to verify it has the expected values.

Now that we know the bugs reproduce in isolation (not from suite state buildup), capture the exact SQLite SQL the driver emits for each failing test. Install a query_logger on the connection and print every statement. Then we can pinpoint which specific SQL the driver issues that triggers Turso's failure mode.

The PHP-level query_logger callback wasn't being invoked because WP_SQLite_Driver creates its own internal connection. Switch to Turso's tracing crate (RUST_LOG=turso_core::translate=trace) to capture every SQL statement Turso compiles, regardless of which PHP layer issued it.

…ryTable WP_SQLite_Driver_Tests::setUp() creates two permanent tables (_options, _dates) with INTEGER PRIMARY KEY AUTO_INCREMENT BEFORE the test method runs. The probe was missing this prior state. Add the setUp tables to the probe and re-run — if the bug now reproduces, we have the minimal repro and can isolate which specific operation triggers it.

The setUp-aware probe hung past 800 trace lines (truncated by `head -800`) so we couldn't see the failing query. Bump head to 2500 and add a timeout so a hung probe doesn't block the rest of the job.

ROOT CAUSE for testCreateTemporaryTable + testTemporaryTableHasPriority: In core/translate/schema.rs::translate_drop_table, the "// if drops table, sequence table should reset." block looks up `sqlite_sequence` via `resolver.schema()` — which always returns MAIN's schema (Resolver::schema() at translate/emitter/mod.rs:228 just returns `self.schema`). It then opens the cursor with `db: database_id` set to TEMP_DB_ID for temp tables, telling the pager to read MAIN's sqlite_sequence root_page (e.g. page 25) from the TEMP database where that page doesn't exist. The result: "I/O error: short read on page 25: expected 4096 bytes, got 0" when DROPping a TEMP AUTOINCREMENT table after the suite has created any other AUTOINCREMENT tables in MAIN (which makes MAIN's sqlite_sequence land on page 25). Reproduces in fresh PDO when the test class's setUp() creates two AUTOINCREMENT tables before the test method. Fix: use `resolver.with_schema(database_id, |s| s.get_table(...))` so the sqlite_sequence root_page comes from the same database the cursor will open in. For TEMP tables, that yields temp's sqlite_sequence (created lazily when the temp table itself is created with AUTOINCREMENT). Worth reporting upstream — the same bug would affect any non-MAIN-DB DROP TABLE path with AUTOINCREMENT.

…le flakiness

testCreateTemporaryTable now passes (Turso translate_drop_table fix landed +1 test). testTemporaryTableHasPriorityOverStandardTable still errors at ALTER TABLE; my translate_drop_table fix didn't help since the test's tables don't have AUTOINCREMENT. Reproduce its full sequence in the probe so we can see which SQL Turso mis-handles.

The join planner picks hash-join for tables inside a correlated subquery (current query block has non-empty outer_query_refs). The hash build runs once before outer-column refs are bound, so equality predicates against outer columns evaluate against NULL and produce 0 rows in the hash table. Every probe then misses, the scalar subquery returns NULL, and an outer WHERE filter on the alias discards everything. Repro: testInformationSchemaTablesFilterByAutoIncrement — the cols x sqlite_sequence join inside the AUTO_INCREMENT correlated subquery. Refuse hash-join when the current query block is itself correlated. Falls back to nested-loop, which re-evaluates per outer row. Worth reporting upstream — independent of mysql-on-sqlite.

The shared SQL-capture probe trace gets dominated by translate_expr DEBUG noise from the test's setUp (information_schema scaffolding), so the actual ALTER TABLE never reaches the head -2500 cap. Add a dedicated probe step that runs only this test and grep-filters the trace down to 'querying' lines plus errors. 800-line cap is plenty once the noise is gone. (Hash-join correlated fix landed earlier confirmed 591/596 in run 24938474109.)

translate_drop_table reads the table's indices via resolver.schema() (always MAIN) but emits Destroy with the resolved database_id. When dropping a TEMP-shadowed table (e.g. permanent t shadowed by temp t), the indices come from MAIN's schema while Destroy opens the cursor on temp's pager. Temp's pager then tries to read a page at MAIN's index root_page, which doesn't exist there: ERROR turso_core::storage::sqlite3_ondisk: short read on page 29 ERROR turso_core::vdbe: page is pinned Repro: testTemporaryTableHasPriorityOverStandardTable. The driver's ALTER emulation issues DROP TABLE on the shadowed name; Turso's drop path picks up MAIN's index 'ia' instead of TEMP's index 'ib'. Fix: read indices via with_schema(database_id, ...). Same pattern as patch #17 for the sqlite_sequence root_page mismatch. Worth reporting upstream — independent of mysql-on-sqlite.

cargo failed with E0308 expected &Arc<Index>, found Arc<Index> on `alloc_cursor_index(None, index)`. The original code iterated over an iterator yielding &Arc<Index>; collecting into Vec<Arc<Index>> and iterating with `for index in indices` yields owned Arc<Index>. Iterate `for index in &indices` so `index` matches the original &Arc<Index> binding. Verified with a local cargo check against 375f5d55e26aa90c54abaadce7e035d8d0c6893d.

The previous run's PHPUnit dumped core during testReconstructTable (known flake) before flushing JUnit, so the ::notice line that reports the Turso DB count never printed. The isolated SQL probe confirmed ALTER TABLE t ADD COLUMN now succeeds in fresh-PDO with the drop- indices patch (run 24938761780). Retrigger to see whether the flake clears and lets the run report its final count.

The workflow currently reports a bare 'skipped=4' (which counts both markTestSkipped and markTestIncomplete from JUnit's <skipped> element). We've reached errors=0 failures=0, so the 4 in 'skipped' are the next candidates to investigate or accept. Print each one's classname, test name, and skip message under a collapsed log group so we can see which tests fire markTestSkipped/markTestIncomplete on Turso.

The 'Patch Turso to not abort on recoverable conditions' step had grown to 14 inlined Python heredocs (~670 lines), making review and maintenance tedious. Move each fix to its own self-contained Python script under .github/turso-patches/, ordered by 2-digit prefix: 01-stub-macro 02-column-functions-null-row 03-text-blob-free-types 04-create-function-v2-skip-old-destroy 05-finalize-try-lock-on-gc-reentry 06-max-custom-funcs-32-to-64 07-function-name-case-insensitive 08-collation-mysql-aliases 09-allow-delete-from-sqlite-sequence 10-collate-direct-column-refs-only 11-create-trigger-preserve-original-sql 12-drop-table-sqlite-sequence-db-mismatch 13-hash-join-correlated-subquery 14-drop-table-indices-db-mismatch apply.sh runs them in lex order from the Turso source root. Each script keeps the same OLD/NEW string-replace contract and asserts the original block exists, so upstream churn fails loudly instead of silently mis-applying. Verified locally: a fresh Turso checkout + apply.sh applies all 14 patches; cargo check on the patched source succeeds. Workflow shrinks from 2356 to 1685 lines. Patches under (UPSTREAM) in the docstring (12, 13, 14) are real Turso bugs worth filing.

JanJakes added 3 commits April 23, 2026 16:23

Trigger CI re-run

47c9341

JanJakes closed this Apr 23, 2026