Conversation
…lite Builds Doltlite from source, replaces the system libsqlite3 with it so pdo_sqlite resolves against the Doltlite library, and runs the mysql-on-sqlite PHPUnit suite without any filters. The libsqlite3 swap is done at /usr/local/lib and in the multiarch dir — LD_PRELOAD is unreliable because pdo_sqlite's DT_NEEDED binds the soname at link time. This commit is the scaffolding only. Doltlite has several deviations from stock SQLite that break the driver; the compatibility patches land in follow-up commits, each with its own verify assertion.
Doltlite silently sets TF_WithoutRowid on every table declared with a primary key, so `SELECT rowid FROM t` fails with "no such column: rowid". The mysql-on-sqlite driver uses `ORDER BY ROWID` as a stable tie-break when reading its internal information_schema tables, so dropping rowid breaks info-schema lookups across the board. Patch by sed: replace the single `tabOpts |= TF_WithoutRowid;` line in src/build.c with a no-op, leaving table creation otherwise untouched. Verified with a PHP one-liner that a composite-PK table still returns a rowid.
prollyBtCursorIndexMoveto's tree-scan loop sets pIdxKey->eqSeen when it finds an exact prefix match, and OP_SeekGE with BTREE_SEEK_EQ relies on that flag to decide whether the seek landed on an exact row. The subsequent findMatchingMutMapEntry call internally runs sqlite3VdbeRecordCompare against each candidate mutmap entry, and each probe resets eqSeen to 0 before the comparison — so after the lookup the flag is whatever the last mutmap probe left behind, not what the tree scan saw. The observable symptom: two sequential DELETEs on a composite NOCASE PK inside a SAVEPOINT. The first DELETE seeds the mutmap; the second goes through prollyBtCursorIndexMoveto, finds the matching tree row, sets eqSeen=1, then loses that flag when checking the mutmap. OP_SeekGE reads eqSeen=0, bails to seek_not_found, and the second DELETE silently does nothing. Patch saves eqSeen before the mutmap lookup and restores it when the lookup returns no override — checked-in as a .patch file and applied before build.
prollyBtCursorIndexMoveto can reposition the merge cursor directly at a mutmap INSERT entry via setCursorToMutMapEntryPhys, which jumps mmIdx past any mutmap entries that sort earlier. A subsequent mergeScan iteration that sees `cmp*dir < 0` — the tree entry is ahead of mutmap[mmIdx] — then emits the tree row without noticing that the mutmap holds a DELETE for that tree key at an earlier order index. The tree row is logically deleted but the scan walks over it anyway; SQLite eventually follows its rowid and hits the deleted slot, tripping the "database disk image is malformed" check in sqlite3VdbeFinishMoveto. Reproduces on an INSERT + DELETE + UPDATE sequence inside one transaction: the INSERT fills the mutmap, the DELETE adds a tombstone for an earlier row, the UPDATE's scan emits that earlier row from the tree, then crashes. Patch consults the mutmap before emitting a tree entry; if a DELETE exists for that key, skip and advance the tree cursor. No behavior change in the common case (no matching mutmap entry), constant- factor overhead otherwise. Checked in as a .patch file.
…tries prollyBtCursorInsert stored no value for non-INTKEY inserts where every record field fits into the sort key (splitKey=0), relying on getCursorPayload to reconstruct the record from the sort key bytes on read. That round-trip is lossless for BINARY but the sort-key encoder in sortkey.c folds A-Z to a-z under NOCASE and strips trailing spaces under RTRIM, so when SQLite picks the auto-index as a covering index for a plain column read (e.g. `SELECT pkcol FROM t` on a composite-PK rowid table with a NOCASE pkcol), OP_Column emits the folded bytes instead of the original column value. Reproduces testAlterTableModifyColumnComplexChange — the driver rebuilds the table via CREATE/INSERT/RENAME and then reads back 'johnny' where it stored 'Johnny'. Patch: when any pKeyInfo collation is non-identity (NOCASE or RTRIM), preserve pPayload->pKey as the value alongside the sort key. getCursorPayload prefers the stored value over reconstruction, so reads return the original bytes. No behavior change for BINARY- only records.
JanJakes
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runs the mysql-on-sqlite PHPUnit suite against Doltlite, a SQLite fork whose storage engine is a content-addressed prolly tree. The suite passes with the same counts as on stock SQLite: 667 tests, ~1.4M assertions, 2 skipped, 2 incomplete.
The workflow builds Doltlite from source, swaps the system
libsqlite3for it so PHP'spdo_sqlitetransparently resolves against Doltlite, verifies the swap took effect, and then runs PHPUnit without any filters. The swap replaces theLD_PRELOADapproach we started with —LD_PRELOADloses topdo_sqlite'sDT_NEEDEDbinding onlibsqlite3.so.0.Doltlite has four deviations from stock SQLite that break the driver. Each one is addressed by a narrow source-level patch applied before build, paired with a one-shot verify assertion in the workflow that fails fast if the patch ever stops applying upstream:
Disable auto-conversion of tables with a primary key to
WITHOUT ROWID, because the driver leans onORDER BY ROWIDas a stable tie-break when reading its owninformation_schematables and that idiom disappears oncerowidgoes away.Preserve the
eqSeenflag across Doltlite's mutmap lookup, because the lookup was resetting it mid-call and causing the second same-tableDELETEinside aSAVEPOINTto silently do nothing.Skip tree entries that the mutmap has already marked for deletion, because otherwise an
INSERT + DELETE + UPDATEsequence inside one transaction trips"database disk image is malformed".Keep the original column bytes in index entries when a column uses
NOCASEorRTRIMcollation, because Doltlite otherwise reconstructs rows from the sort key — which has folded'Johnny'down to'johnny'— whenever SQLite picks the index as a covering index.The patches are CI-only. Upstreaming them to Doltlite is a separate task.
Test plan
PHPUnit Tests (Doltlite)runs green on this PR.