You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Merged PRs
dolt-workbench
1021: Speed up SQL paging, schema and diff pages
The SQL editor re-ran the whole query for every 50-row page and read every row of the result each time. This keeps the query open as a server-side cursor and reads the next page from it, plus a few other slow spots.
Changes
Cursor paging for all six engines (queryFactory/cursor, plus a cursor source each for MySQL/Dolt, Postgres/Doltgres and SQLite/DoltLite). Page 0 opens a cursor; later offsets read the next rows from it. If there's no cursor at that offset, the resolver falls back to the old re-execute-and-slice path.
Dolt schema page: 3 batched information_schema queries instead of 3 per table. Dolt system tables, which information_schema leaves out, keep the per-table lookup.
Working-set diff highlighting uses DOLT_DIFF('HEAD', 'WORKING', t) instead of filtering dolt_diff_<t> by to_commit = 'WORKING', which walks the table's whole history. On tags and commits, which have no working set, it returns no rows as before. Also fixes the IN () error on empty tables.
Startup polls for the GraphQL server every 50ms instead of every 1s.
Scroll race: rows appended while a diff page was loading were dropped.
On a 739k-row DoltLite table (select * from records):
main
this PR
First page
5.4s
3ms
First page + 10 scrolled pages
57s
20ms
Diff stat / row diffs
2.2s / 5.6s
0.1s / 0.1s
Scrolling 2,000 rows of a 100k-row table: Dolt 15.3s → 0.36s, MySQL 3.7s → 0.14s, Postgres 5.7s → 0.07s, Doltgres 16.4s → 0.47s, SQLite 6.9s → 0.08s. Dolt schema page with 152 tables: 3.0s → 72ms.
Keeping cursors safe
Writes run once. A cursor hands a query back to the normal path only when its connection can't have written: a read-only connection (SQLite, DoltLite), a read-only session (Postgres, plain MySQL), or an error raised before the statement started (Doltgres parse errors). MySQL, Dolt and Doltgres statements finish before their first page, because they can still be writing while rows stream (MySQL CALL, Dolt and Doltgres INSERT ... RETURNING). Dolt ignores read-only sessions, and its read-only transactions still allow DDL and Dolt procedures, so it can't stream safely.
Bounded memory. Streaming cursors read one page ahead and drop served rows. Statements that must finish first are capped at 64MB: past that, a read-only cursor stops the statement and falls back, and any other cursor keeps only the first page while the statement finishes, serves it, and then lets later pages re-run.
No stuck locks. At most one cursor is open per connection. It closes when a new query starts, before any GraphQL mutation (CloseCursorsInterceptor), and after 60s idle. A first page that may have written finishes before its cursor closes.
Connections. Abandoned MySQL/Postgres connections are destroyed rather than pooled, so a late KILL QUERY or pg_cancel_backend can't hit the next user. Read-only sessions are reset before release.
Same results as main. SQLite/DoltLite skip the cursor while the main connection has an open transaction, temp tables or attached databases. DoltLite resolves non-branch refs to commits the same way the normal path does. Postgres keeps the database-name check. Warnings are sent once.
Testing
Server 273/273, web 320/320. New: cursor store unit tests, SQLite cursor integration tests, more DoltLite cursor tests.
Compared against main on live Dolt, MySQL 8.4, Postgres 17 and Doltgres servers and SQLite/DoltLite files: empty results, paging to the end, inserts/updates/RETURNING, DDL, errors, procedures (including MySQL procedures that read then write), sequences, multi-statement strings and wrong database names. Results match, apart from main reporting a next page after the last one.
ALTER TABLE with a cursor open on a 100k-row table completes in milliseconds on all four servers.
An independent review found nine issues, including two ways a write could run twice on Doltgres. All are fixed and re-verified with its reproduction scripts.
Known gaps
Doltgres reports the wrong row count for large INSERT ... RETURNING (e.g. "104 rows affected" for 1,000 rows). main shows the same number.
1017: Upgrade doltlite to 0.51.6
Bumps @dolthub/doltlite from ^0.11.51 to ^0.51.6. The caret on 0.x was keeping us on 0.11.x.
The new engine no longer reports no such branch or table: <ref> when checking out a tag or commit hash; it reports dolt does not support a detached head state. The detached fallback now checks dolt_branches instead of matching the error text.
Diff queries are roughly 3× faster on a large database (dolt_diff_stat / dolt_diff_<table> ~2.1s → ~0.65s on datalib-codex).
Server tests pass (240/240).
996: Group related Dependabot dependency updates
Group related Dependabot updates across the web app, GraphQL server, and Cypress packages so dependencies used together can be reviewed in a single PR.
995: Update mas build version to 0.3.106
Increment version
994: Update dmg build version to 0.3.106
Increment version
991: Update versions for release 0.3.76
Increment versions
990: Allow opening database files with any extension
The SQLite/DoltLite file picker currently restricts selection to .db files, preventing users from opening existing databases named with extensions such as .doltlite_db.
Remove the extension filter so users can select any file and attempt to open it as a database. Update the picker placeholder and its existing test to say “database file.” Database validity is still determined when connecting.
Validation: all 11 connection setup tests passed; Prettier passed; ESLint reported no errors (one existing React context warning). The native file dialog was not manually exercised.