Skip to content

0.3.77

Latest

Choose a tag to compare

@github-actions github-actions released this 06 Nov 19:45
62f4424

Merged PRs

dolt-workbench

  • 1021: Speed up SQL paging, schema and diff pages
    The SQL editor re-ran the whole query for every 50-row page and read every row of the result each time. This keeps the query open as a server-side cursor and reads the next page from it, plus a few other slow spots.

    Changes

    • Cursor paging for all six engines (queryFactory/cursor, plus a cursor source each for MySQL/Dolt, Postgres/Doltgres and SQLite/DoltLite). Page 0 opens a cursor; later offsets read the next rows from it. If there's no cursor at that offset, the resolver falls back to the old re-execute-and-slice path.
    • Dolt schema page: 3 batched information_schema queries instead of 3 per table. Dolt system tables, which information_schema leaves out, keep the per-table lookup.
    • Working-set diff highlighting uses DOLT_DIFF('HEAD', 'WORKING', t) instead of filtering dolt_diff_<t> by to_commit = 'WORKING', which walks the table's whole history. On tags and commits, which have no working set, it returns no rows as before. Also fixes the IN () error on empty tables.
    • Startup polls for the GraphQL server every 50ms instead of every 1s.
    • Scroll race: rows appended while a diff page was loading were dropped.
    • Grid: fewer re-renders (typing lag 416ms → 9ms).
    • doltlite 0.51.8, which includes lazy StatementSync.iterate() (dolthub/doltlite-node#12).

    Results

    On a 739k-row DoltLite table (select * from records):
    main this PR
    First page 5.4s 3ms
    First page + 10 scrolled pages 57s 20ms
    Diff stat / row diffs 2.2s / 5.6s 0.1s / 0.1s
    Scrolling 2,000 rows of a 100k-row table: Dolt 15.3s → 0.36s, MySQL 3.7s → 0.14s, Postgres 5.7s → 0.07s, Doltgres 16.4s → 0.47s, SQLite 6.9s → 0.08s. Dolt schema page with 152 tables: 3.0s → 72ms.

    Keeping cursors safe

    • Writes run once. A cursor hands a query back to the normal path only when its connection can't have written: a read-only connection (SQLite, DoltLite), a read-only session (Postgres, plain MySQL), or an error raised before the statement started (Doltgres parse errors). MySQL, Dolt and Doltgres statements finish before their first page, because they can still be writing while rows stream (MySQL CALL, Dolt and Doltgres INSERT ... RETURNING). Dolt ignores read-only sessions, and its read-only transactions still allow DDL and Dolt procedures, so it can't stream safely.
    • Bounded memory. Streaming cursors read one page ahead and drop served rows. Statements that must finish first are capped at 64MB: past that, a read-only cursor stops the statement and falls back, and any other cursor keeps only the first page while the statement finishes, serves it, and then lets later pages re-run.
    • No stuck locks. At most one cursor is open per connection. It closes when a new query starts, before any GraphQL mutation (CloseCursorsInterceptor), and after 60s idle. A first page that may have written finishes before its cursor closes.
    • Connections. Abandoned MySQL/Postgres connections are destroyed rather than pooled, so a late KILL QUERY or pg_cancel_backend can't hit the next user. Read-only sessions are reset before release.
    • Same results as main. SQLite/DoltLite skip the cursor while the main connection has an open transaction, temp tables or attached databases. DoltLite resolves non-branch refs to commits the same way the normal path does. Postgres keeps the database-name check. Warnings are sent once.

    Testing

    • Server 273/273, web 320/320. New: cursor store unit tests, SQLite cursor integration tests, more DoltLite cursor tests.
    • Compared against main on live Dolt, MySQL 8.4, Postgres 17 and Doltgres servers and SQLite/DoltLite files: empty results, paging to the end, inserts/updates/RETURNING, DDL, errors, procedures (including MySQL procedures that read then write), sequences, multi-statement strings and wrong database names. Results match, apart from main reporting a next page after the last one.
    • ALTER TABLE with a cursor open on a 100k-row table completes in milliseconds on all four servers.
    • An independent review found nine issues, including two ways a write could run twice on Doltgres. All are fixed and re-verified with its reproduction scripts.

    Known gaps

    • Doltgres reports the wrong row count for large INSERT ... RETURNING (e.g. "104 rows affected" for 1,000 rows). main shows the same number.
  • 1020: Use eric for update yarn PRs
  • 1017: Upgrade doltlite to 0.51.6
    Bumps @dolthub/doltlite from ^0.11.51 to ^0.51.6. The caret on 0.x was keeping us on 0.11.x.
    • The new engine no longer reports no such branch or table: <ref> when checking out a tag or commit hash; it reports dolt does not support a detached head state. The detached fallback now checks dolt_branches instead of matching the error text.
    • Diff queries are roughly 3× faster on a large database (dolt_diff_stat / dolt_diff_<table> ~2.1s → ~0.65s on datalib-codex).
      Server tests pass (240/240).
  • 996: Group related Dependabot dependency updates
    Group related Dependabot updates across the web app, GraphQL server, and Cypress packages so dependencies used together can be reviewed in a single PR.
  • 995: Update mas build version to 0.3.106
    Increment version
  • 994: Update dmg build version to 0.3.106
    Increment version
  • 991: Update versions for release 0.3.76
    Increment versions
  • 990: Allow opening database files with any extension
    The SQLite/DoltLite file picker currently restricts selection to .db files, preventing users from opening existing databases named with extensions such as .doltlite_db.
    Remove the extension filter so users can select any file and attempt to open it as a database. Update the picker placeholder and its existing test to say “database file.” Database validity is still determined when connecting.
    Validation: all 11 connection setup tests passed; Prettier passed; ESLint reported no errors (one existing React context warning). The native file dialog was not manually exercised.
  • 987: Update yarn

Closed Issues