Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ As of May 2026, SQLRite has:
- Full-text search + hybrid retrieval (Phase 8 complete): FTS5-style inverted index with BM25 ranking + `fts_match` / `bm25_score` scalar functions + `try_fts_probe` optimizer hook + on-disk persistence with on-demand v4 → v5 file-format bump (8a-8c), a worked hybrid-retrieval example combining BM25 with vector cosine via raw arithmetic (8d), and a `bm25_search` MCP tool symmetric with `vector_search` (8e). See [`docs/fts.md`](fts.md).
- SQL surface + DX follow-ups (Phase 9 complete, v0.2.0 → v0.9.1): DDL completeness — `DEFAULT`, `DROP TABLE` / `DROP INDEX`, `ALTER TABLE` (9a); free-list + manual `VACUUM` (9b) + auto-VACUUM (9c); `IS NULL` / `IS NOT NULL` (9d); `GROUP BY` + aggregates + `DISTINCT` + `LIKE` + `IN` (9e); four flavors of `JOIN` — INNER, LEFT, RIGHT, FULL OUTER (9f); prepared statements + `?` parameter binding with a per-connection LRU plan cache (9g); HNSW probe widened to cosine + dot via `WITH (metric = …)` (9h); `PRAGMA` dispatcher with the `auto_vacuum` knob (9i)
- Benchmarks against SQLite + DuckDB (Phase 10 complete, SQLR-4 / SQLR-16): twelve-workload bench harness with a pluggable `Driver` trait, criterion-driven, pinned-host runs published. See [`docs/benchmarks.md`](benchmarks.md).
- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.6: shipped.** `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, and `ConcurrentTx`. WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes go through commit-time validation and abort with `SQLRiteError::Busy` on row-level write-write conflict. Reads via `Statement::query` see the BEGIN-time snapshot. Per-commit GC + `Connection::vacuum_mvcc()` bound the in-memory version chain growth. **11.7 SDK propagation: shipped on this branch.** The C FFI gains `SqlriteStatus::Busy` / `BusySnapshot`. Python adds `sqlrite.BusyError` / `BusySnapshotError` subclasses. Node exports `errorKind(message)` + `ErrorKind` enum. Go adds `ErrBusy` / `ErrBusySnapshot` sentinels (matchable with `errors.Is`) + an `IsRetryable(err)` helper. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.7: shipped.** Engine + SDK error propagation: `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, `ConcurrentTx`. WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes commit-validate against `MvStore` and abort with `SQLRiteError::Busy`. Reads via `Statement::query` see the BEGIN-time snapshot. Per-commit GC + `vacuum_mvcc()` bound the version chain growth. C FFI / Python / Node / Go all propagate `Busy` / `BusySnapshot` as typed retryable errors. **11.8 multi-handle SDK shape: shipped on this branch.** The FFI's `sqlrite_connect_sibling`, Python's `Connection.connect()`, and Node's `db.connect()` mint sibling handles that share backing state — closes the end-to-end gap from 11.7 where `BusyError` was reachable but not exerciseable through any SDK. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
- A fully-automated release pipeline that ships every product to its registry on every release with one human action — Rust engine + `sqlrite-ask` + `sqlrite-mcp` to crates.io, Python wheels to PyPI (`sqlrite`), Node.js + WASM to npm (`@joaoh82/sqlrite` + `@joaoh82/sqlrite-wasm`), Go module via `sdk/go/v*` git tag, plus C FFI tarballs, MCP binary tarballs, and unsigned desktop installers as GitHub Release assets (Phase 6 complete)

See the [Roadmap](roadmap.md) for the full phase plan.
Expand Down
39 changes: 23 additions & 16 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -622,7 +622,7 @@ The headline slice. Multiple sibling `Connection`s can each hold their own open
**Known limitations carried forward (most resolved in 11.5):**

- ~~Reads via `Statement::query` / `Statement::query_with_params` bypass the swap.~~ ✅ Fixed in 11.5 — `Connection.concurrent_tx` is now `Mutex<Option<…>>` and a new `with_snapshot_read` helper threads the swap through `&self`.
- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.8 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror. (Durability already works through the legacy mirror in v0; the WAL log-record format is foundation work for cross-process MVCC.)
- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.9 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror. (Durability already works through the legacy mirror in v0; the WAL log-record format is foundation work for cross-process MVCC.)
- `AUTOINCREMENT` inside `BEGIN CONCURRENT` isn't explicitly rejected; the v0 deep-clone-snapshot model handles concurrent INSERTs by isolating each tx's `last_rowid` bumps to its private snapshot, so two concurrent INSERTs on an `AUTOINCREMENT` column may collide at COMMIT and surface as `Busy`. Adopting the plan's "reject AUTOINCREMENT under MVCC" gate is a clean follow-up.
- Tables touched by `BEGIN CONCURRENT` writes can't carry FTS or HNSW indexes today — `restore_row` only maintains B-tree secondary indexes. Concurrent-tx tests don't exercise FTS / HNSW, but a runtime guard would surface this with a clear error rather than producing inconsistent indexes.

Expand All @@ -647,34 +647,41 @@ Bounds in-memory growth of the [`MvStore`](../src/mvcc/store.rs) version chains.
**What 11.6 doesn't yet do:**

- No background GC thread or `PRAGMA mvcc_gc_interval_ms`. Per-commit sweep + explicit `vacuum_mvcc()` cover the v0 model; the periodic-sweep variant lands as a follow-up if profiles show it's needed.
- GC sweeps don't trigger `Mvcc → Wal` journal-mode downgrades. The `set_journal_mode` setter still rejects the transition while the store carries committed versions; promoting that path requires the checkpoint-integration story from 11.8.
- GC sweeps don't trigger `Mvcc → Wal` journal-mode downgrades. The `set_journal_mode` setter still rejects the transition while the store carries committed versions; promoting that path requires the checkpoint-integration story from 11.9.

### 🚧 Phase 11.7 — SDK propagation of `Busy` / `BusySnapshot` *(in progress, plan-doc "Phase 10.8"; promoted ahead of plan-doc 11.5 checkpoint work for the same reason 11.5 / 11.6 jumped the queue — surfacing retryable errors to SDK callers is what unblocks Python / Node / Go users from actually writing `BEGIN CONCURRENT` retry loops)*
### Phase 11.7 — SDK propagation of `Busy` / `BusySnapshot` *(plan-doc "Phase 10.8"'s first half; promoted ahead of plan-doc 11.5 checkpoint work because surfacing retryable errors to SDK callers is what unblocks Python / Node / Go users from writing `BEGIN CONCURRENT` retry loops)*

- **C FFI** ([`sqlrite-ffi/src/lib.rs`](../sqlrite-ffi/src/lib.rs)): new `SqlriteStatus::Busy = 5` and `SqlriteStatus::BusySnapshot = 6` codes alongside the existing `Ok` / `Error` / `InvalidArgument` set. `SqlriteStatus::is_retryable()` covers both. A new internal `status_of_sqlrite` mapper inspects the engine's `SQLRiteError` variant and routes `Busy` / `BusySnapshot` to the dedicated codes (the generic `status_of` keeps mapping every error to `Error`). `sqlrite_execute` switches to the engine-aware mapper so `BEGIN CONCURRENT` commits surface the dedicated codes through every language binding. Header regenerated automatically via `build.rs`.
- **Python SDK** ([`sdk/python/src/lib.rs`](../sdk/python/src/lib.rs)): two new exception classes `sqlrite.BusyError` and `sqlrite.BusySnapshotError`, both inheriting from `sqlrite.SQLRiteError`. Existing `except sqlrite.SQLRiteError` blocks keep catching them; retry helpers can branch with `except sqlrite.BusyError`. A new `map_engine_err` helper inspects the engine error variant and raises the matching exception class. Every engine-typed call site (open / execute / prepare / query / rows.next) routes through it.
- **Node.js SDK** ([`sdk/nodejs/src/lib.rs`](../sdk/nodejs/src/lib.rs)): new exported `ErrorKind` string enum (`'Busy'`, `'BusySnapshot'`, `'Other'`) and `errorKind(message: string)` classifier function. The engine's `thiserror` Display already prefixes retryable errors with `'Busy: '` / `'BusySnapshot: '`, so the classifier just regex-tests the prefix. JS callers wrap their `BEGIN CONCURRENT` loops in `try / catch (err) { if (errorKind(err.message) === ErrorKind.Busy) continue; }`.
- **Go SDK** ([`sdk/go/sqlrite.go`](../sdk/go/sqlrite.go)): two new sentinel error values `sqlrite.ErrBusy` and `sqlrite.ErrBusySnapshot`, plus an `IsRetryable(err error) bool` helper. `wrapErr` recognises the new FFI status codes and wraps the engine message with `fmt.Errorf("…: %w", ErrBusy)` so `errors.Is(err, sqlrite.ErrBusy)` works through the `database/sql` driver chain.
- **WASM SDK** — deliberately untouched. The browser WASM target is single-threaded; `BEGIN CONCURRENT` is meaningful but multi-handle concurrency through `Connection::connect` isn't yet exposed across `wasm-bindgen`'s lifetime model. When the multi-handle JS shape lands (separate slice), the same `Busy: …` message prefix will be the classifier hook for the WASM bindings too.
- **C FFI** ([`sqlrite-ffi/src/lib.rs`](../sqlrite-ffi/src/lib.rs)): new `SqlriteStatus::Busy = 5` and `SqlriteStatus::BusySnapshot = 6` codes; `SqlriteStatus::is_retryable()` covers both. A new internal `status_of_sqlrite` mapper inspects the engine's `SQLRiteError` variant and routes `Busy` / `BusySnapshot` to the dedicated codes.
- **Python SDK**: two new exception classes `sqlrite.BusyError` and `sqlrite.BusySnapshotError`, both inheriting from `sqlrite.SQLRiteError`. `map_engine_err` helper raises the matching subclass.
- **Node.js SDK**: exported `ErrorKind` string enum (`'Busy'`, `'BusySnapshot'`, `'Other'`) and `errorKind(message: string)` classifier function. The engine's `thiserror` Display prefixes retryable errors with `'Busy: '` / `'BusySnapshot: '` so the classifier just regex-tests the prefix.
- **Go SDK**: two new sentinel error values `sqlrite.ErrBusy` / `sqlrite.ErrBusySnapshot`, plus an `IsRetryable(err error) bool` helper. `wrapErr` recognises the new FFI status codes and wraps the engine message with `fmt.Errorf("…: %w", ErrBusy)`.
- **WASM SDK** — deliberately untouched (browser is single-threaded; multi-handle shape not yet exposed).

**What this slice doesn't do:**
### 🚧 Phase 11.8 — Multi-handle SDK shape *(in progress, was plan-doc 11.8's other half; promoted ahead of plan-doc 11.5 again because the 11.7 retry-error machinery can't be exercised end-to-end through any SDK until siblings are reachable)*

- The multi-handle / sibling-`Connection` shape isn't exposed through any SDK yet. Each `sqlrite.connect(path)` / `new Database(path)` / `sql.Open(...)` builds an independent backing database. End-to-end testing of cross-handle `Busy` is therefore deferred to the multi-handle SDK slice; this PR ships the *plumbing* so once that wiring lands, callers already have the retry idioms they need.
- The WAL log-record durability work (plan-doc 11.5 / our 11.8) stays deferred.
Each pre-11.8 SDK `connect()` / `new Database()` built an *isolated* backing DB; the 11.7 `BusyError` / `errorKind` / `ErrBusy` plumbing was reachable but not actually triggerable from user code. This slice exposes the engine's `Connection::connect()` through every reachable language so apps can mint sibling handles that share state, and finally exercise the 11.7 retry idioms with real cross-handle conflicts.

### Phase 11.8 — Checkpoint integration + crash recovery *(planned, plan-doc "Phase 10.5"; renumbered to follow GC + SDK propagation because durability via the legacy `save_database` mirror already works in v0; this slice is foundation work for cross-process MVCC and column-level WAL deltas)*
- **C FFI** ([`sqlrite-ffi/src/lib.rs`](../sqlrite-ffi/src/lib.rs)): new `sqlrite_connect_sibling(existing, out)` function. Wraps the engine's `Connection::connect`. Callers get a sibling handle with its own `SqlriteConnection` pointer but shared backing database; the sibling must be closed via `sqlrite_close` (its lifecycle is independent — closing one handle doesn't tear down the others while a sibling is still alive).
- **Python SDK** ([`sdk/python/src/lib.rs`](../sdk/python/src/lib.rs)): new `Connection.connect()` instance method that mints a sibling pyclass. Wraps the engine's `Connection::connect` inside the existing `Mutex<RustConnection>`. The new handle inherits the parent's `ask_config`.
- **Node.js SDK** ([`sdk/nodejs/src/lib.rs`](../sdk/nodejs/src/lib.rs)): new `db.connect()` method on the `Database` class. Same shape — sibling shares state, can hold its own `BEGIN CONCURRENT`.
- **Go SDK** — deliberately not changed. Go's `database/sql` already gives callers a connection pool over a single `sql.Open`; each pool connection acquired through `db.Conn(ctx)` is *already* a sibling of the rest at the driver layer. But each `sql.Open("sqlrite", path)` still builds an independent backing DB because the pool is per-`sql.DB`. Exposing a cross-pool sibling shape through the `database/sql` driver model is genuinely non-obvious (it'd require a process-level registry keyed by path); deferred to the multi-handle Go follow-up.
- **WASM SDK** — still untouched. The browser is single-threaded and `wasm-bindgen` lifetimes complicate sibling pyclass-style sharing. Same deferral as 11.7.

Each SDK gets end-to-end tests that exercise `BEGIN CONCURRENT` cross-handle conflicts: two sibling handles, two concurrent transactions on the same row, the second commit hits the SDK's typed retryable error, retry succeeds.

### Phase 11.9 — Checkpoint integration + crash recovery *(planned, plan-doc "Phase 10.5"; renumbered to follow SDK propagation because durability via the legacy `save_database` mirror already works in v0; this slice is foundation work for cross-process MVCC and column-level WAL deltas)*

MVCC log-record WAL frame format (the deferred 11.4 piece). Commit appends log records pre-`save_database`. Reopen replays log records into `MvStore`. Checkpoint drains `MvStore` versions back into the pager (so `Mvcc → Wal` becomes legal once the store is empty). Crash-recovery test: kill mid-commit between log-record append and version-chain push; reopen; verify the committed transaction is visible and the half-written one is not.

### Phase 11.9 — Indexes under MVCC *(deferred-by-design, plan-doc "Phase 10.7")*
### Phase 11.10 — Indexes under MVCC *(deferred-by-design, plan-doc "Phase 10.7")*

Each secondary-index entry becomes its own `RowVersion`. Turso explicitly punted on this; SQLRite's v0 will reject `CREATE INDEX` while `journal_mode = mvcc`.

### Phase 11.10Multi-handle SDK shape + REPL `.spawn` *(planned, was plan-doc 11.8's other half)*
### Phase 11.11 — REPL `.spawn` + bench workload *(planned)*

Expose `Connection::connect()` through the FFI + each SDK so Python / Node / Go callers can mint sibling handles, plus a new REPL `.spawn` meta-command. Without this, the 11.7 retry-error machinery can't actually be exercised end-to-end through an SDK (each SDK `connect()` builds an independent DB). Also adds the "N concurrent writers" benchmark workload.
REPL `.spawn` meta-command for interactive `BEGIN CONCURRENT` demos. New "N concurrent writers" benchmark workload pitting SQLRite-MVCC against SQLite + DuckDB on disjoint-row write throughput. Plus Go SDK multi-handle work (cross-pool sibling shape).

### Phase 11.11 — Docs *(planned, plan-doc "Phase 10.9")*
### Phase 11.12 — Docs *(planned, plan-doc "Phase 10.9")*

Promote the plan to `docs/concurrent-writes.md` and update the cross-references.

Expand Down
34 changes: 34 additions & 0 deletions sdk/nodejs/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,40 @@ impl Database {
})
}

/// Phase 11.8 — mints a sibling `Database` that shares the
/// same underlying state (the in-memory tables, the MVCC
/// store, the pager). Wraps the engine's `Connection::connect`.
///
/// Use this to drive `BEGIN CONCURRENT` from multiple Node
/// handles in the same process: each sibling can hold its own
/// concurrent transaction, and commits validate against the
/// shared MvStore.
///
/// ```js
/// const db = new Database(':memory:');
/// db.exec('PRAGMA journal_mode = mvcc');
/// db.exec('CREATE TABLE t (id INTEGER PRIMARY KEY, v INTEGER)');
/// const sibling = db.connect();
/// // sibling.exec(...) lands on the same backing tables.
/// ```
///
/// The sibling carries its own per-handle prepared-statement
/// cache and concurrent-transaction slot. Closing one handle
/// (`db.close()`) doesn't affect siblings; the underlying
/// database lives as long as any handle holds it.
#[napi]
pub fn connect(&self) -> Result<Database> {
let borrow = self.inner.borrow();
let parent = borrow
.as_ref()
.ok_or_else(|| napi::Error::from_reason("cannot connect: database is closed"))?;
let sibling = parent.connect();
Ok(Database {
inner: RefCell::new(Some(sibling)),
ask_config: RefCell::new(self.ask_config.borrow().clone()),
})
}

/// Closes the connection and releases the OS file lock. Safe to
/// call multiple times.
#[napi]
Expand Down
Loading
Loading