Skip to content

feat(kernel): SQLite concurrency posture — WAL, busy_timeout, OperationalError → WorkspaceBusy (#80) - #95

Merged
JArmandoAnaya merged 1 commit into
mainfrom
feat/80-sqlite-concurrency
Jul 27, 2026
Merged

feat(kernel): SQLite concurrency posture — WAL, busy_timeout, OperationalError → WorkspaceBusy (#80)#95
JArmandoAnaya merged 1 commit into
mainfrom
feat/80-sqlite-concurrency

Conversation

@JArmandoAnaya

Copy link
Copy Markdown
Contributor

Closes #80.

First task of M3. No new capability — this closes the adapter's last untranslated exception before #28 runs ingest in a background task writing through the kernel while request handlers read and write the same file.

What lands

WAL journal mode, set in initialize() rather than on every connection. This is the one non-obvious decision here and the test suite is what forced it: switching a database to WAL writes its header, growing an empty file to a full page. WorkspaceService.open reads format_version before it has decided the file is a workspace, so a connect-time pragma left a 4 KB mark on any stranger's file merely inspected — breaking the invariant that open creates nothing when it refuses (test_open_never_creates_a_schema_in_a_stray_database caught it). initialize() is exactly where the caller has already established the file is ours to write to. WAL persists in the header, so setting it once is what it takes, and re-running it there is how a pre-WAL workspace converts on its next open.

busy_timeout, 5 s by default, on every connection including the one that runs migrations. Keyword-only with a default, so SqliteMetadataStore still satisfies MetadataStoreFactory = Callable[[Path], MetadataStore] and stays usable as a bare class reference; a caller wanting a different wait passes partial(SqliteMetadataStore, busy_timeout_ms=…).

One translation site. _translated is now the adapter's entire exception vocabulary and every entry point routes through it:

SQLAlchemy raises Becomes
IntegrityError ConstraintViolated
OperationalError with a SQLITE_BUSY / SQLITE_LOCKED result code WorkspaceBusy (new)
any other DatabaseError WorkspaceCorrupt

unit_of_work now catches DatabaseError rather than only IntegrityError — that was a real hole: a non-constraint failure raised inside a transaction escaped raw. Contention is told from damage by SQLite's result code (sqlite_errorname), not by message text, so a reworded SQLite release cannot silently reroute a lock into WorkspaceCorrupt.

WAL sidecars are part of the workspace layout. visionset.db-wal and -shm exist while a workspace is open; a clean close() checkpoints and removes them, but a killed process leaves them. _undo_init and both examples' _clear_previous_run now account for them, and docs/workspaces.md carries the warning it predicted for itself — copying only visionset.db from an open workspace loses committed data.

Two decisions worth stating

One new error, not two. Non-lock OperationalError (unable to open database file, disk I/O, disk full) falls through to WorkspaceCorrupt, whose docstring is widened to own it: "present but unusable, whatever the cause". A separate WorkspaceUnavailable would still have no caller — the same objection that kept WorkspaceBusy itself out until #28/#31 supplied one.

No BEGIN IMMEDIATE. docs/workspaces.md listed it as the next hardening, and it is now declined rather than pending, with the reason recorded: unit_of_work() serves reads and writes alike with no read-only variant, so an immediate transaction would take the write lock for every read and serialize exactly the concurrency WAL was adopted for. It stays off unless the unit of work grows a read-only form. An advisory lock file remains open.

Tests

New tests/kernel/test_concurrency.py — the suite's first threaded tests. Sequenced on threading.Event, never sleeps; every thread joined with a timeout and asserted dead. Ten tests: WAL on and persisted, WAL not applied by a bare format_version read, no sidecars after close, the busy timeout present on every connection and configurable, foreign keys surviving the listener rewrite, a reader proceeding while a write transaction is held open (the WAL payoff), and contention arriving as WorkspaceBusy with no SQLAlchemy in its module path. Plus a SQLITE_CANTOPEN case in test_metadata_store.py proving the other half of the OperationalError split.

Ran the concurrency file 20× consecutively: 0 failures.

Checks

ruff format --check, ruff check, mypy src, lint-imports (2 contracts kept) all clean. 896 tests pass, up from 885. Both examples run end to end. No openapi.json drift.

No migrationFORMAT_VERSION stays 10, VERSION stays 0.0.1.dev0, no new service / model / event / dependency.

…onalError → WorkspaceBusy (#80)

Closes the last untranslated adapter exception before background ingest makes
lock contention real.

- WAL journal mode, set in `initialize()` rather than on every connection:
  switching to WAL writes the file header, and a connect-time pragma would
  leave a 4 KB mark on any stranger's file merely inspected by
  `format_version` — breaking the invariant that `open` creates nothing when
  it refuses.
- `busy_timeout`, 5 s by default, on every connection. Keyword-only with a
  default so the class still satisfies `MetadataStoreFactory`.
- One translation site, `_translated`, that every entry point routes through:
  `IntegrityError` → `ConstraintViolated`, `SQLITE_BUSY`/`SQLITE_LOCKED` →
  `WorkspaceBusy`, everything else → `WorkspaceCorrupt`. `unit_of_work` now
  catches `DatabaseError` rather than only `IntegrityError`, closing a hole
  where a non-constraint failure inside a transaction escaped raw.
- Contention is told from damage by SQLite's result code, not by message text.
- WAL sidecars are part of the workspace layout: `_undo_init` and both
  examples' cleanup now account for them.

No migration: FORMAT_VERSION stays 10, VERSION stays 0.0.1.dev0, no
openapi.json drift, no dependency change.
@JArmandoAnaya
JArmandoAnaya merged commit 3c15be4 into main Jul 27, 2026
3 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the feat/80-sqlite-concurrency branch July 27, 2026 19:13
JArmandoAnaya added a commit that referenced this pull request Aug 21, 2026
…onalError → WorkspaceBusy (#80) (#95)

Closes the last untranslated adapter exception before background ingest makes
lock contention real.

- WAL journal mode, set in `initialize()` rather than on every connection:
  switching to WAL writes the file header, and a connect-time pragma would
  leave a 4 KB mark on any stranger's file merely inspected by
  `format_version` — breaking the invariant that `open` creates nothing when
  it refuses.
- `busy_timeout`, 5 s by default, on every connection. Keyword-only with a
  default so the class still satisfies `MetadataStoreFactory`.
- One translation site, `_translated`, that every entry point routes through:
  `IntegrityError` → `ConstraintViolated`, `SQLITE_BUSY`/`SQLITE_LOCKED` →
  `WorkspaceBusy`, everything else → `WorkspaceCorrupt`. `unit_of_work` now
  catches `DatabaseError` rather than only `IntegrityError`, closing a hole
  where a non-constraint failure inside a transaction escaped raw.
- Contention is told from damage by SQLite's result code, not by message text.
- WAL sidecars are part of the workspace layout: `_undo_init` and both
  examples' cleanup now account for them.

No migration: FORMAT_VERSION stays 10, VERSION stays 0.0.1.dev0, no
openapi.json drift, no dependency change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant