Skip to content

DB File Lock (rebase of #1050 by @grfwings)#1175

Merged
RhysSullivan merged 22 commits into
mainfrom
db-file-lock
Jun 28, 2026
Merged

DB File Lock (rebase of #1050 by @grfwings)#1175
RhysSullivan merged 22 commits into
mainfrom
db-file-lock

Conversation

@RhysSullivan

@RhysSullivan RhysSullivan commented Jun 28, 2026

Copy link
Copy Markdown
Collaborator

Rebase of #1050 ("DB File Lock") by @grfwings onto current main, opened as the root of a stack so the desktop recovery fix can stack on top.

All commits are authored by @grfwings. This branch only rebases that work onto main (it was 65 commits behind) and resolves conflicts in apps/local/src/serve.ts and apps/local/src/main.ts against main's newer per-resource MCP handler and auth gate, keeping #1050's ownership/dispose changes.

Credit: original work and design by @grfwings in #1050.

What it does

The DB file-lock / data-dir ownership layer:

  • apps/local/src/db/data-dir-ownership.ts, owned-database.ts
  • v1 to v2 migration hardening, crash-safe migrations, busy_timeout
  • CLI daemon ownership changes (apps/cli/src/daemon.ts, main.ts)

The desktop "attach instead of wedge" recovery fix (#1086) is stacked as a child PR on this branch.

Validation

  • typecheck 42/42
  • DB-ownership tests 21/21 (owned-database, data-dir-ownership, v1-v2-migration)
  • CLI daemon/manifest tests 18/18

Stack

  1. DB File Lock (rebase of #1050 by @grfwings) #1175 👈 current
  2. fix(desktop): attach to a daemon that owns the data dir instead of wedging #1086

Griffin Evans and others added 22 commits June 28, 2026 02:29
…ing eraly, derive expected counts from plan arrays
…cannot reopen while a previous handle is still disposing
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 28, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
executor-marketing 1109289 Commit Preview URL

Branch Preview URL
Jun 28 2026, 09:31 AM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 28, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
executor-cloud 1109289 Jun 28 2026, 09:31 AM

@github-actions

github-actions Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Cloudflare preview

Torn down — the PR is closed.

@pkg-pr-new

pkg-pr-new Bot commented Jun 28, 2026

Copy link
Copy Markdown

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@1175

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@1175

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@1175

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@1175

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@1175

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@1175

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@1175

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@1175

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@1175

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@1175

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@1175

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@1175

executor

npm i https://pkg.pr.new/executor@1175

commit: 1109289

@greptile-apps

greptile-apps Bot commented Jun 28, 2026

Copy link
Copy Markdown

Greptile Summary

This PR introduces a SQLite-based data-directory ownership lock (data.db.owner-lock) that replaces the previous file-hint-only concurrency model, ensuring at most one process can open data.db at any time. It also adds a crash-resumable v1-to-v2 data migration with a journal-based atomic flip.

  • data-dir-ownership.ts acquires a BEGIN EXCLUSIVE transaction in a separate rollback-journal-mode lock DB, released on process exit (including SIGKILL via OS file-lock teardown). owned-database.ts composes the lock with migration and DB open, with correct cleanup ordering (DB close before lock release).
  • v1-v2-migration.ts adds a staged copy/build/flip pipeline with a JSON journal that survives mid-migration crashes; recovery is idempotent and runs before any DB open.
  • CLI and Desktop layers are updated to propagate DataDirOwnershipHeld from the lock layer, enabling a second process to attach to the existing server rather than failing hard.

Confidence Score: 4/5

The ownership lock and migration journal logic are well-constructed and the test suite covers the core race (cross-process SIGKILL recovery). The two style fixes are trivial.

The ownership primitive, crash-recovery journal, and shutdown cleanup chains are all correctly structured. The only substantive finding is that resolveLegacyMigrationsFolder can create up to two leaked tmpdirs per migration run, which accumulates on Windows across reboots. The em-dash violations are mechanical style fixes against the AGENTS.md rule.

apps/local/src/db/v1-v2-migration.ts — the tmpdir materialization inside replayLegacyV1Migrations and tolerantReplayLegacyChain can be called twice in one migration without cleanup.

Important Files Changed

Filename Overview
apps/local/src/db/data-dir-ownership.ts New file: SQLite-based exclusive file-lock primitive using BEGIN EXCLUSIVE in rollback-journal mode, with correct busy_timeout=0, idempotent release guard, and complete isSqliteBusy detection.
apps/local/src/db/owned-database.ts New file: Composes ownership lock, v1-v2 migration, and DB open with correct cleanup ordering (db.close before lock release) and symlink-safe real path derivation.
apps/local/src/db/v1-v2-migration.ts New file: Crash-resumable v1-to-v2 migration with journal-based flip; resolveLegacyMigrationsFolder creates a tmpdir that is never cleaned up, and can be called twice in one migration run creating two leaked tmpdirs.
apps/local/src/db/data-migrations.ts Updated migration registry: adds LOCAL_V1_V2_LEDGER_NAME as first entry (no-op for fresh DBs, idempotent) before the existing auth-config and plugin migrations.
apps/local/src/executor.ts Updated to use openOwnedLocalDatabase; adds sharedHandle lifecycle sequencing and disposeExecutor so the ownership lock is released on shutdown.
apps/local/src/main.ts Belt-and-suspenders disposeExecutor call in disposeServerHandlers ensures DB ownership lock releases even if the Effect runtime finalizer fails.
apps/cli/src/daemon-state.ts New header comment block correctly documents the split between discovery-hint files and the DB-layer ownership lock; uses an em-dash in violation of AGENTS.md style rule.
apps/cli/src/local-server-manifest.ts Refactored manifest module: adds removeLocalServerManifest for supervised daemon stale-manifest cleanup; new header comment uses an em-dash violating AGENTS.md.
apps/desktop/src/main/sidecar.ts Adds cliManagedManifest flag so packaged binary's manifest is handled by the CLI layer; removes the old process-level startup lock comment, deferring to openOwnedLocalDatabase.
apps/local/src/index.ts Exports DataDirOwnershipHeld and findDataDirOwnershipHeld so CLI can detect ownership contention and attach to the existing server instead of failing.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant C as CLI/Desktop
    participant OD as openOwnedLocalDatabase
    participant DO as acquireDataDirOwnership
    participant M as migrateLocalV1ToV2IfNeeded
    participant DB as createSqliteFumaDb
    participant L as Lock DB (data.db.owner-lock)

    C->>OD: openOwnedLocalDatabase(dataDir)
    OD->>DO: acquireDataDirOwnership(dataDir)
    DO->>L: "createClient (journal_mode=DELETE, busy_timeout=0)"
    DO->>L: BEGIN EXCLUSIVE
    alt Lock already held by another process
        L-->>DO: SQLITE_BUSY
        DO-->>C: throw DataDirOwnershipHeld
        Note over C: CLI detects error, attaches to existing server
    else Lock acquired
        DO-->>OD: "{ lockPath, release }"
        OD->>M: migrateLocalV1ToV2IfNeeded(sqlitePath)
        M->>M: recoverV1V2Migration (journal check)
        alt v1 database found
            M->>M: copySqliteFileSet to staging
            M->>M: replay legacy migrations
            M->>M: insertPlan (BEGIN/COMMIT in staging)
            M->>M: verifyStagingDatabase
            M->>M: completeJournaledFlip (atomic rename)
        end
        M-->>OD: LocalV1V2MigrationResult
        OD->>DB: createSqliteFumaDb(sqlitePath)
        DB-->>OD: SqliteFumaDb
        OD-->>C: "OwnedLocalDatabase { db, close }"
        Note over L: Exclusive lock held for process lifetime
    end

    C->>OD: database.close()
    OD->>DB: db.close()
    OD->>DO: ownership.release()
    DO->>L: ROLLBACK + close()
    Note over L: OS releases file lock
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant C as CLI/Desktop
    participant OD as openOwnedLocalDatabase
    participant DO as acquireDataDirOwnership
    participant M as migrateLocalV1ToV2IfNeeded
    participant DB as createSqliteFumaDb
    participant L as Lock DB (data.db.owner-lock)

    C->>OD: openOwnedLocalDatabase(dataDir)
    OD->>DO: acquireDataDirOwnership(dataDir)
    DO->>L: "createClient (journal_mode=DELETE, busy_timeout=0)"
    DO->>L: BEGIN EXCLUSIVE
    alt Lock already held by another process
        L-->>DO: SQLITE_BUSY
        DO-->>C: throw DataDirOwnershipHeld
        Note over C: CLI detects error, attaches to existing server
    else Lock acquired
        DO-->>OD: "{ lockPath, release }"
        OD->>M: migrateLocalV1ToV2IfNeeded(sqlitePath)
        M->>M: recoverV1V2Migration (journal check)
        alt v1 database found
            M->>M: copySqliteFileSet to staging
            M->>M: replay legacy migrations
            M->>M: insertPlan (BEGIN/COMMIT in staging)
            M->>M: verifyStagingDatabase
            M->>M: completeJournaledFlip (atomic rename)
        end
        M-->>OD: LocalV1V2MigrationResult
        OD->>DB: createSqliteFumaDb(sqlitePath)
        DB-->>OD: SqliteFumaDb
        OD-->>C: "OwnedLocalDatabase { db, close }"
        Note over L: Exclusive lock held for process lifetime
    end

    C->>OD: database.close()
    OD->>DB: db.close()
    OD->>DO: ownership.release()
    DO->>L: ROLLBACK + close()
    Note over L: OS releases file lock
Loading

Comments Outside Diff (1)

  1. apps/local/src/db/v1-v2-migration.ts, line 196-209 (link)

    P2 Tmpdir leak in resolveLegacyMigrationsFolder

    When running from a compiled binary (embeddedLegacyMigrations is non-null), every call to resolveLegacyMigrationsFolder creates a new directory under the OS tmpdir that is never removed. replayLegacyV1Migrations can call it twice in one migration run: once inside the applied.length > 0 branch (for the prefix hash check), and again via tolerantReplayLegacyChain when the journal is not a clean prefix. Both tmpdirs are abandoned. On Linux/macOS the OS clears /tmp on reboot, but on Windows %TEMP% persists across reboots and the directories accumulate. Since the migration runs at most once per data dir, the practical impact is small, but passing migrationsFolder into tolerantReplayLegacyChain would eliminate the double materialization entirely.

Reviews (1): Last reviewed commit: "Fix loadSharedHandle being able to poiso..." | Re-trigger Greptile

import * as Effect from "effect/Effect";

// ---------------------------------------------------------------------------
// Daemon discovery + spawn-dedup state — hints, not DB safety.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The new comment block uses an em-dash (), which is banned by AGENTS.md everywhere in the codebase (prose, docs, code comments, commit messages). Replace with a colon or restructure the sentence.

Suggested change
// Daemon discovery + spawn-dedup state hints, not DB safety.
// Daemon discovery + spawn-dedup state: hints, not DB safety.

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

readonly path: string;
}
// ---------------------------------------------------------------------------
// server-control/server.json — a discovery/attach HINT, not an ownership proof.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The new header comment uses an em-dash (), which is banned by AGENTS.md everywhere. The same pattern also appears in daemon-state.ts.

Suggested change
// server-control/server.json a discovery/attach HINT, not an ownership proof.
// server-control/server.json: a discovery/attach HINT, not an ownership proof.

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@RhysSullivan RhysSullivan merged commit edc4411 into main Jun 28, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants