Skip to content

fix(db): use rollback-journal mode for central DB to stop SIGBUS crashes - #1752

Merged
gsxdsm merged 2 commits into
mainfrom
feature/crash-diagnose
Jun 25, 2026
Merged

fix(db): use rollback-journal mode for central DB to stop SIGBUS crashes#1752
gsxdsm merged 2 commits into
mainfrom
feature/crash-diagnose

Conversation

@gsxdsm

@gsxdsm gsxdsm commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

Running more than one fusion process on a host (multiple dashboards/CLIs across worktrees, all attaching ~/.fusion/fusion-central.db) could crash a node process at random — instantly, with no JS stack and nothing in the logs. This happened 3 times in 3 days on one machine. After this change those processes coexist without crashing.

The crash was an OS-level SIGBUS (EXC_BAD_ACCESS, FS pagein error / kernel cluster_pagein past EOF) inside SQLite's walIndexReadHdr. In WAL mode every connection coordinates through a memory-mapped -shm wal-index; on macOS/APFS, when one process resizes/rebuilds that file during a checkpoint while another has it mmap'd, the reader faults on the now-out-of-bounds page. A hardware memory fault can't be caught by node:sqlite or JS, so the whole process dies.

The fix switches the central DB to journal_mode = DELETE (rollback journal), which uses no -shm memory map and coordinates cross-process access via POSIX byte-range locks instead — removing the faulting surface entirely while keeping multi-process access. The existing busy_timeout absorbs the writer serialization that DELETE mode trades for WAL's reader/writer concurrency. Per-project DBs (db.ts) are intentionally left on WAL: they're single-process-per-project and don't hit this cross-process fault. SQLite migrates the existing WAL database on first open (checkpoints -wal into the main file and removes -wal/-shm), so there is no data loss.

Test plan

  • New regression tests in central-db.test.ts assert the central DB reports journal_mode = delete (not wal) and that no -shm wal-index file is ever created even after write traffic — i.e. the exact faulted surface is gone.
  • All 6 central-DB suites pass (221 tests); @fusion/core typechecks clean.

Compound Engineering
Claude Code


Open in Stage

Summary by CodeRabbit

  • Bug Fixes
    • Improved stability when multiple dashboards or CLIs run on the same machine.
    • Switched the local database to a safer journaling mode to reduce rare crash issues on macOS/APFS.
    • Prevented creation of extra database side files during normal operation, while keeping data durability and lock-based coordination in place.

The central DB (~/.fusion/fusion-central.db) is opened concurrently by every
fusion process on a host. In WAL mode those connections coordinate through a
memory-mapped `-shm` wal-index; on macOS/APFS a reader takes a SIGBUS
(walIndexReadHdr / `cluster_pagein past EOF`) when another process resizes it
mid-checkpoint, killing the node process with no JS stack or log. Observed 3x
in 3 days. Switch the central DB to journal_mode=DELETE, which uses no `-shm`
mmap and coordinates cross-process access via POSIX byte-range locks instead;
busy_timeout absorbs the added writer serialization. Per-project DBs keep WAL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ghost

ghost commented Jun 25, 2026

Copy link
Copy Markdown

Ready to review this PR? Stage has broken it down into 3 individual chapters for you:

Title
1 Switch central DB to rollback-journal mode
2 Verify journal mode and file absence in tests
3 Document fix in changeset
Open in Stage

Chapters generated by Stage for commit e53f50e on Jun 25, 2026 6:52am UTC.

@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@gsxdsm, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 28 minutes and 47 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 17f66e24-b5df-4e8c-aa40-2d7c842d10e4

📥 Commits

Reviewing files that changed from the base of the PR and between 50a9471 and e53f50e.

📒 Files selected for processing (2)
  • packages/core/src/__tests__/central-db.test.ts
  • packages/core/src/central-db.ts
📝 Walkthrough

Walkthrough

Central database SQLite setup now uses DELETE journal mode instead of WAL, removes WAL-specific PRAGMAs, and sets synchronous to FULL. Tests were updated to verify the new journal mode and the absence of -shm and -wal files after writes. A changeset documents the patch release.

Changes

Central DB journal mode change

Layer / File(s) Summary
SQLite PRAGMA update
packages/core/src/central-db.ts
The central database constructor now sets journal_mode=DELETE, removes WAL-only PRAGMAs, and sets synchronous=FULL.
Regression tests and release note
packages/core/src/__tests__/central-db.test.ts, .changeset/central-db-delete-journal-sigbus.md
The central database test file now checks DELETE mode, busy_timeout, synchronous=FULL, and absence of -shm and -wal files; the changeset records the patch release.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • Runfusion/Fusion#1692: Also changes packages/core/src/central-db.ts and central-db.test.ts around SQLite PRAGMA behavior, but in the opposite direction (WAL tuning versus DELETE mode).

Poem

🐰 I hopped through the database night,
No WAL ghosts left in sight.
Delete mode keeps the burrow neat,
With FULL sync on for a steady beat.
Thump-thump! The central den feels right.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: switching the central DB to rollback-journal mode to prevent SIGBUS crashes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/crash-diagnose

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Switches the central DB (~/.fusion/fusion-central.db) from WAL mode to DELETE (rollback-journal) mode to eliminate the mmap'd -shm wal-index that caused a hardware SIGBUS (cluster_pagein past EOF) when one process resized it mid-checkpoint while another had it mapped.

  • central-db.ts: The constructor now uses prepare("PRAGMA journal_mode = DELETE").get() instead of exec(...), capturing both the throw (SQLITE_BUSY) and the silent-return ("wal") failure paths during the rolling-upgrade window, and emits an explicit console.warn rather than silently leaving the crash surface in place. WAL-only PRAGMAs (wal_autocheckpoint, journal_size_limit) are correctly dropped; synchronous = FULL and busy_timeout = 5000 are retained.
  • central-db.test.ts: Three new regression tests assert journal_mode = delete, confirm -shm/-wal files are never created after write traffic, and verify the warning path when a WAL holder blocks the migration (using a real DatabaseSync connection with an open read transaction and busyTimeoutMs: 0).
  • Per-project DBs (db.ts) are intentionally unchanged — they are single-process-per-project and do not hit the cross-process fault.

Confidence Score: 5/5

Safe to merge — the fix is surgical, the failure path (WAL holder blocks migration) is explicitly handled with a loud warning rather than a throw, and the regression tests are thorough.

The change is a well-understood SQLite journal mode switch with no schema changes, no data loss risk (SQLite checkpoints WAL into the main file on first open), and a comprehensive test suite covering the normal path, the -shm absence proof, and the rolling-upgrade warning path. The previous review concern about silent failure on WAL→DELETE is fully addressed: the return value is now captured via prepare().get(), both the throw and the no-op paths emit a console.warn, and a new test exercises the blocked-migration scenario end-to-end.

No files require special attention.

Important Files Changed

Filename Overview
packages/core/src/central-db.ts Replaces WAL mode with DELETE (rollback-journal) mode in the constructor; captures both the throw and the silent-return failure paths for the WAL→DELETE transition and emits a loud warning rather than crashing or silently staying in WAL.
packages/core/src/tests/central-db.test.ts Adds three targeted regression tests: journal_mode=delete assertion, absence of -shm/-wal files after write traffic, and the rolling-upgrade path (WAL holder blocks migration → warning emitted, no throw).
.changeset/central-db-delete-journal-sigbus.md Patch-level changeset entry with accurate description of the fix and its motivation.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant P1 as Process A (new)
    participant SQLite as SQLite Engine
    participant FS as Filesystem
    participant P2 as Process B (WAL holder)

    P2->>FS: "PRAGMA journal_mode = WAL (open -shm/-wal)"
    P2->>SQLite: "BEGIN (holds WAL read lock)"

    Note over P1,SQLite: Rolling-upgrade window
    P1->>SQLite: "PRAGMA busy_timeout = 5000"
    P1->>SQLite: "PRAGMA journal_mode = DELETE"
    SQLite-->>P1: "returns wal (exclusive lock blocked)"
    P1->>P1: "journalMode !== delete → console.warn"

    Note over P1,FS: Clean state — no WAL holder
    P1->>SQLite: "PRAGMA busy_timeout = 5000"
    P1->>SQLite: "PRAGMA journal_mode = DELETE"
    SQLite->>FS: "checkpoint WAL, delete -wal/-shm"
    SQLite-->>P1: "returns delete"
    P1->>SQLite: "PRAGMA synchronous = FULL"
    P1->>SQLite: "PRAGMA foreign_keys = ON"
    Note over P1,FS: No -shm mmap — SIGBUS surface eliminated
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant P1 as Process A (new)
    participant SQLite as SQLite Engine
    participant FS as Filesystem
    participant P2 as Process B (WAL holder)

    P2->>FS: "PRAGMA journal_mode = WAL (open -shm/-wal)"
    P2->>SQLite: "BEGIN (holds WAL read lock)"

    Note over P1,SQLite: Rolling-upgrade window
    P1->>SQLite: "PRAGMA busy_timeout = 5000"
    P1->>SQLite: "PRAGMA journal_mode = DELETE"
    SQLite-->>P1: "returns wal (exclusive lock blocked)"
    P1->>P1: "journalMode !== delete → console.warn"

    Note over P1,FS: Clean state — no WAL holder
    P1->>SQLite: "PRAGMA busy_timeout = 5000"
    P1->>SQLite: "PRAGMA journal_mode = DELETE"
    SQLite->>FS: "checkpoint WAL, delete -wal/-shm"
    SQLite-->>P1: "returns delete"
    P1->>SQLite: "PRAGMA synchronous = FULL"
    P1->>SQLite: "PRAGMA foreign_keys = ON"
    Note over P1,FS: No -shm mmap — SIGBUS surface eliminated
Loading

Reviews (2): Last reviewed commit: "Address PR review feedback (#1752)" | Re-trigger Greptile

Comment thread packages/core/src/central-db.ts Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/core/src/central-db.ts`:
- Line 581: The SQLite journal mode switch is being executed without verifying
whether it actually changed to DELETE, so a failed transition can leave the
database in WAL mode and preserve the crash risk. Update the journal-mode
handling in central-db around this.db.exec so the result of PRAGMA journal_mode
= DELETE is captured and checked, and only proceed when the returned mode
confirms DELETE; if it returns a different mode, treat it as a failure and
surface that condition instead of assuming success.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8607ebeb-dbf0-497f-a8d8-abca8e32d13f

📥 Commits

Reviewing files that changed from the base of the PR and between f981679 and 50a9471.

📒 Files selected for processing (3)
  • .changeset/central-db-delete-journal-sigbus.md
  • packages/core/src/__tests__/central-db.test.ts
  • packages/core/src/central-db.ts

Comment thread packages/core/src/central-db.ts Outdated
- Verify the WAL->DELETE journal-mode switch instead of discarding exec()'s
  result. During a rolling upgrade a lingering WAL holder blocks the exclusive
  lock the switch needs, so SQLite either throws SQLITE_BUSY or no-ops and
  returns "wal". Capture both outcomes and warn loudly so the residual -shm
  SIGBUS surface is observable, rather than silently swallowed.
- Do not rethrow: the condition is transient and self-healing (the next start
  after the last WAL holder exits migrates cleanly); hard-failing would make the
  central DB unopenable during the very upgrade window it describes.
- Add a migration-path regression test (a WAL holder blocking the switch) that
  the prior fresh-DB-only tests did not cover.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gsxdsm
gsxdsm merged commit 5cd795a into main Jun 25, 2026
6 checks passed
@gsxdsm
gsxdsm deleted the feature/crash-diagnose branch June 25, 2026 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant