Skip to content

[707/P1] Make crash and snapshot fixtures exercise recovery without illegally copying live Windows stores #933

Description

@ScriptedAlchemy

Priority: P1
Blocks undraft #707: yes
Labels: bug, ci, P1, v2, windows (suggested)

Symptom

Classification: Durability test preconditions; possible underlying file-sharing defects.

Four Windows crash/snapshot tests fail on a native lock while constructing the scenario; one payload test misses its intended size refusal.

Baseline: run 34039753429, head 9d17537b4. Exact test rows below come from the archived reports; Dashboard comes from its job log.

  • windows-2tracedecay-graph-db::durability_crash_contract crc_faulted_store_is_quarantined_with_its_wal_sidecar_and_rebuilt — failed initial and retry (job).
  • windows-3tracedecay-runtime-core sqlite_read_snapshot::tests::copied_snapshot_survives_empty_writer_sidecar_cleanup — failed initial and retry (job).
  • windows-5tracedecay-graph-db runtime::tests::reopen_of_a_dirty_store_copy_replays_the_walled_commit — failed initial and retry (job).
  • windows-5tracedecay-graph-db::durability_crash_contract reopen_collapses_replayed_wal_history_from_an_unclean_shutdown — failed initial and retry (job).
  • windows-2tracedecay-lcm payload::filesystem_authority::authority_tests::oversized_payload_is_rejected_before_allocation_or_read — failed initial and retry (job).

Root cause

Graph crash/reopen and SQLite snapshot tests fail with lock violation 33 while arranging a copy/cleanup. The oversized-payload test keeps its created file handle alive and reports only an unexpected error kind, so its exact earlier refusal must be retained before deciding whether sharing or privacy admission is responsible. These failures do not yet prove WAL replay is wrong.

Exact fix

In tracedecay-graph-db/tests/durability_crash_contract.rs, runtime copy tests, tracedecay-runtime-core/src/sqlite_read_snapshot.rs, and the LCM payload fixture, identify the failing setup syscall and all live handles. For a crash image, run a child, synchronize at the intended durable phase, terminate/join it without a clean database close, and only then copy the remaining bytes/sidecars. For an online consistent backup, use the supported snapshot/backup authority instead of fs::copy of locked files. Close irrelevant creator handles before size-only admission tests. Run durability acceptance with unsafe-fast options unset and verify the actual journal/sync settings of the exercised store.

Source/implementation entry points:

Acceptance

Tests reach their named crash boundary and demonstrate reopen/replay, quarantine, sidecar handling and no partial commit. A locked foreign/live store still refuses safely. Oversized input is rejected before allocation/read for the intended size reason after valid file admission. No false crash proof from a clean close, and no durability claim from a fast/unsafe store configuration.

Notes / related issues

NEW fixture fidelity issue under #903. Distinct from translating try-lock contention to Busy. The baseline Windows workflow exports TRACEDECAY_SQLITE_UNSAFE_FAST=1 to the whole shard; explicitly audit which tested backends it affects before crediting durable acceptance.

All work targets codex/tracedecay-total-redesign-plan-reopened. Keep #707 draft; no undraft or merge is authorized by this draft.


Filed from ChatGPT 6 Pro review of PR #707 CI baseline run 34039753429 (head 9d17537b4). Local draft #20 (P1). Do not undraft/merge #707 without Zack authorization.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions