Skip to content

[707/P0] Finish vector-store executor isolation and bounded route shutdown #915

Description

@ScriptedAlchemy

Priority: P0
Blocks undraft #707: yes
Labels: bug, ci, P0, v2, windows, macos (suggested)

Symptom

Classification: Product liveness; repeat failures plus observed flakes.

Bootstrap shutdown timed out; linked-worktree shutdown timed out; RMCP first attempt exceeded 360 seconds.

Baseline: run 34039753429, head 9d17537b4. Exact test rows below come from the archived reports; Dashboard comes from its job log.

  • linuxtracedecay daemon::tests::bootstrap::mcp_bootstrap_catalog_bypasses_project_writer_gate — failed initial and retry (job).
  • linuxtracedecay daemon::tests::rmcp_route::unix_production_route_selects_rmcp_only_after_initialize — passed retry; CI failed-on-flaky (job).
  • macostracedecay daemon::tests::runtime_identity::concurrent_same_identity_worktrees_keep_exact_server_and_scheduler_bindings — failed initial and retry (job).
  • windows-5tracedecay-code-index-runtime code_index_scheduler::tests::shutdown_signals_code_index_worker_without_taking_busy_scheduler_lock — passed retry; CI failed-on-flaky (job).

Root cause

The Linux bootstrap test fails at shutdown, not at its catalog assertion. The RMCP initialize test timed out on its first attempt; the macOS linked-worktree test also times out during shutdown. These are not proof that catalog dispatch or RMCP selection is wrong. Synchronous vector-store operations inside async methods are a verified starvation mechanism in this branch, but the reports do not prove that every timeout here has that one cause.

Exact fix

Audit crates/tracedecay-usecases/src/store/vector_generations/graph_adapter.rs and its synchronous constructors/read/retention callers, plus crates/tracedecay/src/daemon/project_open_owners/ and route retirement in engine.rs/project_routing.rs. Keep one Arc-owned store state and bounded blocking dispatch for complete operations, not only writer acquisition. Retain permits and settlement ownership inside dispatched jobs. Keep exact-SQL's existing writer authority; do not add a second scheduler-owned SQL writer. Ensure route cancellation reaches every wait and join before releasing the database scope. In daemon/tests/bootstrap.rs:3228 and runtime_identity.rs:382, record the owner/join preventing shutdown rather than masking it with a longer timeout.

Source/implementation entry points:

Acceptance

On a single-worker runtime, hold a writer, queue a semantic operation, and prove a heartbeat and the transaction holder still run and commit. Repeat with constrained CPU, cancellation, and shutdown. No completion may depend on transaction-idle expiry. Run the bootstrap catalog assertion and its teardown separately; both must meet their original bounds. No lingering child or write may outlive the owning database scope.

Notes / related issues

Do not duplicate the now-existing implementation: current-head history contains 559758395 (bounded vector-store dispatch). Update #913's liveness gate and #912 for the demonstrated linked-route component. A fresh exact-head run is required; the old red report does not establish whether that fix succeeded. #863 is a tracking umbrella, not proof of cross-test contamination.

All work targets codex/tracedecay-total-redesign-plan-reopened. Keep #707 draft; no undraft or merge is authorized by this draft.


Filed from ChatGPT 6 Pro review of PR #707 CI baseline run 34039753429 (head 9d17537b4). Local draft #2 (P0). Do not undraft/merge #707 without Zack authorization.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions