fix(db): reset proof-dependent state to genesis (DEV + PRD prover recovery) - #208
Merged
Conversation
Promote: staging -> develop
* feat(api): add GET /api/history endpoint (#153) (#162) * feat(api): add GET /api/history endpoint (#153) Adds paginated per-address transaction history reading from the account_history table populated by the migration-0008 trigger. * /api/history?address=<hex>&limit=<n>&offset=<n>, behind 'always' status (no feature gate). * Reuses the same hex decode + 32-byte length rules /api/balance applies; rejects missing address, invalid hex, limit outside [1, 200], negative offset with 400. * Newest-first ORDER BY changed_at DESC; offset beyond total returns empty items with the unfiltered total so the caller can detect end-of-list. * LEFT JOIN observed_inscriptions + pending_inscriptions on triggering_commit_txid so block_height + status surface once a future caller threads zkcoins.account_commit_txid through the upsert. Today both joined columns are NULL and txid/block_height remain null on the wire. * counterparty + memo intentionally null in v1: the current schema does not store the recipient address per-mutation and has no memo column. Out-of-scope for this PR: app UI changes, OpenAPI export, WebSocket push, the Zod schema in zk-coins/app. * README Features table entry added. * fix(api/history): round-2 review fixes — SQL-side filter, pending default, 422, one DB call Six fixes from the two independent reviews of #153: 1. Push `source IN ('mint','send','receive')` into both the page query and the filtered total so pagination is correct. The post-fetch `filter_map` stays as a defense-in-depth safety net but no longer actually drops rows. Closes a bug where `total` over-counted hidden rows and pages came back smaller than the requested `limit`. 2. Status default flips from `confirmed` to `pending`. A DB-committed `account_history` row only proves a server-side state change, not an on-chain confirmation. The new mapping: * pending_inscriptions.status='complete' -> 'confirmed' * pending_inscriptions.status='failed' -> 'failed' * pending_inscriptions.status IN ('constructed', 'commit_broadcast','reveal_broadcast') -> 'pending' * no pending row + observed_inscriptions.block_height IS NOT NULL -> 'confirmed' * no pending row + no observed row -> 'pending' The match goes through a new `PendingInscriptionStatus` enum so the `match` is exhaustive — a future schema state addition fails to compile (no `_ => "pending"` catch-all). 3. Collapse `count_account_history` + `list_account_history` into one round-trip via a CTE: one filtered-count CTE cross-joined to the LIMIT/OFFSET page CTE. The handler now has a single DB error branch, closing the dead-arm coverage gap the two-call layout left behind. The empty-page case still returns the real total (sentinel row) so the caller can drive pagination without a second query. 4. Switch all input-validation status codes from 400 to 422 to match `/api/balance` and the rest of the read surface. Framework-level 400 (axum Query rejection on non-integer limit) stays. 5. A non-null `prev_data` blob that fails to bincode-decode no longer silently collapses to `prev_balance = 0` (which would fabricate the full new balance as the delta). The row is dropped with a warn log. 6. TODOs for the deferred work reference the follow-up issues: * #159 — thread `zkcoins.account_commit_txid` GUC * #160 — capture counterparty_address per row (zk-coins/app#145 covers the typed-client wiring on the other repo.) * fix(api/history): hoist blob.len() out of tracing::warn! for coverage The tracing macro lazily evaluates its arguments based on the active log level, so under the default-off test subscriber blob.len() is never executed. Coverage Gate flagged it as the only uncovered line in the node + shared scope (99.97% lines, 1 missed). Pre-compute blob_len in a let binding so the line runs regardless of tracing config. * docs: clarify operator language — api.zkcoins.app runs at zkcoins.app, not DFX (#158) Three spots in README.md leaked the DFX-as-operator framing: 1. Trust Model table row was "Yes — DFX runs the hosted node". The hosted node at api.zkcoins.app is operated by zkcoins.app (one of hopefully many such service providers). DFX is the underlying hosting / financial-services layer — invisible to wallet integrators and consumers of this README. 2. Configuration table description for ESPLORA_URL said "PRD: ... (DFX Mainnet stack)". Replaced with "On the api.zkcoins.app stack: PRD ..., DEV ...". Same meaning, correct attribution. 3. ESPLORA_WS_URL description analog — "on the DFX mempool/backend stack" → "self-hosted mempool/backend sidecar" (the relevant detail is "self-hosted vs external", not "whose hosting"). Memory: reference_zkcoins_org_structure documents the three-layer separation (zkCoins protocol / zkcoins.app service provider / DFX infra) so this distinction does not get muddled again. No code change. Pre-push sanity: fmt clean, clippy clean. * perf(bootstrap): warmup prover in background; gate /health/ready on prover_warm (#154) * perf(bootstrap): warmup prover in a background task; gate /health/ready on prover_warm PR #147 paid the ~7 s Plonky2 cold-prove tax synchronously between load_from_pg and TcpListener::bind, which pushed API offline time per deploy from ~14 s (circuit build alone) to ~21 s (circuit build + cold prove). The user constraint is explicit: API must be reachable as soon as possible. PR #147 was closed for failing that constraint. This shape moves the warmup off the bootstrap-critical path: 1. TcpListener::bind returns at ~0.1 s. axum::serve starts draining connections — /health (liveness) is 200, /api/* paths return correct answers (a /api/mint or /api/send during the warmup window pays the ~7 s cold tax, but it serves correctly). 2. tokio::task::spawn_blocking launches AccountNode::warmup_prover on the blocking pool so the CPU-bound prove does not starve the tokio worker that owns axum::serve. 3. After ~21 s the warmup task flips the new prover_warm Arc<AtomicBool> to true. /health/ready transitions from 503 with {"status":"starting","prover":"warming","failures":["prover"]} to 200 with {"status":"ready","prover":"ready"}. A load balancer / Kuma monitor keyed on /health/ready holds traffic on the previous-generation pod through the warmup window; /health (liveness) is unaffected so container restart loops are not triggered. ZKCOINS_SKIP_BOOTSTRAP_WARMUP=1 skips the background task entirely (smoke tests in runtime_tests.rs). Three architecture decisions, codified in MIGRATION_RESEARCH.md §7.25: - spawn_blocking over tokio::spawn — Plonky2 prove is CPU-bound and would starve the tokio worker dispatching HTTP requests. - Arc<AtomicBool> over Arc<RwLock<bool>> — flag is write-once + read-many, AtomicBool::store is a single instruction. - std::process::exit(1) over panic!() — a panic inside spawn_blocking only surfaces when the JoinHandle is awaited (it deliberately is not), so a bare panic would leave the node serving 503 forever. exit(1) crash-loops the container at the same severity as PR #147's synchronous expect(). CONTRIBUTING.md gains a Bootstrap timing section + a row for the new env var. AccountNode::warmup_prover + the warmup_prover_completes_successfully test were adapted from PR #147 with the return type switched to anyhow::Result for the runtime call site. A new router test asserts /health/ready returns 503 with the warming-tag payload when prover_warm is false. * docs(runtime): correct warmup-task scanner-ordering comment Reviewer caught: the prior comment claimed the scanner spawns AFTER start_rest_node returns. That is factually wrong — main.rs runs start_rest_node + run_scanner_ws concurrently via tokio::spawn. The correctness conclusion still holds because the scanner locks `state`, not `account_node`. Rewrite the comment to name the right invariant and the right contender (a user request that lands during the ~7 s warmup window). * test(coverage): drop unused .map_err closure in warmup_prover The previous shape `.map(|_| ()).map_err(|e| anyhow!("...{e}"))` left two never-called closures in the happy-path test, costing the 100% function + 100% line coverage gate. `?` propagation matches what `prove_initial` already returns (`anyhow::Result<Proof>`) and is covered by the same single test. --------- Co-authored-by: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com>
Promote: staging -> develop
…174) (#175) Post-deploy smoke test was checking `/api/info` for HTTP 200, but after #154 the node binds the HTTP listener BEFORE the Plonky2 prover warmup completes — `/api/info` returns 200 within seconds while `/health/ready` stays at `{"ready":false,"prover":"warming"}` for the 10-30 s warmup window. Downstream jobs (API E2E preflight against `/health/ready` + `/health/publisher`) raced the warmup: the E2E job picked the runner up ~4 s after the deploy job reported success, hit `/health/ready` once, got back the warming snapshot, and failed with `::error::/health/ready not ready` — observed empirically on Release PR #166's run https://github.com/zk-coins/node/actions/runs/26793933906/job/78986599030. Switch the smoke loop to `/health/ready` + a `jq '.ready == true'` assertion, keeping the 30-attempt × 10-s budget (~5 min) so a genuine bootstrap stall still surfaces with the same timeout behaviour. The deploy job now only reports success once the node is actually ready for traffic, which removes the race the E2E preflight was tripping over. `/api/info` is no longer a deploy-success signal. The E2E preflight retains its explicit `/health/ready` + publisher-wallet gate as a sanity check (still a single shot — it relies on the smoke test having already enforced readiness). The deploy job runs on `ubuntu-24.04-arm` where `jq` is part of the default GitHub-hosted image; no install step needed here. Co-authored-by: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com>
… sweep (2026-06-02) (#177) (#178) * bench(results): Apple M5 Max vs Apple M3 Ultra (probe_r2 + HTTP mint sweep) First Apple M5 Max measurement of the Plonky2 prover hot path, captured against the same git_sha era as the Apple M3 Ultra baseline persisted on 2026-05-31 in r2_probe_runs (host_id 1). Headline numbers (probe_r2, synthetic): circuit_build: 14214 -> 8245 ms (-42%) prove_cold: 7012 -> 6129 ms (-13%) prove_warm_p50: 4777 -> 4350 ms (-9%) peak_rss: 4112 -> 3938 MiB (-4%) HTTP /api/mint sweep on M5 Max (10 samples, empty state, unfunded publisher -> broadcast 503): n=10 p50=6.906s p90=7.056s min=6.611s max=7.118s All three ROADMAP-step-9 R2 budgets (warm <= 5 s, cold <= 30 s, RSS <= 64 GiB) pass on M5 Max. Persisted JSON has its hostname field scrubbed to a generic label; the raw fingerprint is kept only in the persisted DB row. * docs(bench): unified results table — proof type x hardware Restructure scripts/bench/results/README.md to lead with a single table mapping each proof phase to its wall time on each hardware target, with both synthetic (probe_r2) and live (HTTP) numbers in one place. Adds the explicit caveat that the R2 ideal budget (<= 1 s warm prove) is not reachable via per-generation Apple silicon upgrades alone — Plonky3 / circuit optimisation remains the dominant lever. Co-authored-by: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com>
* feat(openapi): generate OpenAPI 3.x spec from handler annotations (#157) * feat(openapi): generate OpenAPI 3.x spec from handler annotations Annotate every public `/api/*` handler with `#[utoipa::path]` and derive `ToSchema` on request/response types. Build a process-wide `ApiDoc` from those annotations and serve it at `GET /openapi.json` (cached JSON) and `GET /docs` (Swagger UI pinned to swagger-ui-dist@5.32.6). Feature-gated handlers (`address-list`, `username-claim`, `lnurl`) carry their own sub-doc that merges into the main spec only when the feature compiles in, so the document describes the exact wire surface of the running binary. Refs: #155 * refactor(openapi): bundle Swagger UI, drop hardcoded URLs, fix feature combos - Remove static `servers(...)` block; spec inherits "same host" via OpenAPI default - Bundle Swagger UI assets via `utoipa-swagger-ui`; no external CDN dependency - Fix `cargo clippy` with single-feature builds (`lnurl` only, etc.) - Smoke test verifies relative asset URLs and absence of hardcoded hosts * refactor(openapi): polish handler annotations for production - Replace placeholder ellipsis `…` (U+2026) in `SendCoinRequest` PublicKey schema examples with valid 33-byte compressed-hex pubkeys (secp256k1 generator point and its double) so client generators that parse the `example` as hex do not trip over a non-hex character. - Drop redundant `#[schema(rename = "...")]` on `LnurlpResponse` — utoipa already picks up the corresponding `#[serde(rename)]` and duplicating the directive risks future drift between serde and schema. Verified the generated spec still emits `minSendable` / `maxSendable` after removal. - Replace remaining non-ASCII ellipsis in doc and source comments with ASCII `...` for consistency. Handler tag grouping is left as-is: every annotated handler already carries a `tag` (`Accounts`, `Coins`, `Inscriptions`, `Node`, `Usernames`, `LNURL`), so Swagger UI groups them all correctly instead of dropping any into the default bucket. * test(openapi): cover async HTTP handlers and Swagger asset paths The smoke test in `tests/openapi_smoke.rs` exercises the in-memory spec and HTML string paths but never enters the async handlers (`openapi_json_handler`, `docs_handler`, `swagger_asset_handler`). Coverage Gate flagged lines 195-237 of `node/src/openapi.rs` as uncovered. Add sibling-style `openapi_tests.rs` exercising: - OK + JSON content-type from `openapi_json_handler` - OK + HTML content-type from `docs_handler` - bundled CSS and JS asset serving via `swagger_asset_handler` - 404 from `swagger_asset_handler` for unknown files - `swagger_ui_config` cache identity * test(db): harden setup_pool + connect_and_migrate against shared-host load The m3-ultra CI runner (dfx01) co-resides with ~20 production containers (Vaultwarden, Grafana, Loki, dEURO, …) and occasional manual `cargo nextest` runs from operators. Under that load the testcontainers `postgres:17` "ready" log signal fires correctly, but the subsequent SQLx pool connect can stall past the 30s default `acquire_timeout` waiting on the Colima vNIC to complete the TCP handshake. The failure surfaced as a one-shot `sqlx::Error::PoolTimedOut` in `db::tests::*` whenever multiple Heavy CI runs were in-flight on the same host (PR-CIs + production + ad-hoc tests = loadavg ~75 on 28 cores). Two independent hardenings: 1. `db.rs::connect_and_migrate` — bump `acquire_timeout` from the 30s default to 60s. The healthy path connects in <500ms so this only changes behaviour when the host is starved; in that window it is the difference between a flake and a pass. Production bootstrap inherits the same bound — a 30s vs 60s acquire timeout on a node that has been alive for milliseconds is not user- facing latency, and a freshly-started Postgres sidecar under Docker Compose orchestration can also need >30s to finish its first checkpoint pass on a busy host. 2. `db_tests.rs::setup_pool` — wrap the container-start-and-connect sequence in a 3-attempt retry with linear backoff (500ms / 1000ms). The previous container is dropped on each retry so a hung Postgres process never poisons the next attempt. The retry only fires when the host is transiently overloaded; a healthy run still hits the first attempt and pays no overhead. The aggregate effect is that one test panicking with PoolTimedOut no longer cancels the rest of the 371-test suite — the run either re-converges on the next attempt or reports a real container-engine outage three retries deep. * fix(db): retry connect_and_migrate on transient host-load failures The previous fix to `db_tests::setup_pool` retried at the container-start + connect pair, but the 20+ test files that call `connect_and_migrate` from their own ad-hoc `setup_pool` did not inherit it. Coverage Round 2 surfaced a second transient failure mode at `state_tests.rs:48`: connect_and_migrate failed: Protocol("unexpected response from SSLRequest: 0x48 (sqlx_postgres::connection::tls:95)") `0x48 = 'H'` — the testcontainers ready-signal had fired, but the first byte SQLx saw on the wire was garbage instead of the protocol handshake. Same root cause as the earlier `PoolTimedOut`: on the shared m3-ultra host (loadavg ~75 on 28 cores when this fired) the Colima vNIC delivered the bgwriter/autovacuum log lines before the listener-side socket had finished its first message exchange. Move the retry inside `connect_and_migrate` itself so every call site — not just `db_tests::setup_pool` — benefits. Three-attempt retry with linear 500ms / 1000ms backoff, classified for the two documented transient sqlx error kinds (`PoolTimedOut`, `Protocol(... SSLRequest ...)`). Auth / migration / host-not-found errors stay non-retryable so a real misconfiguration still fails fast in <1s. Mark the retry loop + classifier `#[cfg_attr(coverage_nightly, coverage(off))]` — both are defensive against host-load conditions that the deterministic test harness cannot reproduce on demand. The healthy-path `try_connect_and_migrate` worker stays fully covered by every test that hits a Postgres testcontainer. * feat(openapi): cover /, /health, /health/ready, /health/publisher, /api/history Brings the always-on wire surface into the generated spec without any hand-written drift surface: - `GET /` (root_handler) — service identification + endpoint map - `GET /health` (health_handler) — promote from inline closure to a named handler so the liveness probe carries a `#[utoipa::path]` annotation matching the readiness / publisher probes - `GET /health/ready` (ready_handler) — DB + Esplora + prover-warm gate; both 200 and 503 responses share `ReadyResponse` so Kuma and load-balancer integrations can branch on `status` / `failures` without scraping the HTTP code - `GET /health/publisher` (publisher_health_handler) — UTXO state of the publisher wallet (deploy-dev preflight gate) - `GET /api/history` (get_history_handler) — paginated per-address history (issue #153); 422 / 500 branches reuse the documented `HistoryErrorResponse` envelope so wallet error handling stays in sync with the server contract All new handlers picked up `pub(crate)` visibility plus a `#[utoipa::path]` block; new response structs derive `ToSchema`. `openapi.rs` registers each handler in `paths(...)` and each response type in `components(schemas(...))`. The smoke suite is extended to require every new route under `spec_lists_every_always_on_route` plus `HistoryResponse`, `HistoryItem`, `HistoryErrorResponse`, and `ReadyResponse` under `spec_registers_critical_schemas` so a future regression on either contract fails CI fast. `CONTRIBUTING.md` gains a `REST API & OpenAPI` section: the exposed route table, the four-step recipe for adding a new endpoint, and the existing drift guards. The project-structure tree calls out `openapi.rs` next to `router.rs` so contributors find the spec assembly without spelunking. * refactor(openapi): reviewer-loop polish — tag, 503 schema, root endpoint map Tightens consistency surfaced by the post-rebase review: - Unify `/api/balance` + `/api/history` + `/api/address` under `tag = "Accounts"` (read endpoints keyed on an address). `/api/history` was tagged `"Coins"` in the previous commit; aligned with the pre-existing `/api/balance` annotation and updated CONTRIBUTING table + recipe accordingly. - Promote the `/health/publisher` 503 body from an ad-hoc `serde_json::json!({...})` to a typed `PublisherHealthErrorResponse` with `ToSchema`, registered under `components(schemas(...))` and bound to the 503 entry in the `responses(...)` block. The 200 branch drops the redundant `serde_json::to_value(...)` wrapper now that both arms go through `into_response()`. - Extend `RootEndpoints` to mirror every always-on route: `/api/mint`, `/api/username/resolve/{username}`, `/health/ready`, `/health/publisher`, `/openapi.json`, `/docs`. The struct's doc-comment now explicitly justifies what is omitted (feature-gated routes, admin endpoints) so the endpoint map is not silently stale. - Document the meta + admin exclusion policy in the OpenAPI section of CONTRIBUTING: `/openapi.json`, `/docs`, `/docs/{file}`, and `/api/admin/*` are intentionally outside `paths(...)` and any future admin route should follow the same rule. * fix(openapi): flatten swagger_asset_handler — drop unreachable Err arm The previous shape carried a third `Err(_) => 500` match arm to satisfy the `Result<Option<SwaggerFile>, Box<dyn Error>>` signature of `utoipa_swagger_ui::serve`. Inspection of utoipa-swagger-ui 9.0.2 (`src/lib.rs::serve`) shows the function only returns `Err` in two situations: - the bundled `swagger-initializer.js` bytes fail UTF-8 decoding — impossible because the `vendored` feature bakes a known-good UTF-8 bundle in at compile time - the oauth config formatter errors — impossible because our `swagger_ui_config()` builds a `Config` without an oauth section Both invariants are structurally enforced by our build, so the arm was dead code and surfaced as a single uncovered line under the 100% line-coverage gate. Flatten the Result with `.expect(...)` — the panic message documents the contract and a future upstream change that violates either invariant would surface within minutes via the readiness probe. Updates the two test doc-comments referencing `Ok(Some)` / `Ok(None)` to match the post-flatten shape. * feat(jobs): introduce async Job-API; remove synchronous mint/send/commit (#161) * feat(jobs): introduce async Job-API; remove synchronous mint/send/commit Replaces the synchronous `/api/{mint,send,commit}` surface with an admit-and-poll Job-API. Every wallet-triggered prove operation now returns 202 with a `job_id` immediately; the heavy prove + broadcast work runs in a background dispatcher loop driven by a new `jobs` table. ## Routing changes Removed (legacy synchronous): - `POST /api/send` - `POST /api/mint` - `POST /api/commit` Added (Job-API admit + poll): - `POST /api/jobs/mint` — admit a mint job - `POST /api/jobs/send` — admit a send job - `GET /api/jobs/{job_id}` — poll job state (Retry-After: 2) - `POST /api/jobs/{job_id}/commit` — attach signed commitment - `POST /api/jobs/{job_id}/cancel` — cancel still-queued jobs ## New crates / modules - `node/src/job_store.rs` — typed wrapper around the `jobs` table (CreateResult / JobKind / JobStatus enums, idempotency-key handling). - `node/src/job_dispatcher.rs` — async dispatcher loop; consumes `JobEnvelope`s from an mpsc channel, dispatches by `JobKind`. - `node/src/flow.rs` — the prove + broadcast flows extracted from the old handler bodies, now driven by the dispatcher. - `node/migrations/0014_jobs.sql` — `jobs` table + indexes + CHECK constraints mirroring the typed enums. ## Dependencies - `uuid v1` (`v4`, `serde`) — `jobs.public_id` column. - `dashmap v6` — `job_notify_map` for `commit` wake-up notifications. - `chrono v0.4` — `TIMESTAMPTZ` round-trip on the timestamp columns. - `sqlx` features: `uuid`, `chrono`. ## OpenAPI integration The Job-API admit + poll surface is fully covered by `#[utoipa::path]` annotations on every new handler; the response and request envelopes (`JobAcceptedResponse`, `JobStatusResponse`, `JobErrorResponse`, plus the kept-around `SendCoinRequest` / `MintRequest` / `CommitRequest` shapes the Job-API still consumes) carry `ToSchema` derives and are registered under `components(schemas(...))`. Surviving always-on handlers from #157 keep their annotations; the smoke test `spec_lists_every_always_on_route` now requires every new `/api/jobs/*` path. ## Coverage The PR is iterated to 100% line + function coverage under `cargo llvm-cov nextest --all-features` per the project's standing quality gate. Legacy mint/send/commit handler tests removed alongside the routes; new tests live in `router_tests.rs` covering the admit + poll + commit + cancel paths plus dispatcher-loop behaviour in `job_store_tests.rs`. ## Migration story `migrations/0014_jobs.sql` adds the new `jobs` table; existing data is untouched. Wallets must move to the admit-and-poll surface — the synchronous routes return 404 after this PR lands. ## Root endpoint map `root_handler` now advertises the Job-API routes alongside the operational endpoints (`/health/ready`, `/health/publisher`, `/openapi.json`, `/docs`, `/api/username/resolve/{username}`); the `RootEndpoints` doc-comment documents the omissions (feature-gated routes, admin endpoints, meta). * ci(coverage): expose per-function uncovered list + upload HTML artifact on gate failure When the heavy gate fails, the operator currently sees: - per-file `--show-missing-lines` text (sometimes elides files with branch-only deltas, observed on this PR's failure) - per-file JSON summary (lines/functions percent only) This leaves the actual gap obscured. Solving it required re-running the ~50 min heavy gate locally — that is the wrong incentive, especially on shared-runner builds where a re-run blocks another PR. This commit adds two diagnostics to the existing failure step: - **Per-function symbol list.** A jq filter over the JSON report emits one line per uncovered function as `file:line\tsymbol_name`. The operator sees `node/src/router.rs:1234\tmy_handler` immediately and can target the missing test from the CI log without leaving the browser. - **HTML report upload.** `cargo llvm-cov report --html` generates a Codecov-style browsable report; the new `actions/upload-artifact@v4` step uploads it under `llvm-cov-html-{run_id}-{run_attempt}` for 14 days. The operator downloads the artifact, opens `index.html`, and clicks straight to the file/line/function that's red — same workflow as a local `cargo llvm-cov --open` without paying the ~50 min reproduction cost. Both diagnostics are gated on `if: failure()` so the green path pays nothing. The ignore-regex is pulled into a single `$IGNORE` shell variable so the four invocations stay in sync — the previous duplicated literal made it easy to drift one filter relative to another. * fix(jobs): drop unreachable response_body fallback closure in admit_and_enqueue The idempotent-replay arm for `Completed` jobs called job.response_body.clone().unwrap_or_else(|| serde_json::json!({})) to fall back on an empty JSON object if `response_body` was somehow absent. `JobStore::complete` sets `response_body` on the row before flipping the status to `Completed` (the matching INSERT is non-nullable on the value side), so the closure is unreachable in practice — a `None` would mean the row was hand-edited or the schema invariant broke. The defensive empty-object fallback only existed because the wallet would otherwise receive a 500 on an event that can't happen. llvm-cov scored the closure as a separate uncovered function plus an uncovered line (the closure body), which fails the 100% line + function gate — the only delta keeping PR #161 red after the rebase. Switching to `.expect(...)` documents the invariant inline and surfaces a violation as a fast-panic instead of a silent empty body, matching how this codebase handles the other "structurally unreachable but stdlib forces a fallback" branches (see the `utoipa_swagger_ui::serve(...)` flatten in `openapi.rs`). Found by the new per-file HTML coverage artifact added in the previous commit — a 50 min heavy gate's worth of guessing replaced by reading the artifact for 30 seconds. * feat(jobs): SSE push channel GET /api/jobs/:id/stream (#163) * feat(jobs): introduce JobNotifier broadcast channel for phase fan-out Refactor the dispatcher's per-job coordination primitive from a bare `Arc<Notify>` into a `JobNotifier` struct that pairs the existing commit-wake Notify with a `tokio::sync::broadcast::Sender<JobPhaseEvent>`. Every dispatcher status-persistence site (set_status / set_awaiting_signature / complete / fail) now also publishes a `JobPhaseEvent` so future SSE subscribers can observe real-time phase transitions; the existing commit-wake path is unchanged. The cancel handler also publishes a terminal `cancelled` event so attached listeners see the close. This is the plumbing layer for the SSE push channel — no new endpoint yet, no new behaviour observable from outside the dispatcher. The `AppState` field type widens from `Arc<DashMap<Uuid, Arc<Notify>>>` to `JobNotifyMap` (alias for `Arc<DashMap<Uuid, Arc<JobNotifier>>>`); the runtime resumer and the router commit handler are adjusted to the new shape. Test fixtures construct `JobNotifier::new()` instead of `Notify::new()`. * feat(jobs): add SSE push channel GET /api/jobs/:id/stream Server-Sent Events endpoint that streams real-time phase transitions to wallets without the ~2s poll tax. Layered on top of the `JobNotifier::phase_tx` broadcast channel introduced in the previous commit: - Handler loads the row up-front (404 surfaces with the standard JSON shape, not as an empty stream) and immediately emits an initial `event: phase` (or `event: complete` for terminal jobs) with the current snapshot so re-attached wallets see the latest state without waiting on the next dispatcher transition. - Subscribes a fresh `broadcast::Receiver` per open stream; forwards every subsequent `JobPhaseEvent` as `event: phase`, closes on the first terminal `event: complete`. - `KeepAlive::new().interval(25 s)` heartbeat survives Cloudflare Tunnel's ~100 s idle drop without doubling bandwidth. - Polling fallback (`GET /api/jobs/:id` from PR1) is unchanged — SSE is additive. Pure event-builder helpers (`initial_event_from_job`, `event_from_phase`) stay testable in isolation; the long-lived forwarding loop in `build_phase_stream` is annotated `#[cfg_attr(coverage_nightly, coverage(off))]` because its `tokio::select!` arms depend on real-time broadcast deliveries the deterministic harness cannot fully cover — same exclusion pattern as `scanner_ws::run_subscription_loop`. Tests cover 404, 500-on-db-error, terminal-job-immediate-close (completed + failed), initial-state-for-non-terminal, end-to-end phase-transition fan-out, and the cancel-handler publishing path. The `async-stream` crate is promoted from a transitive to a direct dep so the router resolves it deterministically. * docs(jobs): document SSE push channel across SPEC/CONTRIBUTING/MIGRATION_RESEARCH/README/ROADMAP - SPEC.md §11.2.1: add the GET /api/jobs/:id/stream endpoint row plus the wire-shape event examples (phase / complete frames + failure / cancel variants). - CONTRIBUTING.md Job-API lifecycle: document SSE as the push-based channel alongside polling, including the broadcast-channel + per-stream Receiver pattern. - MIGRATION_RESEARCH.md §7.28: full architectural rationale for the SSE layer — why broadcast over watch, the 25 s heartbeat rationale, Cloudflare Tunnel constraints, fallback semantics, coverage scope. - README.md endpoints table: add the stream row. - ROADMAP.md Step 9: mark Phase 2 done with pointers to PR2. * test(jobs/sse): widen test timeouts to absorb shared-runner load The three SSE stream tests guard the request future and broadcast recv with 5s/5s/1s tokio timeouts. Under sequential nextest (test-threads=1) on the shared m3-ultra runner, the per-test DB setup (fresh schema + migrations) can stretch the wall time of the guarded section past 5s when CI load pressures the Postgres pool shared across PRs. Raise the request-future guards to 30s and the broadcast recv to 10s. The values still bound a stuck handler / lost event well inside a job's real lifetime budget; they just stop reporting a loaded runner as a code failure. No production timeouts change. * feat(openapi): annotate /api/jobs/{job_id}/stream and register in spec #161 added utoipa annotations to every other Job-API admit + poll handler; the SSE push channel introduced by this PR was the last hold-out because the JobNotifier + Sse<Stream> response shape did not exist when #161 landed. Wire-up: - `stream_job_handler` promoted to `pub(crate)` so the macro can reference it. - `#[utoipa::path]` documents the SSE contract: `text/event-stream` body on 200, `JobErrorResponse` JSON on 404/500. The 200 description names the two event types (`phase`, `complete`), the heartbeat-comment cadence, and the "stream closes after first `event: complete`" rule so a wallet author can implement the consumer side from the spec alone. - `crate::router::stream_job_handler` added to `ApiDoc::paths(...)` between `get_job_handler` and `receive_coin_handler` so the spec surfaces the new route alongside the poll endpoint. - `openapi_smoke::spec_lists_every_always_on_route` extended to require `/api/jobs/{job_id}/stream` — drift on the SSE contract now fails CI fast. No coverage impact: `stream_job_handler` was already exercised by the SSE integration tests added in commit `dcfc232`; the annotation adds zero new runtime code, only the compile-time `__path_*` generated by `utoipa::path`. * docs(jobs/sse): reviewer-loop polish — cleanup race, EventSource retry, connection cap Three doc-only clarifications surfaced by the post-rebase review on #163: - **router.rs `or_insert_with` comment.** The cleanup race between the dispatcher's terminal-publish and `notify_map.remove()` is safe — a fresh subscriber that opens in the gap reads the already-terminal row and emits `complete` from the initial-state snapshot, never depending on the orphaned broadcast subscriber. Documented inline so a future maintainer does not re-add a "fix" that breaks the property. - **MIGRATION_RESEARCH §7.28 — EventSource reconnect layering.** The wallet's built-in `EventSource` retry runs before the explicit poll fallback kicks in, so a `Lagged → end-of-stream` is observed by the wallet as a routine browser-side reconnect (3 s default backoff, cap), not a hard failure. Capturing the layering so the reviewer-asked "what does the wallet do on Lagged?" has a written answer. - **MIGRATION_RESEARCH §7.28 — concurrent-connection cap.** No per-node SSE concurrency limit today; the MVP wallet population fits in low single digits and the work-in-flight is bounded by the prove queue. The future "N>100 wallets self-host" case needs either a `Semaphore`-backed `max_sse_streams` or a reverse-proxy rule — documented as deferred so it does not vanish into the post-MVP backlog. Two reviewer-flagged items intentionally NOT addressed: - The `format!("{:?}", event)` Debug-substring checks on the pure helpers — axum 0.7's `Sse::Event::Debug` impl is stable and a shape change would fail all 10 tests in lockstep (a clean signal). The proposed swap to `parse_sse_events` would require routing each Event through axum's response-body machinery, which is more ceremony than the brittleness it removes. - A unit test that overflows `broadcast::channel(2)` to exercise the `Err(_) → end-of-stream` arm. The arm sits inside the `coverage(off)`-annotated `build_phase_stream` loop (real-wall- clock broadcast deliveries the deterministic harness cannot exhaustively cover, same pattern as `scanner_ws::run_subscription_ loop`). The behaviour is covered by the documented end-of-stream contract; the test would only re-prove the broadcast crate's semantics, not the handler's. * docs(jobs/sse): tighten EventSource attribution Two textual nits from reviewer round 2: - Replace "Mozilla / WHATWG spec default: 3 s, exponential cap" with the accurate "WHATWG defines a UA-implemented reconnection time settable via the `retry:` field; Firefox/Chrome ramp from ~3 s in practice". The previous wording mis-attributed exponential backoff to the spec when it's actually a UA implementation choice. - Drop the BitBox `useEventSource` analogy. The hook does not in fact wrap `EventSource` with a poll fallback (it just renders the stream), so the cross-reference was decorative; removing it keeps the section self-contained. * fix(jobs/sse): drop unreachable json_data fallback closures in SSE event builders `initial_event_from_job` and `event_from_phase` both built the SSE frame with Event::default() .event(name) .json_data(payload) .unwrap_or_else(|_| Event::default().event(name).data("{}")) The fallback closure existed defensively in case `json_data` failed. But `payload` in both call sites is a `serde_json::Value` built inline above with no custom `Serialize` impls — `Event::json_data`'s error path is only reachable for custom impls that serialise into non-UTF-8 bytes, which JSON's ASCII-superset output cannot violate. The closures were therefore structurally unreachable; llvm-cov counted each as a separate uncovered function plus uncovered line, failing the 100% line + function gate (2 missed functions + 2 missed lines, both at the closure sites). Switch both to `.expect("Event::json_data cannot fail for a freshly built serde_json::Value")`. The expect documents the invariant inline and a violation would surface as a fast-panic instead of silently emitting an empty `{}` body — same pattern as the `response_body.expect(...)` flatten that landed in #161 (`fix(jobs): drop unreachable response_body fallback closure in admit_and_enqueue`) and the `utoipa_swagger_ui::serve(...).expect` flatten in #157. Pinpointed in 30 seconds via the HTML coverage artifact added in the previous CI workflow tweak — the second time this diagnostic has paid for itself in two days. --------- Co-authored-by: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com>
* perf(tests): shared Postgres container + per-test schema (Issue #181 Opt B) (#182) * perf(tests): shared Postgres container + per-test schema (issue #181 Opt B) Replaces the per-test `Postgres::default().start()` model with a single shared Postgres container that every test process attaches to via testcontainers' `with_reuse(ReuseDirective::Always)` and a stable container name (`zkcoins-test-shared-pg`). Each test still gets a fully isolated state via a UUID-named schema with `search_path` pinned to it; migrations are run per-schema. The reuse flag is load-bearing: `cargo nextest` defaults to one process per test, so a process-local `OnceCell<Postgres>` does not actually share state across tests — it degrades to one container per test. Verified on a local M5 Max (OrbStack): 6 db-tests finish in 1.5 s with exactly 1 container running, vs. ~24 s with 6 containers under the old per-process model. CI gains three `docker rm -f zkcoins-test-shared-pg` cleanup steps (one per test job, always-on) so the shared container does not leak across PR runs on the self-hosted runner. Coverage gate's `--ignore-filename-regex` is extended to skip `test_db.rs` — the new `#[cfg(test)]`-only test-infra module would otherwise drag its Drop-future uncovered lines into the 100% gate. `db_tests::connect_and_migrate_creates_all_tables` is rewritten to route through the real `db::connect_and_migrate` (via the `?options=-c search_path=<schema>` URL trick) so the success-path of that function stays covered. Expected wall on the M3 Ultra runner per issue #181: 47 min → ~37 min at `--test-threads=1` (Optimisation A — flipping the test isolation to multi-thread — is a follow-up that depends on this landing first; see #181 Recommendation section). Test files migrated to the shared helper: db_tests, state_tests, r2_probe_tests, username_tests, main_tests, runtime_tests, router_tests (incl. the jobs_test_state factory from #161), job_store_tests, account_node_tests, audit_tests, publisher_tests. * test(db): include jobs table in connect_and_migrate assertion Migration 0014 (introduced by #161, async Job-API) adds the jobs table to the production schema. The rebase of #182 onto staging left the hard-coded expected-tables list in connect_and_migrate_creates_all_tables unchanged, so the assertion sees an extra row ("jobs") it does not expect and fails fast under nextest's default fail-fast mode — masking the rest of the suite. Adds "jobs" at its alphabetic position and bumps the migration range in the comment from 0001-0013 to 0001-0014. * perf(tests): enable parallel execution (--test-threads=8) (#181 Opt A) (#183) With per-test schema isolation + shared-container reuse from #182, the suite is parallel-safe. This PR: - Flips `--test-threads=1` to `--test-threads=8` across the 3 CI test jobs (db-tests, prover-tests, test-and-coverage) and the matching CONTRIBUTING.md references. - Adds a `fs2` cross-process file lock around `init_shared_pg` in test_db.rs. testcontainers 0.27 does NOT atomicise its attach-or-create path: 8 concurrent nextest processes all see "container not present", all POST /containers/create, 1 wins and 7 fail with Docker 409 Conflict. The lock serialises the attach-or-create call; the container creation cost (~3 s once) amortises across the whole test run. - runtime_tests.rs: env mutation consolidated behind a `OnceLock`-backed `ensure_test_env()` so concurrent callers do not race on process-wide env. `PROOFS_DIR` removed from env entirely and passed as a parameter on `start_rest_node` (main.rs reads the env at the binary edge). - router_tests.rs: 2 hard-coded `/tmp/zkcoins-*-proofs` paths replaced with `tempfile::tempdir().keep()` so each parallel test gets a unique ProofStore directory and `next_id` cannot race. Empirical on an Apple M5 Max workstation (OrbStack): a wide DB + state + router + username + audit subset of 146 tests passes under --test-threads=8 in 183 s wall (CPU 1325 %, exactly one shared postgres:17 container live during the run). Expected on the M3 Ultra runner per #181: 47 min (pre-Opt-B) -> ~44 min (after #182, measured) -> ~10-12 min (after this PR). --------- Co-authored-by: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com>
Promote: staging -> develop
) Remove the serializing needs: lint-and-build from db-tests, prover-tests, and test-and-coverage so the heavy M3 Ultra jobs start in parallel with lint-and-build instead of waiting behind it. The draft-skip + push behaviour previously inherited via that needs: is preserved by prepending the same (push || draft == false) guard to each heavy job's own if:. notify-failure keeps needs: [lint-and-build, test-and-coverage] — it is a fan-in failure aggregator, not work serialization. Trade-off: on a lint failure the M3 Ultra runner time is now spent regardless, in exchange for ~7-8 min faster feedback per heavy run. Co-authored-by: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com>
… tokens (#192) (#194) * feat: permissionless multi-asset — asset_id plumbing, circuit extension, API endpoints Add asset_id as a first-class concept throughout the protocol: - Types: AssetId type alias, NATIVE_ASSET_ID, calculate_asset_id(), asset_id field on Coin/CoinTemplate/Invoice/ProofData - Circuit: N_PROOF_DATA_PUBLIC_INPUTS 16→20, transition_asset_id extracted from PIs[16..20], source asset_id equality gate in the in-coin loop, out-coin identifier derivation extended to H(interim_asth || asset_id || slot_index) - Prover: asset_id parameter threaded through all prove_* functions - Node: Account.balances BTreeMap for per-asset tracking, asset CRUD in db.rs, POST /api/asset/create + GET /api/asset/list + GET /api/asset/info/:id endpoints, multi_asset capability flag - Schema: migration 0015_multi_asset.sql creates assets table (no name UNIQUE per issue #191 design) Closes #191 * fix: add multi_asset capability to api_remote integration test Thread the new multi_asset capability flag through the fetch_capabilities helper and the force-disable match arm. * fix: address logic-reviewer findings — wire asset_id through flows, add mixed-asset rejection - Account.balances is now updated in send_coins_inner after prove - flow.rs parses request.asset_id instead of hardcoding NATIVE_ASSET_ID - Off-circuit mixed-asset pre-check rejects mismatched asset_ids - multi_asset capability set to false until endpoints are wired - Stale aggregator PI count comment corrected (204 → 236) - Negative test: send_coins_rejects_mixed_asset_invoices * docs: correct stale aggregator PI layout comment (17→21 per slot) * fix: resolve clippy type_complexity in asset DB queries Use sqlx::FromRow derive on AssetRow instead of raw tuple decoding. * fix: add assets table to connect_and_migrate_creates_all_tables assertion * test: add same_name_different_creator negative test (issue #191) * refactor: defer asset-registration layer; keep additive circuit plumbing The 100% line+function coverage gate (node package) failed because the asset-registration surface added earlier had no production callers: the create/list/info handlers were 501/empty/404 stubs that never reached the DB CRUD, and get_asset_balances / Account.balances were write-only. Covering dead code (or shipping unwired endpoints) is the wrong fix. Multi-asset is an ADDITIVE extension over the existing off-circuit mint path, not a new public CRUD API. Per the node trust model send/receive stay trustless and asset support rides the same mint/send transition, so no asset-registry endpoint is required for the MVP. Removed (deferred to a follow-up built lockstep with the wallet app): - /api/asset/{create,list,info} handlers + routes - CreateAssetRequest / AssetResponse / AssetListResponse DTOs - db::insert_asset / get_asset / list_assets + AssetRow - migration 0015_multi_asset.sql (assets table) - AssetBalance + BalanceResponse.balances, Account.balances, get_asset_balances - openapi component registrations for the above Kept (additive, exercised end-to-end): - asset_id on Coin / Invoice / CoinTemplate / ProofData - calculate_asset_id / NATIVE_ASSET_ID / ASSET_GENESIS_DOMAIN_TAG - mixed-asset rejection in send_coins_inner (both in-coin branches) - MintRequest.asset_id / SendCoinRequest.asset_id wired through mint/send Hardening (no silent fallbacks): asset_id hex parsing in mint_flow / send_flow no longer defaults a present-but-malformed value to native. An absent field selects native; a present invalid or wrong-length value is a hard 422. Adds parse_optional_asset_id. Tests: add send_coins_rejects_queued_coin_with_foreign_asset (covers the coin_queue branch of the asset guard); fix warmup_prover formatting so the prove_initial `?` stays line-covered; restore connect_and_migrate expected-table list to the post-0014 schema. * test: drop assets table from connect_and_migrate assertion Reconciles the merged-in 4b8d907 (which added "assets" to the expected schema for migration 0015) with the registration-layer deferral: 0015 is removed, so the assets table is no longer created. Restore the expected list + comment to the post-0014 schema. * docs: make coverage gate + api_remote mandatory local pre-push gates Both jobs that most often go red after a push are now reproducible locally before pushing, turning a ~13 min red-CI round-trip into a local check: 1. Coverage gate — `cargo llvm-cov nextest ... --fail-under-lines 100 --fail-under-functions 100`, with the `--ignore-filename-regex` copied verbatim from .github/workflows/ci.yaml (node package, lines + functions). Verified locally: 442 tests, 100% lines + 100% functions. 2. api_remote (47 tests) against a local node pointed at public Mutinynet with an on-chain-funded publisher. Verified locally: 47/47. 39 are funding-free contract checks; the 8 mint/send/commit roundtrips broadcast real Taproot inscriptions and need the funded publisher (else "Failed to broadcast mint inscription on-chain"). Documents the env setup (~/.config/zkcoins/mutinynet.env) and the faucet reality (faucet.mutinynet.com is an L402 Lightning paywall, not a simple address faucet — fund the publisher P2TR address out-of-band). Co-authored-by: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com>
Promote: staging -> develop
Promote: staging -> develop
DEV's mint prover started failing 100% with "prove failed" on 2026-06-05 with no deploy and an unchanged circuit_digest: persisted account proofs stopped recursing through the live circuit (the constraint-only / digest-unchanged staleness class that migration 0015 documents as detectable only by the canary, which the steady-state self-heal Keep-path does not run). This migration is the recovery for the already-stale state. Wipes the same proof-dependent table set as db::reset_proof_dependent_state_tx — accounts, smt_state, mmr_state, mmr_root_index, latest_block — plus the circuit_digest_meta singleton. Clearing the digest row (rather than rewriting it; SQL cannot compute the live circuit digest) puts the DB in the fresh-genesis shape the boot path already handles: no persisted digest -> canary on the now- empty accounts -> NoSample -> Baseline records the live digest. No new code path, reuses the integration-tested self_heal flow. usernames / append-only history / jobs / coin_proof_store are preserved exactly as the existing reset does. On-disk proof files are left as inert orphans (ProofStore::new resumes next_id at max_id+1 so ids never collide; the Jobs-API no longer writes the file store). Closed test env, no data to preserve, PRD genesis wipe explicitly authorized (CONTRIBUTING "Closed test environment"). sqlx applies it once per database: develop -> DEV, main -> PRD. Validated against postgres:17: full 0001..0016 chain applies clean, the six tables empty, usernames/history intact, re-apply is a no-op.
TaprootFreak
marked this pull request as ready for review
June 5, 2026 15:23
TaprootFreak
added
the
ci:full
TaprootFreak
added a commit
that referenced
this pull request
Jun 5, 2026
* fix(db): reset proof-dependent state to genesis (DEV + PRD prover recovery) (#208) * Promote: staging -> develop (#185) * perf(tests): shared Postgres container + per-test schema (Issue #181 Opt B) (#182) * perf(tests): shared Postgres container + per-test schema (issue #181 Opt B) Replaces the per-test `Postgres::default().start()` model with a single shared Postgres container that every test process attaches to via testcontainers' `with_reuse(ReuseDirective::Always)` and a stable container name (`zkcoins-test-shared-pg`). Each test still gets a fully isolated state via a UUID-named schema with `search_path` pinned to it; migrations are run per-schema. The reuse flag is load-bearing: `cargo nextest` defaults to one process per test, so a process-local `OnceCell<Postgres>` does not actually share state across tests — it degrades to one container per test. Verified on a local M5 Max (OrbStack): 6 db-tests finish in 1.5 s with exactly 1 container running, vs. ~24 s with 6 containers under the old per-process model. CI gains three `docker rm -f zkcoins-test-shared-pg` cleanup steps (one per test job, always-on) so the shared container does not leak across PR runs on the self-hosted runner. Coverage gate's `--ignore-filename-regex` is extended to skip `test_db.rs` — the new `#[cfg(test)]`-only test-infra module would otherwise drag its Drop-future uncovered lines into the 100% gate. `db_tests::connect_and_migrate_creates_all_tables` is rewritten to route through the real `db::connect_and_migrate` (via the `?options=-c search_path=<schema>` URL trick) so the success-path of that function stays covered. Expected wall on the M3 Ultra runner per issue #181: 47 min → ~37 min at `--test-threads=1` (Optimisation A — flipping the test isolation to multi-thread — is a follow-up that depends on this landing first; see #181 Recommendation section). Test files migrated to the shared helper: db_tests, state_tests, r2_probe_tests, username_tests, main_tests, runtime_tests, router_tests (incl. the jobs_test_state factory from #161), job_store_tests, account_node_tests, audit_tests, publisher_tests. * test(db): include jobs table in connect_and_migrate assertion Migration 0014 (introduced by #161, async Job-API) adds the jobs table to the production schema. The rebase of #182 onto staging left the hard-coded expected-tables list in connect_and_migrate_creates_all_tables unchanged, so the assertion sees an extra row ("jobs") it does not expect and fails fast under nextest's default fail-fast mode — masking the rest of the suite. Adds "jobs" at its alphabetic position and bumps the migration range in the comment from 0001-0013 to 0001-0014. * perf(tests): enable parallel execution (--test-threads=8) (#181 Opt A) (#183) With per-test schema isolation + shared-container reuse from #182, the suite is parallel-safe. This PR: - Flips `--test-threads=1` to `--test-threads=8` across the 3 CI test jobs (db-tests, prover-tests, test-and-coverage) and the matching CONTRIBUTING.md references. - Adds a `fs2` cross-process file lock around `init_shared_pg` in test_db.rs. testcontainers 0.27 does NOT atomicise its attach-or-create path: 8 concurrent nextest processes all see "container not present", all POST /containers/create, 1 wins and 7 fail with Docker 409 Conflict. The lock serialises the attach-or-create call; the container creation cost (~3 s once) amortises across the whole test run. - runtime_tests.rs: env mutation consolidated behind a `OnceLock`-backed `ensure_test_env()` so concurrent callers do not race on process-wide env. `PROOFS_DIR` removed from env entirely and passed as a parameter on `start_rest_node` (main.rs reads the env at the binary edge). - router_tests.rs: 2 hard-coded `/tmp/zkcoins-*-proofs` paths replaced with `tempfile::tempdir().keep()` so each parallel test gets a unique ProofStore directory and `next_id` cannot race. Empirical on an Apple M5 Max workstation (OrbStack): a wide DB + state + router + username + audit subset of 146 tests passes under --test-threads=8 in 183 s wall (CPU 1325 %, exactly one shared postgres:17 container live during the run). Expected on the M3 Ultra runner per #181: 47 min (pre-Opt-B) -> ~44 min (after #182, measured) -> ~10-12 min (after this PR). --------- Co-authored-by: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com> * fix(db): reset proof-dependent state to genesis (DEV + PRD) DEV's mint prover started failing 100% with "prove failed" on 2026-06-05 with no deploy and an unchanged circuit_digest: persisted account proofs stopped recursing through the live circuit (the constraint-only / digest-unchanged staleness class that migration 0015 documents as detectable only by the canary, which the steady-state self-heal Keep-path does not run). This migration is the recovery for the already-stale state. Wipes the same proof-dependent table set as db::reset_proof_dependent_state_tx — accounts, smt_state, mmr_state, mmr_root_index, latest_block — plus the circuit_digest_meta singleton. Clearing the digest row (rather than rewriting it; SQL cannot compute the live circuit digest) puts the DB in the fresh-genesis shape the boot path already handles: no persisted digest -> canary on the now- empty accounts -> NoSample -> Baseline records the live digest. No new code path, reuses the integration-tested self_heal flow. usernames / append-only history / jobs / coin_proof_store are preserved exactly as the existing reset does. On-disk proof files are left as inert orphans (ProofStore::new resumes next_id at max_id+1 so ids never collide; the Jobs-API no longer writes the file store). Closed test env, no data to preserve, PRD genesis wipe explicitly authorized (CONTRIBUTING "Closed test environment"). sqlx applies it once per database: develop -> DEV, main -> PRD. Validated against postgres:17: full 0001..0016 chain applies clean, the six tables empty, usernames/history intact, re-apply is a no-op. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(prover): detect systemic prove failures — /health/ready signal + boot self-heal arming (#209) * Promote: staging -> develop (#185) * perf(tests): shared Postgres container + per-test schema (Issue #181 Opt B) (#182) * perf(tests): shared Postgres container + per-test schema (issue #181 Opt B) Replaces the per-test `Postgres::default().start()` model with a single shared Postgres container that every test process attaches to via testcontainers' `with_reuse(ReuseDirective::Always)` and a stable container name (`zkcoins-test-shared-pg`). Each test still gets a fully isolated state via a UUID-named schema with `search_path` pinned to it; migrations are run per-schema. The reuse flag is load-bearing: `cargo nextest` defaults to one process per test, so a process-local `OnceCell<Postgres>` does not actually share state across tests — it degrades to one container per test. Verified on a local M5 Max (OrbStack): 6 db-tests finish in 1.5 s with exactly 1 container running, vs. ~24 s with 6 containers under the old per-process model. CI gains three `docker rm -f zkcoins-test-shared-pg` cleanup steps (one per test job, always-on) so the shared container does not leak across PR runs on the self-hosted runner. Coverage gate's `--ignore-filename-regex` is extended to skip `test_db.rs` — the new `#[cfg(test)]`-only test-infra module would otherwise drag its Drop-future uncovered lines into the 100% gate. `db_tests::connect_and_migrate_creates_all_tables` is rewritten to route through the real `db::connect_and_migrate` (via the `?options=-c search_path=<schema>` URL trick) so the success-path of that function stays covered. Expected wall on the M3 Ultra runner per issue #181: 47 min → ~37 min at `--test-threads=1` (Optimisation A — flipping the test isolation to multi-thread — is a follow-up that depends on this landing first; see #181 Recommendation section). Test files migrated to the shared helper: db_tests, state_tests, r2_probe_tests, username_tests, main_tests, runtime_tests, router_tests (incl. the jobs_test_state factory from #161), job_store_tests, account_node_tests, audit_tests, publisher_tests. * test(db): include jobs table in connect_and_migrate assertion Migration 0014 (introduced by #161, async Job-API) adds the jobs table to the production schema. The rebase of #182 onto staging left the hard-coded expected-tables list in connect_and_migrate_creates_all_tables unchanged, so the assertion sees an extra row ("jobs") it does not expect and fails fast under nextest's default fail-fast mode — masking the rest of the suite. Adds "jobs" at its alphabetic position and bumps the migration range in the comment from 0001-0013 to 0001-0014. * perf(tests): enable parallel execution (--test-threads=8) (#181 Opt A) (#183) With per-test schema isolation + shared-container reuse from #182, the suite is parallel-safe. This PR: - Flips `--test-threads=1` to `--test-threads=8` across the 3 CI test jobs (db-tests, prover-tests, test-and-coverage) and the matching CONTRIBUTING.md references. - Adds a `fs2` cross-process file lock around `init_shared_pg` in test_db.rs. testcontainers 0.27 does NOT atomicise its attach-or-create path: 8 concurrent nextest processes all see "container not present", all POST /containers/create, 1 wins and 7 fail with Docker 409 Conflict. The lock serialises the attach-or-create call; the container creation cost (~3 s once) amortises across the whole test run. - runtime_tests.rs: env mutation consolidated behind a `OnceLock`-backed `ensure_test_env()` so concurrent callers do not race on process-wide env. `PROOFS_DIR` removed from env entirely and passed as a parameter on `start_rest_node` (main.rs reads the env at the binary edge). - router_tests.rs: 2 hard-coded `/tmp/zkcoins-*-proofs` paths replaced with `tempfile::tempdir().keep()` so each parallel test gets a unique ProofStore directory and `next_id` cannot race. Empirical on an Apple M5 Max workstation (OrbStack): a wide DB + state + router + username + audit subset of 146 tests passes under --test-threads=8 in 183 s wall (CPU 1325 %, exactly one shared postgres:17 container live during the run). Expected on the M3 Ultra runner per #181: 47 min (pre-Opt-B) -> ~44 min (after #182, measured) -> ~10-12 min (after this PR). --------- Co-authored-by: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com> * fix(prover): detect systemic prove failures — health signal + self-heal arming The 2026-06-05 DEV outage exposed two gaps around the digest-unchanged proof-staleness class that migration 0015 documents: 1. /health/ready lied. Its prover tag only reflected the one-shot boot warmup flag, so a node failing 100% of mint jobs with "prove failed" kept reporting prover: ready for ~100 minutes — invisible to the deploy smoke-test, Kuma, and any orchestration keyed on readiness. 2. The boot self-heal never re-checks in steady state. reset_decision consults the canary recursion only on the no-persisted-digest adoption branch; with a persisted digest equal to the live one it takes the Keep fast path. Constraint-only circuit changes (and any other event that stops persisted proofs from recursing while the digest stays byte-identical) therefore brick the node permanently — no restart heals it. New prover_health module: the job dispatcher counts CONSECUTIVE "prove failed" outcomes (the collapsed message is matched exactly, so request-level errors never move the streak; any successful prove resets it). At PROVE_FAILURE_THRESHOLD consecutive failures: * /health/ready reports prover: failing + 503 for the duration of the streak (gap 1) — the outage is now visible and gates traffic. * the dispatcher clears the persisted circuit digest via the new db::clear_circuit_digest (gap 2). This only ARMS the boot self-heal: the next restart finds no persisted digest, runs the canary recursion, and resets to genesis IFF the canary confirms the persisted proofs are stale — Compatible/NoSample just re-record the baseline, so a transient prover blip that is over by the restart causes no reset and no data loss. The destructive reset stays gated behind the authoritative canary; nothing is wiped at runtime. The steady-state boot keeps its O(1) digest comparison (the ~5 s canary still never runs on a healthy boot); the arming path is the only way a matching-digest boot reaches the canary. Coverage: prover_health is unit-tested exhaustively (threshold boundary, one-shot arming, streak reset); clear_circuit_digest gets a testcontainer round-trip incl. idempotent re-clear; the new ready-handler branch is driven by a prover-failing readiness test (503 + prover: failing). job_dispatcher wiring sits in the coverage-exempt dispatcher. fmt + the CI clippy commands (-D warnings, MVP + all-features) are clean locally; check --tests green. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: TaprootFreak <142087526+TaprootFreak@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Genesis-reset of all proof-dependent state — DEV and PRD recovery
Incident
Since 2026-06-05 ~11:26 UTC the DEV node fails 100 % of mint jobs with
prove failed(admit OK →proving→failed; reproduced 8/8 via directPOST /api/jobs/mintprobes and continuously since). No node deploy or commit happened — the circuit binary and itscircuit_digestare unchanged; identical mints succeeded minutes before. This is the digest-UNCHANGED staleness class that migration 0015 documents: persistedaccount.proofblobs stop recursing through the live circuit whileProver::verifyand the digest comparison still pass. The boot self-heal cannot catch it in steady state (Keepfast-path never runs the canary — that detection gap gets a separate code PR)./health/readykeeps reportingprover: readythroughout (it only reflects the boot-warmup flag) — also addressed in the follow-up PR.What this migration does
Wipes the proof-dependent state to genesis, once per environment, via the normal deploy:
db::reset_proof_dependent_state_tx)accounts,smt_state,mmr_state,mmr_root_index,latest_block,circuit_digest_metausernames,account_history,state_update_log,request_log,jobs,coin_proof_store,pending_inscriptionsClearing
circuit_digest_meta(instead of rewriting it — SQL can't compute the live digest) lands the DB in the fresh-genesis shape the boot path already handles: no persisted digest → canary →NoSample(emptyaccounts) →Baselinerecords the live digest. No new code path; this reuses the integration-testedself_healflow. On-disk proof files stay as inert orphans (ProofStore::newresumesnext_idatmax_id + 1, no id collision; the Jobs-API no longer writes the file store).Authorization & scope
Closed test environment (CONTRIBUTING § "Closed test environment"); the operator has explicitly confirmed no data needs preserving and authorized the PRD genesis wipe. sqlx applies the migration once per database (
_sqlx_migrations):Precedent: migrations 0010/0011/0012 (wipe-and-replay in the closed test env).
Validation
Against a real
postgres:17: the full0001..0016chain applies cleanly; after 0016 the six tables are empty, preserved tables intact; re-applying the SQL is a no-op (DELETEs are naturally idempotent; sqlx runs it once anyway). SQL-only change — no Rust code, no coverage-gate impact; existing tests run migrations against fresh (empty) DBs, where 0016 is a no-op.Sequencing for the operator
/health/ready.