henyey mainnet daily — 2026-05-18

[tomerweller]

Validator

• Build: be066514 (uptime 11h 55m; 1 deploy in last 24h after 2 quarantined builds; ~5m avg build)
• State: 🟢 Validating (agree=21/missing=0, lag_ms=445)
• SCP propagation (first_to_self_externalize_seconds): 51ms
• SCP slot-cycle (externalized_seconds): 4.77s
• Memory: RSS 14.3G, frag 24.5% (heap_components_mb=5205 stable since unwedge replay)
• Disk: data 87% (1.4T of 1.7T), session 2.3G, mainnet 26G

Deploys (1 effective)

• be066514 Convert INV-H2 panic to corrective branch in assert_lcl_consistency (INV-H2 panic at herder.rs:940 during catchup replay (separate from #2789) #2791) (Convert INV-H2 panic to corrective branch in assert_lcl_consistency (#2791) #2792)
  • Files touched: crates/app, crates/herder
  • Watch sentinel: fired and recovered (7 wedge events post-deploy, all self-resolved via Validating → Catching Up → Validating with WARN + tracking advance; INV-H2 corrective WARN count steady at 5 from the initial replay burst)

Note: also landed but rolled back (quarantined) before deploy:

• 52ac1665 Tighten henyey-db API-narrowing doc (Tighten henyey-db API-narrowing doc to acknowledge rusqlite exposure #2786) — quarantined after INV-H2 crash loop
• 0df88603 Narrow metrics: recovery-stalled burst (delta=240) + 155k overlay backpressure on post-deploy catchup-behind #2713 hard-reset suppression to at-or-near-tip (metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac1665 at 17h uptime #2789) (Narrow #2713 hard-reset suppression to at-or-near-tip (#2789) #2790) — quarantined after second INV-H2 crash on same line 940

Incidents (1 major resolved)

• 🟢 INV-H2 panic crash loop → fix shipped
  • Window: 2026-05-17T23:18Z → 2026-05-18T01:39Z (2h21m total downtime)
  • Symptom: validator panicked at crates/herder/src/herder.rs:940 with LCL ≥ tracking_consensus_slot during fast-tracking catchup burst-close
  • Root cause: prior 5a90e50 fix addressed the same panic family at line 894 but missed a sibling assertion at line 940; assertion fired during any catchup-replay where the ledger-manager closed ledgers faster than the herder updated tracking_slot
  • Resolved by: be066514 (Convert INV-H2 panic to corrective branch in assert_lcl_consistency (#2791) #2792 — converted panic to WARN + metric + idempotent self-correct that advances tracking to LCL+1)
  • Three crashes in sequence: 23:22 on 52ac166, 23:25 on 52ac166 restart, 00:18 on 0df8860 (metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac1665 at 17h uptime #2789-fix-only). Each filed a separate quarantine; only be06651 (which converts panic→WARN) is running

Issues activity

Filed today (4):

• 🔴 metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac1665 at 17h uptime #2789 metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac166 at 17h uptime — resolved by 0df88603 (Narrow #2713 hard-reset suppression to at-or-near-tip (#2789) #2790)
• 🔴 INV-H2 panic at herder.rs:940 during catchup replay (separate from #2789) #2791 INV-H2 panic at herder.rs:940 during catchup replay (separate from metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac1665 at 17h uptime #2789) — resolved by be066514 (Convert INV-H2 panic to corrective branch in assert_lcl_consistency (#2791) #2792)
• ⚪ /review-pr SKILL.md PR-detection jq query has bug — closedByPullRequestsReferences has no .state field #2793 /review-pr SKILL.md PR-detection jq query has bug — resolved by b3e4ec83
• ⚪ Diagnose missing PROJECT_BOARD_TOKEN with structured error #2798/Make SCP tx-set purge transactional #2799 (auto-filed by triage pipeline for related sub-issues)

Closed today (9):

• /review-pr SKILL.md PR-detection jq query has bug — closedByPullRequestsReferences has no .state field #2793 b3e4ec83 /review-pr PR-detection bug
• INV-H2 panic at herder.rs:940 during catchup replay (separate from #2789) #2791 be066514 INV-H2 panic (catchup replay)
• metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac1665 at 17h uptime #2789 0df88603 mid-run recovery cycle
• /review-pr Step 2 not reliably followed by agent — paraphrases rule, ignores SKILL.md update #2787 039c7b5a /review-pr Step 2 paraphrasing
• Follow-up: zsh regression test for monitor-decisions.sh suffix-strip #2782 c33f3b19 zsh regression test for monitor-decisions
• CI: Archive Done Project Items workflow fails — PROJECT_BOARD_TOKEN secret missing or unscoped #2777 74f4b1a2 Archive Done Project Items PROJECT_BOARD_TOKEN diagnose
• HERDER: Make ScpPersistenceManager::purge_unreferenced_tx_sets transactional #2770 7380c388 ScpPersistenceManager purge transactional
• HERDER: Wire ScpPersistenceManager::persist_scp_state into envelope-emit hot path #2768 bb431484 ScpPersistenceManager persist wired
• Audit and fix (Owner, TempDir) drop-order pattern across test helpers #2761 22c5a88d (Owner, TempDir) drop-order audit

Still open (3 long-running):

• enhancement HERDER: Wire ScpPersistenceManager::restore_scp_state into startup #2769 HERDER: Wire ScpPersistenceManager::restore_scp_state into startup
• enhancement HERDER: Event-driven consensus trigger scheduling (Phase 2) #2702 HERDER: Event-driven consensus trigger scheduling (Phase 2)
• ⚪ Spec adherence audit: suite overview (Protocol 26) #2660 Spec adherence audit: suite overview (Protocol 26)

Watch items

• Post-fix wedge cycles: 4 wedges observed in 12h since be06651 deploy (Add quickstart CI workflow for integration testing #5 02:07, Eliminate offer_locations reverse index from OfferIndex #6 03:15, setup a recurring verify execution automation #7 07:26, setup a recurring mainnet monitor automation #8 11:40), each ~5-7 min, identical shape: LCL stuck while next_checkpoint not yet published by archive (5 min wait by design). metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac1665 at 17h uptime #2789 fix narrows hard-reset suppression correctly but spawn_catchup then bounces off "archive hasn't published checkpoint yet" gate. Validator survives every time (INV-H2 panic at herder.rs:940 during catchup replay (separate from #2789) #2791 fix prevents the resulting INV-H2 crash). Inter-wedge interval has lengthened from 1h7m → 4h11m → 4h13m. Flagged on metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac1665 at 17h uptime #2789 as possible follow-up.
• INV-H2 corrective WARN count: 5 fired during initial catchup-replay at 01:32:20-01:32:24, then zero since (15h). Confined to fast-tracking burst path; not a steady-state concern.
• Two builds quarantined: 52ac1665, 0df88603 in /home/tomer/data/deploy_quarantine.txt. Monitor will not auto-redeploy them.

Tick aggregates (last 24h)

• Cron: 6c74937d (daily-summary at 13:07 UTC) — 1 expected, 0 fired today (manual post; cron fires consistently 10-30 min late, see open question below). The 20m monitor-tick is run by /loop self-pacing, not cron.
• Outcomes: 36 OK / 4 ACTION / 8 WARNING / 7 OFFLINE (55 ticks)
• Alarm-firings: 5 ticks with ≥1 firing alarm (11 alarm-firings total across the window — concentrated during 3 successive INV-H2 crashes and 4 wedge recurrences)
• metrics_ratio: 0 breaches, 20 skipped (ledger-age-high gating during wedges)
• recovery_stalled: 3 non-zero-delta ticks (all during wedges; deltas 234, 291, 358)
• Self-reflection events: 0
• Skill commits today: none touching .claude/skills/ directly today; b3e4ec83/039c7b5a/57438a74/c33f3b19 are skill or script changes already listed in deploys

Open questions

1. Daily-summary cron reliability: 6c74937d ("0 13 * * *") has fired 10-28 min late every day this week (13:17 on May 17, 13:32 on May 16, 13:35 on May 15) and didn't fire by 13:22 today (this post is manual). The current cron is "session-only" so it only persists as long as the host Claude session does. Migrate to GH Actions cron or accept the lateness pattern?

2. Wedge follow-up: 4 wedges/12h on be066514 (metrics: mid-run recovery cycle (lost-sync + hard-reset + recovery-stalled) on 52ac1665 at 17h uptime #2789 fix landed). Each costs ~5min of RPC unhealth + Catching Up state. The fix narrowed one suppression but spawn_catchup still blocks on "archive hasn't published checkpoint yet". Worth a follow-up issue to either (a) skip that gate when peer_gap > N, or (b) accept the 5-min wedge as a "wait for next checkpoint" by design?