Skip to content

fix(metrics): count orphaned-workflow governance incidents in the window rollup#17

Merged
sturlese merged 1 commit into
mainfrom
fix/bughunt-orphan-governance-undercount
Jul 12, 2026
Merged

fix(metrics): count orphaned-workflow governance incidents in the window rollup#17
sturlese merged 1 commit into
mainfrom
fix/bughunt-orphan-governance-undercount

Conversation

@sturlese

Copy link
Copy Markdown
Owner

Bug

The window governance counters and the all-time counters disagreed for a workflow that had been deleted/renamed in config while its historical runs remained in the store. For an in-window budget block on such an orphaned workflow_id:

  • governance.blocked_budget (window) = 0
  • governance.blocked_budget_all = 1

The _all counters are documented as a superset ("since the beginning, so old incidents stay visible"), so the window counter reading 0 while _all reads 1 for an incident that is in the window is internally inconsistent and understates the window's governance posture.

Root cause

The window counters (blocked_budget / blocked_policy / failed) are incremented inside the for workflow_id, workflow in org.workflows.items() loop, so a run whose workflow_id isn't a current workflow is never seen. The _all counters are tallied in a per-run pass with no workflow lookup, so they include orphaned runs.

Fix

After the per-workflow loop, a small pass over the window runs whose workflow_id is absent from the config classifies their blocks/failures the same way (budget vs policy, and failed). Runs of known workflows were already tallied in the loop above, so the pass is disjoint — no double counting.

Because every run in a normal org belongs to a current workflow, this changes nothing for existing orgs (verified: the full suite and the deterministic demo are unaffected); it only surfaces incidents that were previously dropped.

Test

test_window_governance_counts_orphaned_workflow_incidents: seeds a budget block, a policy block, and a failed run for a deleted-wf in the window, then asserts each window counter now equals its all-time counterpart. Fails on main (window=0, all=1), passes with the fix.

Full suite: 177 passed, 92.74% coverage (gate 85%); ruff clean; offline flightdeck demo + audit verify intact.

Found by an autonomous bughunt iteration.

…dow rollup

The window governance counters (blocked_budget, blocked_policy, failed)
were incremented inside the per-workflow loop, so a run whose workflow_id
is no longer in the config (a pilot deleted/renamed while its historical
runs remain) was never counted — while the all-time counters, tallied in a
per-run pass with no workflow lookup, DO count it. So for an in-window
incident on a since-deleted workflow the window counter read 0 while the
all-time counter read 1, breaking the documented "_all is a superset of the
window" relationship and understating the window's governance posture.

Add a small pass over the window runs for workflows absent from the config,
classifying their blocks/failures the same way. Runs of known workflows are
already tallied above, so the pass is disjoint — no double counting, and no
change for any org whose runs all belong to current workflows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sturlese

Copy link
Copy Markdown
Owner Author

bughunt gate — /code-review high ✅ clean

Reviewed across 8 angles — no findings. The new pass is disjoint from the per-workflow loop (known vs. unknown workflow_id — no double count) and purely additive (a no-op for any org whose runs all belong to current workflows, so existing numbers and the deterministic demo are unchanged — 177-pass suite confirms). The budget/policy/failed classification mirrors the two existing sites, restoring the documented _all ⊇ window superset relationship. The classification triplication matches the codebase's pre-existing pattern; extracting a helper would exceed this fix's scope. Regression test fails on main (window=0, all=1), passes with the fix. Full local gate: 177 passed, 92.74% coverage, ruff clean, demo intact. Full mode: watching CI to merge on green.

@sturlese
sturlese merged commit c4aa1e7 into main Jul 12, 2026
5 checks passed
@sturlese
sturlese deleted the fix/bughunt-orphan-governance-undercount branch July 12, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant