Skip to content

Monitor branch protection and open Dependabot alerts - #15

Merged
tschm merged 4 commits into
mainfrom
feat/protection-and-dependabot-metrics
Aug 30, 2026
Merged

Monitor branch protection and open Dependabot alerts#15
tschm merged 4 commits into
mainfrom
feat/protection-and-dependabot-metrics

Conversation

@tschm

@tschm tschm commented Aug 30, 2026

Copy link
Copy Markdown
Member

Adds two metric groups, both on the existing per-repo loop in github.py.

Branch protection

jq_branch_protected, jq_branch_required_reviews, jq_branch_allows_force_push.

Every other metric on this board reports what happened. This one reports what is allowed to happen — the thing you change once and it holds. On the current 25-repo fleet it already finds:

  • exactly one repo (Jebel-Quant/rhiza) protects its default branch
  • and that one still allows force pushes — a green tile on a branch anyone can rewrite

Dependabot

jq_dependabot_alerts_enabled, jq_dependabot_open_alerts{severity}.

Four repos in the fleet have alerts switched off entirely. One (tschm/proximal) has an open low-severity alert that nothing on the board showed before.

The part worth reviewing

Both endpoints return 404 in two different situations, and conflating them makes the board lie in opposite directions:

Endpoint 404/403 can mean Handled as
branches/{b}/protection unprotected or token lacks admin metric absent, never 0
dependabot/alerts no open alerts or feature disabled separate _enabled gauge; counts absent when off

Exporting 0 for unknown protection would put every repo on the "unprotected" list on the strength of a permission gap — a finding the board invented. absent and 0 are distinguishable in PromQL; two zeros are not.

Likewise "nothing to fix" and "nobody is looking" must not render as the same green tile.

Severities are zero-filled so a repo that clears its criticals reports 0 rather than dropping out of the query and stranding its last value on the graph.

Cost

~2 extra calls per repo: a pass over 25 repos goes from ~450 to ~700 calls, 37s to 40s. That fits only because #14 moves JQ_GITHUB_INTERVAL to 600s — this should merge after #14, or the refresh goes further over an already-exceeded budget.

Testing

4 new tests covering both unknown-vs-zero distinctions; full suite of 48 passes. Verified against the live fleet — the numbers quoted above are from the running collector, not fixtures.

🤖 Generated with Claude Code

tschm and others added 2 commits August 30, 2026 13:37
Two calls per repo, both on the existing per-repo loop.

Branch protection is the lever the board was missing. Every other metric here
reports what happened; this one reports what is *allowed* to happen, which is
the thing you can change once and have hold. On the current fleet it already
finds that one repo protects its default branch, and that one still permits
force pushes - a green tile on a branch anyone can rewrite.

Both endpoints 404 in two different situations, and conflating them would
make the board lie in opposite directions:

  * protection 404s for an unprotected branch AND for a token without admin.
    Exporting 0 for both would invent findings out of a permission gap, so
    jq_branch_protected is absent rather than zero when GitHub will not say.
    Absent and 0 are distinguishable in PromQL; a zero is not.
  * the alerts endpoint 404s or 403s when the feature is off, which is
    indistinguishable from "no alerts open". "Nothing to fix" and "nobody is
    looking" are opposite facts, so jq_dependabot_alerts_enabled carries that
    separately and the counts are absent entirely when it is off. Four repos
    in the fleet turn out to have alerts disabled.

Severities are zero-filled, so a repo that clears its criticals reports 0
rather than dropping out of the query and stranding its last value on the
graph.

Cost is ~2 extra calls per repo, taking a pass over 25 repos from ~450 to
~700. That fits only because JQ_GITHUB_INTERVAL moved to 600s; at the old
300s this would have pushed an already-over-budget refresh further over.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two changes that belong together: the collector could not say "unprotected",
and the board had nowhere to say it.

branch_protection now reads the 404 body. GitHub answers an unprotected
branch with "Branch not protected" and a permission gap with something else,
so the two 404s are distinguishable after all. Collapsing them lost the
finding entirely: protection was exported only where it existed, and 22 of
25 repos simply had no series - indistinguishable from a fleet nobody had
looked at. Unknown is still absent rather than zero, so a permission gap
never becomes an accusation.

Four tiles in Fleet at a glance, each linked to a drill-down table:

  Default branches unprotected   22
  Protected but force-pushable    1
  Dependabot off                  4
  Open Dependabot alerts          1

"Protected but force-pushable" earns its own tile because it is the case a
protection tile hides: the one repo on the fleet that protects its default
branch also permits rewriting it, so counting protected repos alone would
report that as solved. "Dependabot off" sits beside the alert count for the
same reason - four repos contribute a truthful zero to a total that would
otherwise read as an all-clear.

The tiles use sum(x == bool 0), not count(). count() over a bool comparison
counts every series regardless of the comparison, which read 23 unprotected
of 23 and 25 Dependabot-off of 25 - the total dressed up as a finding, and
plausible enough to miss.

Deliberately not added to fleet-public.json. A world-readable list of which
repos are unprotected and which have open advisories is a target list.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tschm

tschm commented Aug 30, 2026

Copy link
Copy Markdown
Member Author

Pushed two more commits: the collector can now say unprotected, and the board shows it.

The metric was half-blind

branch_protection treated both 404s as unknown. But the bodies differ — GitHub answers an unprotected branch with "Branch not protected", and a permission gap with something else. Collapsing them meant protection was exported only where it existed: 22 of 25 repos had no series at all, which is indistinguishable from a fleet nobody had looked at.

Now unprotected is a fact (jq_branch_protected 0) and only genuine gaps stay absent. The two private repos are the only unknowns left — free-plan private repos cannot use protection at all.

Four tiles in Fleet at a glance

Tile Live value
Default branches unprotected 22
Protected but force-pushable 1
Dependabot off 4
Open Dependabot alerts 1

Each links to a drill-down table in the existing collapsed row.

Two of these exist to stop a tile lying by omission. Protected but force-pushable is the case a protection count hides — the single repo that protects main also allows rewriting it. Dependabot off sits beside the alert total because four repos contribute a truthful 0 to a number that would otherwise read as an all-clear.

A bug worth flagging

The tiles first used count(x == bool 0). A bool comparison yields 1/0 for every series, so count returns the series total — it reported 23 unprotected of 23, and 25 Dependabot-off of 25. Wrong, and plausible enough to ship. Now sum(x == bool 0), matching the idiom the existing tiles already use, and verified against live Prometheus.

Not on the public board

fleet-public.json is unchanged, deliberately. A world-readable list of which repos are unprotected and which have open advisories is a target list.

45 tests pass; check-dashboard.py validates both dashboards.

tschm and others added 2 commits August 30, 2026 14:12
Two of the four new tiles linked to a table that excluded, by construction,
exactly the repos the tile counted:

  Dependabot off -> the open-alerts table, which lists only repos where
  Dependabot is enabled. Clicking "4 repos are off" showed repos that are on.

  Protected but force-pushable -> the unprotected table, which matches
  jq_branch_protected == 0. A force-pushable branch is protected, so the one
  repo behind that tile could never appear.

Both were reachable in one click and answered a question with its own
opposite, which is worse than having no link: a drill-down is read as the
evidence for the number above it.

Adds the two missing tables and repoints the tiles. Each of the four now
matches its own count against live Prometheus - 22, 1, 4 and 0 rows for
tiles reading 22, 1, 4 and 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Dependabot commits left two files unformatted and never re-ran the
public dashboard generator, so CI's two guards - ruff format --check and
the "fleet-public.json is stale" diff - both fired.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tschm
tschm merged commit 36227d2 into main Aug 30, 2026
4 checks passed
@tschm
tschm deleted the feat/protection-and-dependabot-metrics branch August 30, 2026 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant