Skip to content

[Dependency/Sync] Publish maintenance-efficiency metrics and SLOs #2883

Description

@stranske

Why

The maintenance system currently exposes individual workflow and campaign state, but it does not publish a durable efficiency view. Without PR amplification, stale/replacement rate, CI runs per delivered source change, and agent-exception episodes, a change can look operationally successful while total maintenance work grows.

The recent audit required a one-off reconstruction of those measures. This is a latent control defect: future Renovate, sync, or controller changes can regress productivity without a named test or SLO detecting the regression.

Scope

Add a repository-owned, reproducible maintenance-efficiency report and initially advisory SLO evaluation for dependency-bot, sync-generated, and dev-tool-sync work. Report stranske/Collab-Admin separately and exclude it from primary fleet SLOs because it regenerates from the admin surface.

Non-goals

  • Do not use PR count alone as a quality target.
  • Do not penalize explicit security updates for bypassing routine batching windows.
  • Do not turn the first baseline run into a merge-blocking historical-data gate.
  • Do not count a dashboard workflow scaffold as delivered until fixture-backed calculations and a deliberate threshold breach are verified.

Implementation tasks

  • Add scripts/dependency_sync_efficiency_metrics.py to classify PRs by dependency-bot, sync-generated, dev-tool-sync, traditional, and Collab-Admin-excluded lanes.
  • Compute weekly created/merged/closed/stale counts, source-change-to-consumer-PR amplification, replacement PRs per repo/batch, Actions runs/jobs per delivered source change, time-to-terminal-disposition, and distinct agent-exception fingerprints.
  • Add fixture data and unit tests covering Renovate authors/branches, Dependabot compatibility, sync titles, dev-tool wave IDs, supersession, telemetry-only failures, and Collab-Admin exclusion.
  • Add .github/workflows/health-83-dependency-sync-efficiency.yml to publish JSON and Markdown artifacts on a weekly schedule and by manual dispatch.
  • Update a durable tracking issue only when the SLO state or material evidence changes; do not append an unchanged weekly comment.
  • Establish an advisory baseline and documented targets: no more than 40 generated PRs/week, under 5% stale/replacement PRs, zero avoidable replacement PRs per repo/batch, and no more than 5 distinct agent-exception episodes/week.
  • Keep targets advisory for four complete weeks, then require an explicit repository decision before any threshold becomes blocking.
  • Document calculation definitions, exclusions, data-retention limitations, and security-window handling in docs/ops/DEPENDENCY_SYNC_EFFICIENCY.md and link it from durable-tracker guidance.

Acceptance criteria

  • Unit tests produce stable, auditable metric values from a fixed GitHub event/PR fixture and identify every classified PR by lane.
  • The report includes numerator, denominator, period, excluded Collab-Admin figures, and source evidence for every rate or ratio.
  • Two identical scheduled inputs produce no duplicate durable-tracker comment or agent handoff.
  • Security-bypass PRs remain visible but are separated from routine cadence compliance.
  • A manual workflow run uploads both machine-readable JSON and human-readable Markdown and records any incomplete GitHub history explicitly.
  • Deliberate-break test: change a fixture to exceed stale/replacement and exception thresholds, verify the evaluator reports named breaches, then revert and verify the baseline passes.
  • Run python scripts/dev_check.py --action test and workflow validation successfully.

Delivery notes

The initial report is diagnostic, not punitive. Its purpose is to make maintenance cost visible enough that future changes can be judged by delivered updates per PR, CI run, and agent episode rather than by activity volume.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions