💡 Runner Image Migration Readiness & Fleet Testing Standard

[github-actions[bot]]

Summary

Establish an org-wide standard for proactively testing fleet CI workflows against upcoming GitHub-hosted runner images before GitHub's default cutover. Includes a preview-image test matrix, known-incompatibility registry, and fleet-wide impact assessment gate in the compliance audit.

Market Signal

GitHub is actively releasing new runner images: windows-2025-vs2026 (Windows Server 2025 + VS 2026) and macos-26-large are in public preview. The ubuntu-latest migration from 22.04 to 24.04 has already caused breakages across the ecosystem. GitHub explicitly recommends "early validation before default image migrations" in their February 2026 changelog. Runner image transitions are accelerating — the days of multi-year stability on a single image are over.

User Signal

Issue #867 is a live, open bug: "kcov not available on ubuntu-24.04 (ubuntu-latest) — coverage job breaks when bats tests exist." This breakage affects every repo in the fleet that has bats tests with coverage enabled. The fleet's standardized CI workflows (standards/workflows/) mean a single image incompatibility cascades to all repos simultaneously — a blast radius multiplier unique to fleet-managed CI.

Technical Opportunity

The standards-deploy driver (scripts/standards-deploy-driver.sh, merged in #854) already pushes workflow updates across the fleet. Adding a scheduled preview-image test job that runs the fleet's CI templates against upcoming images would catch breakages like #867 weeks before they hit. The compliance audit (scripts/compliance-audit.sh) can flag repos not testing against preview images. The existing ci.yml template's matrix strategy is the natural extension point.

Assessment

Dimension	Score	Rationale
Feasibility	high	Uses existing CI matrix features and GitHub's preview runner images. No new infrastructure required.
Impact	med	Prevents fleet-wide CI breakages from image migrations. Reduces reactive firefighting to proactive readiness.
Urgency	high	Active bug #867 from the ubuntu-24.04 migration. New image previews (windows-2025-vs2026, macos-26-large) are already available.

Adversarial Review

Strongest objection: This is just adding a matrix strategy to CI — not worth a full standard. Runner image changes are infrequent and the breakages are easy to fix after the fact.

Rebuttal: The fleet-wide impact makes this more than a CI tweak. When ubuntu-latest shifted, kcov broke across every repo with bats tests — a single point of failure multiplied by the entire fleet. A standard ensures: (1) preview images are tested before migration, (2) known incompatibilities are documented in a registry (e.g., "kcov unavailable on 24.04, use gcovr"), (3) fleet-wide impact assessment happens before GitHub's cutover, not after. The compliance audit already enforces CI standards; this extends it to image readiness.

Suggested Next Step

Fix the immediate kcov breakage (#867) with a gcovr or alternative solution. Then add a preview-image CI matrix entry to the fleet CI template that runs weekly against the next ubuntu/macOS image. Document known incompatibilities in a new standards/runner-image-readiness.md registry with columns: image, tool, status, workaround.