Skip to content

ci: harden scoped test selection before re-enabling — third distinct red-CI edge case this week #8194

Description

@JSONbored

Problem

Scoped test selection (--changed=origin/main on PRs) has produced three DISTINCT full-red CI modes in one week, each from a legitimate PR shape its design didn't anticipate:

  1. Zero test files matched (docs-only diff under a scoped path) — fixed by fix(ci): carry the no_tests_matched escape through every coverage/results upload step #8173's no_tests_matched guard after PR docs(discovery-plane): finalize operator guide now that the plane is shipped and live #8165 died on it.
  2. Re-shallowed merge-base (git fetch --depth=1 cutting origin/main's history) — silently selected zero files on every scoped PR; fixed 2026-07-21.
  3. Tests matched but exercised zero src/** files (test-only/docs-drift diffs, e.g. PR fix(docs): repair #8182 fallout — audit manifest, mangled runner YAML snippet, kill-switch route assertion #8192): vitest passes, --coverage.all=false writes an EMPTY lcov, and 'Verify coverage report exists' fails the green run. The no_tests_matched guard can't see this case — tests DID match.

In a one-shot gate culture any red check auto-closes contributor PRs, so each of these was a contributor-facing outage, not an inconvenience. Scoped selection is now DISABLED via the repo variable SCOPED_TEST_SELECTION_ENABLED=false (its own designed kill switch, set 2026-07-23): every PR runs the full sharded suite until this issue lands. Sharding itself has caused none of this and stays.

Requirements

⚠️ Success signal must come from the test run itself, not from coverage-file side effects. The verify step's job is 'the suite really ran and passed', and lcov emptiness is not evidence of that in scoped mode.

  • Rework 'Verify coverage report exists' for the scoped branch: treat vitest exit 0 + a written junit/blob report as the pass signal; an empty/missing lcov in scoped mode with a passing run is legitimate (record it as a step output so codecov upload is skipped the same way no_tests_matched already is).
  • Add the missing regression shapes to whatever harness guards ci.yml (test-only diff, packages-only diff, docs-only diff) so the next edge case is caught before it ships.
  • Only after both: re-enable the variable, in the same PR that documents the re-enable decision.

Links & Resources

.github/workflows/ci.yml (validate-tests), #8173, #8165, PR #8192 (case 3's victim), scripts/compute-test-shards.ts

Boundaries

Keep duration-aware sharding untouched. No coverage-threshold changes.

maintainer-only — CI trust infrastructure.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions