Skip to content

mutation: a survivor introduced by a pull request should fail a check, and no workflow runs the campaign at all #408

Description

@Yaraslaut

What is wrong

A mutation score that is measured once and never again is a fact about a
Tuesday. This ticket is the standing gate: a workflow that runs the framework
mutation campaign and fails on a survivor that is not allowlisted.

It is filed separately from morph#405 on purpose. That ticket must produce a
first number before this one can state a target, and a target invented before
the measurement is exactly the aspirational-gate failure codecov.yml already
warns about:

An aspirational target would show every PR as failing-but-ignored, which reads
as noise and gets tuned out.

Why it cannot simply be added to the existing CI

The mutation campaign rebuilds and re-runs the suite once per mutant. morph_tests
is a 102-file Catch2 binary over a header-only template library; the campaign's
wall-clock is the binding constraint and is unknown until morph#405 measures it.
The likely shapes, in increasing order of what they cost to operate:

  • a nightly or weekly scheduled run rather than per-PR;
  • a per-PR run scoped to the subsystems the PR touches;
  • a full per-PR gate, which is only affordable if the campaign turns out to be
    minutes rather than hours.

The self-hosted Linux runners added in morph#263 are the reason this is worth
filing at all — a campaign of this shape has nowhere to run on hosted minutes.
Note morph#397: every self-hosted job can currently fail in Configure, so that
capacity is not dependable yet.

The constraint

The gate's unit of failure must be a new survivor, not an absolute score.
An absolute threshold ratchets against refactors that legitimately change the
mutant population; a new-survivor gate is the mutation analogue of a patch
coverage status and is what makes the check actionable on the PR that caused it.

Survivors judged equivalent go in an allowlist with a reason each — the shape
scripts/scenario/coverage_allowlist.json already uses, and the reason
scripts/check_rung_filters.sh exists: an allowlist nothing validates rots.

Verification status: not applicable — this is a proposal, not a defect report

No premise here is measured, and it deliberately asserts none. Its predecessor
morph#405 carries the empirical work.

What would close this

  1. A workflow runs the framework mutation campaign on a stated cadence, and its
    cadence is justified by the measured runtime from morph#405 rather than
    chosen.
  2. A new, non-allowlisted survivor fails a check — demonstrated on a real
    pull request, in the manner morph#402 requires of the coverage gate.
  3. The allowlist is validated: an entry naming a mutant that no longer exists
    fails the build, so a stale exemption cannot outlive its reason.
  4. docs/ records how to reproduce the campaign locally, including the clang
    version constraint.

Blocked by morph#405. No target, no cadence and no allowlist format can be
specified before a first campaign has run.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: ciSubsystem: cienhancementNew feature or requesttriage: rescopeReal problem, wrong framing; rewrite before building

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions