What is wrong
A mutation score that is measured once and never again is a fact about a
Tuesday. This ticket is the standing gate: a workflow that runs the framework
mutation campaign and fails on a survivor that is not allowlisted.
It is filed separately from morph#405 on purpose. That ticket must produce a
first number before this one can state a target, and a target invented before
the measurement is exactly the aspirational-gate failure codecov.yml already
warns about:
An aspirational target would show every PR as failing-but-ignored, which reads
as noise and gets tuned out.
Why it cannot simply be added to the existing CI
The mutation campaign rebuilds and re-runs the suite once per mutant. morph_tests
is a 102-file Catch2 binary over a header-only template library; the campaign's
wall-clock is the binding constraint and is unknown until morph#405 measures it.
The likely shapes, in increasing order of what they cost to operate:
- a nightly or weekly scheduled run rather than per-PR;
- a per-PR run scoped to the subsystems the PR touches;
- a full per-PR gate, which is only affordable if the campaign turns out to be
minutes rather than hours.
The self-hosted Linux runners added in morph#263 are the reason this is worth
filing at all — a campaign of this shape has nowhere to run on hosted minutes.
Note morph#397: every self-hosted job can currently fail in Configure, so that
capacity is not dependable yet.
The constraint
The gate's unit of failure must be a new survivor, not an absolute score.
An absolute threshold ratchets against refactors that legitimately change the
mutant population; a new-survivor gate is the mutation analogue of a patch
coverage status and is what makes the check actionable on the PR that caused it.
Survivors judged equivalent go in an allowlist with a reason each — the shape
scripts/scenario/coverage_allowlist.json already uses, and the reason
scripts/check_rung_filters.sh exists: an allowlist nothing validates rots.
Verification status: not applicable — this is a proposal, not a defect report
No premise here is measured, and it deliberately asserts none. Its predecessor
morph#405 carries the empirical work.
What would close this
- A workflow runs the framework mutation campaign on a stated cadence, and its
cadence is justified by the measured runtime from morph#405 rather than
chosen.
- A new, non-allowlisted survivor fails a check — demonstrated on a real
pull request, in the manner morph#402 requires of the coverage gate.
- The allowlist is validated: an entry naming a mutant that no longer exists
fails the build, so a stale exemption cannot outlive its reason.
docs/ records how to reproduce the campaign locally, including the clang
version constraint.
Blocked by morph#405. No target, no cadence and no allowlist format can be
specified before a first campaign has run.
What is wrong
A mutation score that is measured once and never again is a fact about a
Tuesday. This ticket is the standing gate: a workflow that runs the framework
mutation campaign and fails on a survivor that is not allowlisted.
It is filed separately from morph#405 on purpose. That ticket must produce a
first number before this one can state a target, and a target invented before
the measurement is exactly the aspirational-gate failure
codecov.ymlalreadywarns about:
Why it cannot simply be added to the existing CI
The mutation campaign rebuilds and re-runs the suite once per mutant.
morph_testsis a 102-file Catch2 binary over a header-only template library; the campaign's
wall-clock is the binding constraint and is unknown until morph#405 measures it.
The likely shapes, in increasing order of what they cost to operate:
minutes rather than hours.
The self-hosted Linux runners added in morph#263 are the reason this is worth
filing at all — a campaign of this shape has nowhere to run on hosted minutes.
Note morph#397: every self-hosted job can currently fail in Configure, so that
capacity is not dependable yet.
The constraint
The gate's unit of failure must be a new survivor, not an absolute score.
An absolute threshold ratchets against refactors that legitimately change the
mutant population; a new-survivor gate is the mutation analogue of a patch
coverage status and is what makes the check actionable on the PR that caused it.
Survivors judged equivalent go in an allowlist with a reason each — the shape
scripts/scenario/coverage_allowlist.jsonalready uses, and the reasonscripts/check_rung_filters.shexists: an allowlist nothing validates rots.Verification status: not applicable — this is a proposal, not a defect report
No premise here is measured, and it deliberately asserts none. Its predecessor
morph#405 carries the empirical work.
What would close this
cadence is justified by the measured runtime from morph#405 rather than
chosen.
pull request, in the manner morph#402 requires of the coverage gate.
fails the build, so a stale exemption cannot outlive its reason.
docs/records how to reproduce the campaign locally, including the clangversion constraint.
Blocked by morph#405. No target, no cadence and no allowlist format can be
specified before a first campaign has run.