What is wrong
Nothing in this repository has ever checked that morph_tests asserts
anything about the code it executes.
Coverage answers "did a test run this line". It cannot answer "would a test have
noticed if this line were wrong" — and those come apart exactly where a test
drives code without asserting on its result. The check for the second question
is mutation testing: change the program, and require a test to fail.
The repository does contain a mutator, and it is the wrong one for this.
scripts/scenario/mutate_scenario.py flips comparison operators in scenario
files — the wire-level corpus under scripts/scenario/scenarios/, which
exercises the example rungs through a running server. It never touches
include/morph. The framework's own 102-file Catch2 suite has never been run
against a single mutant of the framework's own C++.
Concretely: invert any if in include/morph/core/bridge.hpp and nothing in
this repository will tell you whether the suite catches it.
Why coverage will not answer it
include/morph is at 95.69% lines on master, and 120 of its lines are
partial. A high line number is compatible with a suite that calls everything and
checks little — and this suite has a specific reason to be suspected of it,
recorded in codecov.yml itself:
the first measurement put rule_model.cpp at 60.76%, which turned out to be
two documented obligations nothing verified
and, for crm:
ContactModel::attachActionLog and SavedViewModel::attachActionLog are
called by no test at all
Those were found by coverage because they were never executed. A path that is
executed but not asserted is invisible to every measurement this repository
currently takes.
The constraint that makes this harder than it looks
- Tooling against clang 20/22.
include/morph is header-only and
template-heavy. The realistic candidates are Mull (LLVM IR-level, needs an
LLVM version match) and Dextool mutate (source-level, clang tooling). Neither
is installed here, and Mull's supported-LLVM matrix has historically lagged
several major versions behind. CI pins clang 20; this workstation runs
clang 22. A tool that does not build against either is the most likely way
this ticket dies, and settling that is its first deliverable, not an
assumption.
- Mutant count. Header-only templates instantiate per translation unit; a
naive IR-level mutator produces one mutant per instantiation. Runtime is the
binding constraint, and a subsystem-at-a-time scope is likely the only
affordable shape.
- Equivalent mutants are undecidable in general. A literal 100% mutation
score is not a coherent target. The honest target is 100% of non-equivalent
mutants, which requires a survivor allowlist carrying a stated reason per
entry — the shape this repository already uses in
scripts/scenario/coverage_allowlist.json and in codecov.yml's
documented-ceiling audits.
Verification status: reproduced
mull-runner, mull-cxx, dextool, mutate: none present on PATH.
grep -rn mutate .github/workflows/ — no occurrence in any workflow.
scripts/scenario/mutate_scenario.py's pair list operates on scenario text;
its inputs are scripts/scenario/scenarios/**.
tests/CMakeLists.txt:7 defines morph_tests (Catch2), over 102 test_*.cpp
files.
clang --version → 22.1.8 locally; .github/workflows/ci.yml pins clang 20.
Not verified: whether any mutation tool currently builds and runs against
this tree. That is the question this ticket exists to answer.
What would close this
Close this when all hold:
- A documented, reproducible command produces a mutation score for
include/morph driven by morph_tests — the tool named, its version
pinned, and its clang-version constraint recorded next to the pin.
- That first score is written down, whatever it is. A low number is the
finding, not the failure; this is the same discipline morph#383's acceptance
applies to the scenario mutator.
- Survivors are triaged into killed / equivalent / genuinely unasserted,
with the equivalent ones in an allowlist carrying a reason each — not a bare
count.
- At least one survivor is turned into a failing-then-passing test, proving the
loop closes end to end rather than that a report can be generated.
Not closable by adopting a tool that runs but is scoped so narrowly it
mutates nothing interesting — the report must cover core and forms, the two
subsystems carrying the dispatch and rendering logic.
Deliberately not in scope: running the whole thing in CI, and any target
number. Both are morph#408's, and neither can be specified before this ticket
produces a first measurement.
What is wrong
Nothing in this repository has ever checked that
morph_testsassertsanything about the code it executes.
Coverage answers "did a test run this line". It cannot answer "would a test have
noticed if this line were wrong" — and those come apart exactly where a test
drives code without asserting on its result. The check for the second question
is mutation testing: change the program, and require a test to fail.
The repository does contain a mutator, and it is the wrong one for this.
scripts/scenario/mutate_scenario.pyflips comparison operators in scenariofiles — the wire-level corpus under
scripts/scenario/scenarios/, whichexercises the example rungs through a running server. It never touches
include/morph. The framework's own 102-file Catch2 suite has never been runagainst a single mutant of the framework's own C++.
Concretely: invert any
ifininclude/morph/core/bridge.hppand nothing inthis repository will tell you whether the suite catches it.
Why coverage will not answer it
include/morphis at 95.69% lines onmaster, and 120 of its lines arepartial. A high line number is compatible with a suite that calls everything and
checks little — and this suite has a specific reason to be suspected of it,
recorded in
codecov.ymlitself:and, for crm:
Those were found by coverage because they were never executed. A path that is
executed but not asserted is invisible to every measurement this repository
currently takes.
The constraint that makes this harder than it looks
include/morphis header-only andtemplate-heavy. The realistic candidates are Mull (LLVM IR-level, needs an
LLVM version match) and Dextool mutate (source-level, clang tooling). Neither
is installed here, and Mull's supported-LLVM matrix has historically lagged
several major versions behind. CI pins clang 20; this workstation runs
clang 22. A tool that does not build against either is the most likely way
this ticket dies, and settling that is its first deliverable, not an
assumption.
naive IR-level mutator produces one mutant per instantiation. Runtime is the
binding constraint, and a subsystem-at-a-time scope is likely the only
affordable shape.
score is not a coherent target. The honest target is 100% of non-equivalent
mutants, which requires a survivor allowlist carrying a stated reason per
entry — the shape this repository already uses in
scripts/scenario/coverage_allowlist.jsonand incodecov.yml'sdocumented-ceiling audits.
Verification status: reproduced
mull-runner,mull-cxx,dextool,mutate: none present on PATH.grep -rn mutate .github/workflows/— no occurrence in any workflow.scripts/scenario/mutate_scenario.py's pair list operates on scenario text;its inputs are
scripts/scenario/scenarios/**.tests/CMakeLists.txt:7definesmorph_tests(Catch2), over 102test_*.cppfiles.
clang --version→ 22.1.8 locally;.github/workflows/ci.ymlpins clang 20.Not verified: whether any mutation tool currently builds and runs against
this tree. That is the question this ticket exists to answer.
What would close this
Close this when all hold:
include/morphdriven bymorph_tests— the tool named, its versionpinned, and its clang-version constraint recorded next to the pin.
finding, not the failure; this is the same discipline morph#383's acceptance
applies to the scenario mutator.
with the equivalent ones in an allowlist carrying a reason each — not a bare
count.
loop closes end to end rather than that a report can be generated.
Not closable by adopting a tool that runs but is scoped so narrowly it
mutates nothing interesting — the report must cover
coreandforms, the twosubsystems carrying the dispatch and rendering logic.
Deliberately not in scope: running the whole thing in CI, and any target
number. Both are morph#408's, and neither can be specified before this ticket
produces a first measurement.