Skip to content

mutation: the framework's own C++ has never been measured against a single mutant, so an inverted if in include/morph fails no known test #405

Description

@Yaraslaut

What is wrong

Nothing in this repository has ever checked that morph_tests asserts
anything about the code it executes.

Coverage answers "did a test run this line". It cannot answer "would a test have
noticed if this line were wrong" — and those come apart exactly where a test
drives code without asserting on its result. The check for the second question
is mutation testing: change the program, and require a test to fail.

The repository does contain a mutator, and it is the wrong one for this.
scripts/scenario/mutate_scenario.py flips comparison operators in scenario
files
— the wire-level corpus under scripts/scenario/scenarios/, which
exercises the example rungs through a running server. It never touches
include/morph. The framework's own 102-file Catch2 suite has never been run
against a single mutant of the framework's own C++.

Concretely: invert any if in include/morph/core/bridge.hpp and nothing in
this repository will tell you whether the suite catches it.

Why coverage will not answer it

include/morph is at 95.69% lines on master, and 120 of its lines are
partial. A high line number is compatible with a suite that calls everything and
checks little — and this suite has a specific reason to be suspected of it,
recorded in codecov.yml itself:

the first measurement put rule_model.cpp at 60.76%, which turned out to be
two documented obligations nothing verified

and, for crm:

ContactModel::attachActionLog and SavedViewModel::attachActionLog are
called by no test at all

Those were found by coverage because they were never executed. A path that is
executed but not asserted is invisible to every measurement this repository
currently takes.

The constraint that makes this harder than it looks

  1. Tooling against clang 20/22. include/morph is header-only and
    template-heavy. The realistic candidates are Mull (LLVM IR-level, needs an
    LLVM version match) and Dextool mutate (source-level, clang tooling). Neither
    is installed here, and Mull's supported-LLVM matrix has historically lagged
    several major versions behind. CI pins clang 20; this workstation runs
    clang 22. A tool that does not build against either is the most likely way
    this ticket dies
    , and settling that is its first deliverable, not an
    assumption.
  2. Mutant count. Header-only templates instantiate per translation unit; a
    naive IR-level mutator produces one mutant per instantiation. Runtime is the
    binding constraint, and a subsystem-at-a-time scope is likely the only
    affordable shape.
  3. Equivalent mutants are undecidable in general. A literal 100% mutation
    score is not a coherent target. The honest target is 100% of non-equivalent
    mutants
    , which requires a survivor allowlist carrying a stated reason per
    entry — the shape this repository already uses in
    scripts/scenario/coverage_allowlist.json and in codecov.yml's
    documented-ceiling audits.

Verification status: reproduced

  • mull-runner, mull-cxx, dextool, mutate: none present on PATH.
  • grep -rn mutate .github/workflows/ — no occurrence in any workflow.
  • scripts/scenario/mutate_scenario.py's pair list operates on scenario text;
    its inputs are scripts/scenario/scenarios/**.
  • tests/CMakeLists.txt:7 defines morph_tests (Catch2), over 102 test_*.cpp
    files.
  • clang --version → 22.1.8 locally; .github/workflows/ci.yml pins clang 20.

Not verified: whether any mutation tool currently builds and runs against
this tree. That is the question this ticket exists to answer.

What would close this

Close this when all hold:

  1. A documented, reproducible command produces a mutation score for
    include/morph
    driven by morph_tests — the tool named, its version
    pinned, and its clang-version constraint recorded next to the pin.
  2. That first score is written down, whatever it is. A low number is the
    finding, not the failure; this is the same discipline morph#383's acceptance
    applies to the scenario mutator.
  3. Survivors are triaged into killed / equivalent / genuinely unasserted,
    with the equivalent ones in an allowlist carrying a reason each — not a bare
    count.
  4. At least one survivor is turned into a failing-then-passing test, proving the
    loop closes end to end rather than that a report can be generated.

Not closable by adopting a tool that runs but is scoped so narrowly it
mutates nothing interesting — the report must cover core and forms, the two
subsystems carrying the dispatch and rendering logic.

Deliberately not in scope: running the whole thing in CI, and any target
number. Both are morph#408's, and neither can be specified before this ticket
produces a first measurement.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: ciSubsystem: cienhancementNew feature or requesttriage: validWell-framed; implement as written

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions