T tests lane: AUTOMOC include climb, the coverage denominator, a branch gate, and the framework's first mutation score - #429
Merged
Merged
Conversation
moc's default include for a class's own header is a path relative to the generated file, with a run of ".." long enough to climb back to the source root. A quoted include is resolved against every -I entry as well as the including file's own directory, so that climb is attempted from each of the target's include directories too -- and whether it lands on a second, different file is arithmetic on how deep the checkout sits inside its parent directories. With a git worktree placed inside the repository it checks out (.claude/worktrees/<name>/, where the agent harness puts them) it lands: six levels up from <worktree>/examples/<rung>/include is the outer checkout, which has examples/<rung>/gui_lib/<same name>.hpp. Clang reports -Wshadow-header once per moc'd class and -Werror fails every AUTOMOC target, so no ladder_<rung>_tests binary can be built under the project's own warning set at all. CMAKE_AUTOMOC_PATH_PREFIX makes moc emit the header path relative to the include directory the header was found under, so the generated include resolves through the target's -I set and never ascends. That removes the ambiguity rather than the diagnostic: -Wno-shadow-header is the only thing that would report a genuine cross-checkout header pickup, which this layout makes reachable. Refs #372 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
The ascending include moc emits by default is present in every build, in every checkout; it only becomes a compile error where the directory arithmetic happens to land on a second header of the same name. So the defect is silent on CI's own layout and would sit in the generated output indefinitely -- which is why the regression gate reads the generated moc sources directly rather than relying on a build failing. scripts/check_automoc_includes.sh scans a build tree for moc output (moc_*.cpp, *.moc) and fails on any quoted include carrying a ".." segment. Finding no moc output at all is also a failure: that is what an unbuilt tree looks like, and a gate that scanned nothing must not report success. scripts/test_check_automoc_includes.sh asserts all three directions against fixtures in tests/lint/automoc_includes/ -- non-ascending includes and dotted file names accepted, each ascending shape rejected on its own, and an empty directory rejected. Both are wired into ci.yml's linux-qt and ladder-tests jobs, after their builds: the gate inspects generated output, so it needs an AUTOMOC target to have actually been built, which a dependency-free lint job does not have. Refs #372 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
/simplify pass over origin/master..fix/372-worktree-automoc: the ascent match was a grep piped into a grep when one -E pattern covers both the leading and mid-path forms, the two diagnostic blocks were runs of `echo ... >&2`, and ladder-tests carried a verbatim copy of linux-qt's step comment. Refs #372 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
/code-review medium --fix over origin/master..fix/372-worktree-automoc. The self-test asserted rejection by exit status alone, inherited from test_check_test_type_names.sh -- but this checker has a second nonzero path, "no moc-generated sources found", so a fixture directory whose files stopped matching the checker's find patterns would still read as rejected while the checker no longer looked at that kind of file at all. Deleting `-o -name '*.moc'` from the checker demonstrates it: mid_path_climb holds only board.moc. The self-test now requires the ascending-include diagnostic and a file named under the fixture directory, and fails on that mutation. Two swallowed statuses in the checker itself: find inside a process substitution (pipefail does not reach it) reported an unreadable build tree as an unbuilt one, and `grep || true` turned a file grep could not read into a clean bill of health. Both are now their own diagnostic. The gate also runs in linux-all-features. That is the only leg that compiles moc output under -Weverything -- where -Wshadow-header is what actually fires -- and the only one that builds morph_forms_module's AUTOMOC output. Refs #372 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
/code-review high --fix over origin/master..fix/372-worktree-automoc. CMAKE_AUTOMOC_PATH_PREFIX only reaches a header that sits under one of its target's INCLUDE_DIRECTORIES. When none matches, CMake passes moc no -p and moc falls back to the ascending path with no warning -- reachable today, as morph_add_rung.cmake gives ladder_<rung>_tests no include directory for a rung's tests/, so the first Q_OBJECT helper header added there would fail the gate with a diagnostic that named only the other cause. Both causes are now enumerated, in the gate's message and in the CMake comment. The scan prunes _deps: FetchContent checks dependency sources out under the build tree, and a third-party file named like moc output is not this project's to fix. Two pipelines in the self-test used `printf | grep -q`, where grep exiting on first match SIGPIPEs the writer and pipefail makes that the pipeline's status -- a long enough diagnostic would read as not matching the pattern it contains. Here-strings instead. The self-test moves out of the three build jobs into its own dependency-free drift-guard job, matching every sibling gate in the repo: it needs no Qt, no toolchain and no build, and behind a 20-minute build it would not report at all when that build failed. Refs #372 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
scripts/coverage.sh named the binaries it profiled by hand. It named three
families while the tree built nine. llvm-cov resolves a .profraw's counters
through a *binary*'s coverage mapping, so morph_net_tests -- instrumented, run,
its profile data merged by llvm-profdata -- contributed nothing, because the
binary itself was passed neither positionally nor as -object. morph_qt_tests,
morph_offline_sqlite_tests and morph_net_qt_interop_tests were a step further
back still: none of the three called apply_coverage() at all, so there was not
even data to drop.
Measured here, on one build, with and without this change (clang 22.1.8, the
CI coverage leg's configure, all 2435 ctest cases passing both times):
include/morph/net absent from the report -> 1172 lines, 83.53%
(7 files: socket_backend, socket_server, and
detail/{tcp_socket,ws_handshake,ws_frame,sha1,base64})
include/morph/offline 88.82% -> 94.49% lines
sqlite_offline_queue 66 of 183 lines missed -> 29
include/morph/qt 24 lines, 87.50% -> 45 lines, 97.78% (qt_tls.hpp appears)
include/morph overall 96.24% -> 95.06% lines, 90.42% -> 88.67% branches
The library's number goes *down*, which is the point: 1172 previously unscored
lines at 83.53% were being left out of an average that read 96.24%.
This is the third occurrence of one defect. morph#141 (rungs 2-4 shipped
without ever being added here, ~15k lines outside the number) and morph#179
(the hand-copied rung list had drifted past ledger and lims, while codecov.yml
scored a set of files no report contained) were the first two, and both were
fixed by deleting a hand-maintained list. The test-executable list never got
that treatment, and this script's own comment predicted the result: "Nothing
fails when a rung is forgotten -- the script runs, the report uploads, and the
figure is simply computed over a shrinking fraction."
So the list is derived and the silence is broken, in three parts:
apply_coverage() registers what it instruments. An EXECUTABLE named
<something>_tests -- the convention every test binary in the tree already
follows, including examples/common's and every rung's -- is written to
build/coverage_objects.txt with its generator-resolved path. TEST forces
registration for a test binary deliberately not named that way. Libraries,
GUI shells and demos stay out: handing llvm-cov a demo would add
instantiations only the demo has and score them as uncovered, moving the
number for a reason unrelated to what any test checks.
coverage.sh reads that file instead of naming anything. A binary cannot be
missing from it without also being missing from the build.
check_coverage_objects.sh fails the coverage leg when ctest runs a binary
llvm-cov is not given. It asks ctest which binaries it runs rather than
grepping CMake files -- a grep passes just as happily on a target that is
defined, instrumented and never registered as a test -- and it reads every
element of a test's command, not command[0], because two of this
repository's tests are `node <script> <binary>` and `test_repl.sh <binary>`,
where the subject of the test is an argument. Anything unprofiled must be
named in the script with its reason.
Written the first time it ran, that gate found three more instances the
ticket did not know about, all of them in the CI coverage leg
(MORPH_BUILD_EXAMPLES defaults ON):
morph_concepts_tests 22 ctest cases over include/morph's journal, offline
queue, validation, transport-limit, versioning,
connection-scope, observability and shutdown paths;
no apply_coverage() call at all
morph_forms_demo instrumented, driven by two ctest tests, never handed
to llvm-cov -- morph#403's defect exactly
morph_qt_tls_example run by qt_tls_example_runs, never instrumented
Each is a small edit in a file under examples/, which is another lane's, so
each is recorded in coverage_exclusion_reason() as a GAP with the fix it needs
and prints on every coverage run until it lands. The alternative -- leaving
them unlisted -- is the silence this commit exists to end.
Deliberate exclusions, each stated with its reason in the same place:
morph_bench (a benchmark), morph_soak (long-running, re-drives covered paths),
the two morph_journal_skew_* probes (they register a model existing nowhere
else, and what they assert is about a file on disk), and the two libFuzzer
harnesses (different instrumentation, not built by this leg).
The gate has a self-test, because a gate nobody tests reports green whether or
not it still detects anything -- and this one guards a defect that is already a
silence, so a blind gate and a clean gate look identical. It runs in
drift-guard.yml with the other checkers' self-tests, needs no build, and
asserts six directions including the one that matters: an unprofiled suite must
be rejected *and named*.
Also fixed, found on the way: linux-coverage's "Upload coverage HTML" step
carried `if: matrix.preset == 'clang-coverage'`, left behind when the job was
split out of linux-sanitizers' matrix. The job has no matrix, so the condition
has been false on every run and the report has never been uploaded. A skipped
step and a successful one render identically, which is why nothing reported it.
Closes #403
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
…ay so
Review of the previous commit found a live instance of the defect it fixes,
inside the change itself.
ladder_<rung>_headless is instrumented (cmake/morph_add_rung.cmake) and spawned
as four separate OS processes by examples/kanban/tests/
test_kanban_process_separation.cpp through testkit/process_pool.hpp. QProcess
passes LLVM_PROFILE_FILE down, the children exit normally and so flush their
counters, and llvm-cov was handed none of it -- while scripts/coverage.sh named
examples/<rung>/src among its SOURCES the whole time. Measured: with the
registration, examples/kanban/src/headless/main.cpp appears in the report for
the first time at 107 lines, 70.09%; without it, the file is simply absent.
That is the same shape as tests/qt's qt_test_server/qt_test_client, and it takes
the same TEST keyword. It is also the case scripts/check_coverage_objects.sh
structurally cannot catch: a binary reached through a compile definition appears
in no ctest command, so the gate sees nothing to complain about. Said plainly in
both files rather than left for the next person to rediscover -- "excluded" and
"invisible" render identically in the gate's output, and only one of them is a
decision.
The rest is review feedback on the previous commit:
A gap now reads as a gap. morph_concepts_tests, morph_forms_demo and
morph_qt_tls_example printed "deliberately not profiled", which is exactly
the sentence that lets an unfixed defect stop being read as one. They print
as `warning:` and the summary counts them ("3 unfixed gap(s)"), while
genuine decisions keep `note:`. A self-test asserts the distinction, so the
two cannot quietly merge back together.
The gate no longer passes by examining nothing. `ctest --show-only` over an
unconfigured build directory returns valid JSON with zero tests, which
satisfied every check by having nothing to check -- the failure this gate
exists to detect, committed by the detector. Same guard
scripts/check_rung_filters.sh has at its own end, plus one for a manifest
that parses to no entries.
coverage.sh stops guessing CMake's output paths. Two `-x
$OUT/examples/<...>/<name>` probes still decided which *sources* were named,
so half the mechanism was derived and half hand-composed: a layout change
would keep supplying objects while silently dropping a rung's sources. Both
now look the binary up in the manifest, which holds the resolved paths.
A coverage configure always rewrites the manifest, empty included, so a tree
reconfigured from "tests on" to "tests off" cannot be left holding a stale
list of binaries that happen to still exist.
The manifest reader resolves each distinct command string once rather than
once per occurrence -- Catch2 registers 2,435 ctest cases across 18 binaries,
so that was ~4,900 realpath() walks to learn eighteen answers -- and the 2 MB
of ctest JSON is streamed into it instead of being held in a shell variable
and re-emitted down a pipe.
The profraw check runs before the gate, so driving this by hand without
having run ctest fails immediately instead of after the gate's ctest query.
The self-test's assertions use `grep <<<` rather than `printf | grep -q`:
under pipefail an early-exiting grep can leave the pipeline reporting
printf's SIGPIPE status, and every assertion there inverts a grep.
apply_coverage()'s docstring named the morph_journal_skew_* probes as TEST's
motivating case; they take no apply_coverage() at all and the gate lists them
as deliberate non-participants. It now names the case TEST actually exists
for, which is the spawned-child one above.
The coverage HTML artifact gets retention-days, now that the step it belongs
to actually runs.
Verified end to end after these changes: configure registers 16 binaries,
2435/2435 ctest cases pass, scripts/coverage.sh exits 0 reporting "16 test
binary/binaries profiled; 18 ctest binary/binaries checked; 3 unfixed gap(s)",
and the eight-case self-test passes.
Refs #403
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
component_management named eight components and every one was under examples/.
include/morph -- the product, and the largest single thing in the uploaded
report -- was scored only by the `default` status, which carries no target at
all. Eight example applications each had a stated number to hold to; the library
they exist to demonstrate had none (morph#402).
It has one now: `framework`, over include/morph/**, target 93%.
Where 93% comes from. Measured over the LCOV this repository actually uploads
(build/clang-coverage/coverage.lcov, after aggregate_lcov_branches.py), by
Codecov's own arithmetic rather than llvm-cov's: a DA line with zero hits is a
miss, a hit line carrying an untaken BRDA branch is a partial, and coverage is
hits / (hits + misses + partials). That is what makes Codecov's headline sit
below llvm-cov's line percentage, and it is why the two numbers must not be
mixed when deriving a target. It gave 7,977 hits of 8,488 lines = 93.98%, 332
misses, 179 partials -- so 93% leaves the same small margin below a measured
ceiling that every other target in this file leaves.
The method was checked against a number nobody here computed: examples/common
comes out at 90.72%, against the 90.39% morph#402 read from the Codecov API for
master. A third of a point apart, on a component this change does not touch.
The per-subsystem split is recorded next to the target, because a single
library-wide figure hides the only fact worth acting on: net is 77.61% and
carries 26% of the library's misses in 14% of its lines, while core, util,
session, qt and journal are all above 97%. That gap is visible at all only
because morph#403 stopped dropping include/morph/net's profile data -- it
contributed zero files to every report before it.
The target is marked provisional in the file, and says what has to happen to it:
it must be re-derived from the first Codecov report produced after morph#403
lands. Not because 93% is a guess, but because morph#403 moved the denominator.
Handing llvm-cov the binaries whose profile data it used to drop brought whole
files into the report that had never been in it, so include/morph's line count
is no longer the 5,716 the API reported for master, and no adjustment
reconciles the two. The measurement has to be retaken, which is exactly what
morph#403 said would happen.
`informational: true`, like every other component here. Making any of these
blocking is a decision this file already records as the repo owner's, to be
taken when the ladder is finished; this commit changes what the file says about
the framework, not what it enforces.
Also: crm's three `ignore:` entries, missing since rung 7 landed. Every rung
from bookmarks onwards got `tests/**`, `gui/**` and `gui_wasm/**` as it landed
-- including ledger and lims, whose components are scoped to models only, so
the entries are about the `default` status rather than about the rung's own
component. crm's were never added. Nothing failed, because coverage.sh's
SOURCES array names only examples/crm/{include,src,gui_lib} and so never put
those files into a report for the entries to exclude; the entries state the
intent rather than leaving it to be an accident of what another script happens
to name. scripts/check_rung_filters.sh checks that a rung has a `component_id`,
not that it has these, so the omission was unguarded.
Verified: `curl -X POST --data-binary @codecov.yml https://codecov.io/validate`
returns "Valid!" and its parse shows the framework component with
paths `(?s:include/morph/.*)\Z` and both statuses at 93% -- worth doing
explicitly, since a config that fails validation is discarded wholesale and
Codecov's own defaults silently take its place, which is how this file was inert
until morph#133. scripts/check_rung_filters.sh passes all 37 checks, and its own
self-test passes.
Two of this ticket's three closing conditions are not met by this commit and are
not mine to meet; both are reported to the sprint manager.
Refs #402
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
The repository measures branch coverage and then gates on lines. coverage.sh
exports BRDA records and runs aggregate_lcov_branches.py specifically to keep
branch data rather than drop it, Codecov receives it -- and every status and
every component in codecov.yml scores lines, while Codecov has no branch target
to set. So a branch taken one way only is visible in the report and gated by
nothing (morph#404).
A line target cannot stand in. A line is covered the moment control reaches it,
whatever the condition evaluated to: `if (a && b)` is one line and counts as hit
for any a and b. The property this repository wants held -- a dropped `!`, a `<`
that should be `<=`, a comparison written backwards is caught by a test -- is a
statement about branches.
scripts/check_branch_coverage.py measures it, from the *aggregated* LCOV and
refusing the .raw by name. That constraint is load-bearing rather than
pedantic: llvm-cov emits one BRDA record per template instantiation, and over a
header-only template library a raw branch percentage counts one source branch
dozens of times and calls it partial for every instantiation that happened not
to take an arm. aggregate_lcov_branches.py's collapse is what makes a branch
number mean anything here, so the gate is defined on its output.
Measured, clang 22.1.8, the coverage leg's configure, 2435/2435 ctest cases
passing:
include/morph 91.19% 2,588 arms 179 partial lines
net 75.15% 338 arms 66
offline 86.25% 160 arms 15
forms 92.36% 458 arms 25
core 93.22% 900 arms 51
util 93.99% 366 arms 18
session 95.83% 72 arms 3
journal 99.23% 130 arms 1
detail/qt/render 100.00% 164 arms 0
Floors are set from that, three points below, and the margin says why in the
file: these were measured on clang 22.1.8 while the leg pins clang 20, and arm
counts are a property of the instrumentation rather than of the tests. Three
points absorbs a toolchain difference while still catching the kind of drop
codecov.yml's own history records as the one nobody noticed (5.46 points). The
margin should be tightened to about a point once CI has produced a clang-20
number. A gate that cannot fire on toolchain noise is worth more than a tight
one that fires on it and gets deleted.
The gate fails in three directions, not one. Below a floor. A subsystem this
gate names that contributes no branch records -- which is morph#403 recurring,
since include/morph/net was absent from every uploaded report for exactly that
reason and absence read as nothing to check. And a subsystem in the report with
no floor of its own, so a new directory under include/morph cannot arrive scored
by nothing.
For the partial lines, scripts/branch_partial_allowlist.json holds the ones no
test can cover, with a reason each -- and is audited in both directions on every
run, which is what separates it from a suppression list. An entry whose line is
no longer partial fails: the test that covers it now exists, and the entry would
go on hiding the next regression there. An entry with no reason fails, because
a bare suppression is not a disposition.
It is keyed on the source line's *text*, not its number. A record citing a bare
line number is the defect this repository has now found three times (morph#349,
morph#355, morph#419, the last of which is open), and an allowlist keyed that
way would rot the same way while still suppressing something. If the code moves
and the text is still unique, the gate resolves it and fails with the new number
rather than silently following it.
Three entries so far, all in rational.hpp and all genuinely uncoverable: an
assert() whose false arm calls abort(), and two `if (!std::is_constant_evaluated())`
whose untaken arm is constant evaluation -- which executes no instrumented code,
so no counter can ever increment there. That leaves 176 partial lines that are
simply untested, enumerated per file by the gate on every run. Disposing of each
is the substance of morph#404 and is not the work of one commit; what this
lands is the mechanism that makes the backlog visible, countable, and impossible
to quietly suppress.
On -fcoverage-mcdc, recorded in cmake/compiler_options.cmake next to the flags
it would join, because the ticket asks for a decision either way. It works:
measured on tests/test_bridge_local.cpp with clang 22.1.8, +5.5% compile time
(17.3s against 16.4s), +1.7% object size, and no diagnostics at all -- in
particular none of LLVM's "maximum number of conditions" warnings, so morph's
decisions all fit the cap. Not adopted yet, for two reasons of sequence rather
than cost. llvm-cov's LCOV export carries no MC/DC records and Codecov has no
MC/DC concept, so the number cannot ride the upload path that already exists.
And MC/DC is strictly stronger than branch coverage, which is not yet at its
ceiling: all 176 untested partial lines are MC/DC failures too, so turning it on
today produces a large number saying what the branch number already says. The
order that buys something is to dispose of the partial branches first. Not
verified on clang 20, which is the CI pin and is not installed here.
Also in this commit, from the correctness review of the previous two:
file(GENERATE) wrote a manifest whose content is a join of $<TARGET_FILE:>
expressions to a path with no $<CONFIG> in it. Those resolve per
configuration, so CMake refused to write it and failed the *generate* step:
`cmake -G "Ninja Multi-Config" -DAF_COVERAGE=ON` would not configure at all.
Reproduced on a minimal project ("Evaluation file to be written multiple times
with different content. CMake Generate step failed."), fixed by naming the
file per configuration under a multi-config generator, and verified both ways:
the minimal repro fails without the $<CONFIG> and succeeds with it, and this
tree now configures under Ninja Multi-Config and writes three manifests.
Three assertions in the coverage-object self-test still used the exact
`printf | grep -q` shape that file's own `mentions()` helper exists to avoid.
One of them was not inverted, so a pipefail SIGPIPE would have read as "the
wrapper was not reported" and printed ok -- failing open, in the self-test
whose whole justification is that a blind gate must not look green.
The gate's ctest query gave a bare Python traceback on anything that is not a
clean JSON document, and because it runs inside a command substitution under
`set -e`, coverage.sh died with no indication of which script failed. ctest
prints a discovery script's stdout before the document, and emits nothing
parseable when it fails outright, so this is reachable.
That same query runs every PRE_TEST discovery script, which executes the
instrumented binaries with --list-tests; with no LLVM_PROFILE_FILE set they
each dropped a default.profraw into the build tree, which the *next* run of
coverage.sh would have swept into llvm-profdata merge. It now writes into a
scratch directory it deletes.
The exclusion table only knew the CI leg's configuration, so
`cmake --preset clang-coverage -DMORPH_BUILD_HMAC_EXAMPLES=ON` (or the bank
example, or the QML forms) turned coverage.sh from "produces a report" into
"exits 1 and produces nothing". Those seven suites are the same unfixed gap as
the three the leg builds, and are now listed as such. A gate against silent
omission should not make a legitimate configure unusable.
Verified: the branch gate's ten-case self-test passes, the coverage-object
gate's eight-case self-test passes, scripts/coverage.sh runs end to end and
exits 0 on the real tree, and no default.profraw is left behind.
Refs #404
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
…aths A coverage build ran through the shared compiler-cache launcher, the launcher served objects compiled in a different worktree of this repository, and clang had embedded that worktree's absolute source paths into their coverage mappings. scripts/coverage.sh filters by relative path, so those records matched nothing and were dropped rather than mis-attributed: 246 of 688 records foreign in the run that found it, and with them all eight of examples/crm/src/models/*.cpp -- the entire src side of codecov.yml's crm component -- while its tests ran and passed throughout. The mechanism, demonstrated directly rather than inferred: compile one translation unit in worktree A through the launcher, then the byte-identical unit in worktree B, and B's binary exports `SF:/.../wtA/lib.hpp`. The same unit compiled with no launcher exports B's own path, which is what identifies the launcher rather than the compiler as the cause. A cache entry is keyed on content; the object it returns embeds an absolute path, because nothing in this build passes -ffile-prefix-map or -fcoverage-prefix-map. So AF_COVERAGE now defaults USE_COMPILER_CACHE to OFF, set before cmake/CompileCache.cmake is included so its option() picks it up under CMP0077. An explicit -DUSE_COMPILER_CACHE=ON is still honoured: on a single-checkout runner the cache is safe and worth having, and only whoever configures the build knows whether theirs is shared. The prefix-map alternative was measured and rejected. It does produce worktree-independent paths (`SF:lib.hpp`), but it does not keep the cache hits it was supposed to buy -- the flag carries the absolute source root, so the two worktrees key differently and miss anyway. And it breaks the report it was meant to fix: with relative recorded paths llvm-cov's positional source filters match nothing, and llvm-cov's behaviour when a filter matches nothing is to emit every file, so coverage.sh would silently widen to include demos, `gui/` shells, fetched _deps and the test files it deliberately excludes. Correct paths bought by rewriting the filter mechanism whose silent shrinkage this issue is about was the worse trade. scripts/check_coverage_roots.sh is the other half, and is not redundant with the default: the default is overridable, the hazard returns for any cache configured to key path-independently, and the failure is a silence -- every command in the pipeline exits 0 while the figure is computed over what is left. It checks the *unfiltered* mapping, since the filtered exports contain only records that already matched a relative filter and checking those would be vacuous; `-summary-only` keeps it to under half a second over sixteen objects. Its self-test runs in drift-guard.yml beside the other checkers', and asserts that a sibling directory sharing the checkout's prefix as a string still fails -- the same family of bug as the path filter the gate guards. llvm-cov's `N functions have mismatched data` warning is recorded there as explained rather than gated on. Measured in this tree, which has zero foreign paths, against one profile and a growing object list: 0, 183, 413, 998, 2184 at 1, 2, 4, 8 and 16 objects. It scales with the number of objects because a header-only template library instantiates the same function differently in each binary, so it says nothing about where the sources came from. Closes #426 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
The marker above the three Bridge subscription cases in test_coverage_gaps.cpp named a type that existed nowhere in the tree and three line ranges that had drifted onto a doc comment, a ModelId load and a parkIfInFrame guard. It also described the wrong behaviour: it claimed a weak-lock failure "when handler dies mid-flight", and none of the cases under it arranges for a binding to die while an entry for it is still listed. What they do reach is the empty and non-matching arms of the same bookkeeping: removeSubscription's erase_if with nothing to erase (bridge.hpp:977-980), publishResult's fan-out and dispatch loops with no subscription to deliver to (1026-1031, 1033-1039), and deregisterHandler's find_if predicate returning false for a live binding that is not the one being removed (1307-1310). Read against bridge.hpp rather than assuming the old ranges had shifted by a fixed amount; the type they are really about is Bridge::InstanceSubscription (1848). The weak-lock arms the old text named -- `!owner` (979), `entry.binding.expired()` (1024), a falsy `sptr` (1309) -- are now called out as what these cases do *not* cover, since a marker that overstates its cases is how this one came to be believed for as long as it was. morph#355 deleted the sibling marker at :575 because it had no cases under it at all. This one is repointed rather than deleted for the opposite reason: deleting it would leave three real cases with no statement of what they are for. The dead type name is deliberately not quoted in the replacement. This is the third of these (morph#349, morph#355, morph#419) and a grep for the dead symbol is how the next one gets found, so a comment that still spells it would read as a live reference. Closes #419 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
…g off Four defects found by a review of this branch, all in the two gates it adds. The branch-coverage gate refused the flow its own sibling documents. `cmake --preset clang-coverage` with nothing else profiles exactly one binary -- morph_tests, confirmed from the configure's own "1 test binary/binaries will be profiled" line -- and morph_tests compiles nothing under include/morph/net or include/morph/qt, which come from tests/net and tests/qt behind two options that default OFF. The gate's vacuity check then failed, naming morph#403 as the cause: a fixed defect blamed for an option simply being off, which is the same shape of wrong diagnosis as a citation that has drifted onto the wrong line. So the gate now reads coverage_objects.txt and knows which suites the build actually contained. A subsystem whose suite was not profiled is reported and not enforced, with a line saying the floors are calibrated to CI's configure -- and not enforced at all rather than only for the missing subsystems, because in CI several binaries contribute coverage to the same header, so a subset build reads low on subsystems that are present too. Scoring one configure against another one's denominator is what produced the wrong number in the first place. The mapping from subsystem to suite is checked in both directions so it cannot rot into a permanent skip: records present while the mapped suite was absent fails as a stale table, and an absent subsystem whose suite *was* profiled stays fatal as morph#403's shape. A subsystem not in the table is required unconditionally, so a new directory defaults to the strict side. With no manifest at all -- a hand run on an LCOV -- everything is enforced as before. Second, the subsystem key was the first three path components, so a header directly under include/morph/ would key on itself: `include/morph/version.hpp is in the report but carries no floor` is a demand no FLOORS edit can satisfy while the key names a file. Both such headers are macros and constexpr today and emit no records, so this was latent. They now land in one clearly-labelled bucket that is counted in the TOTAL and exempt from needing a floor of its own. Third, check_coverage_roots.sh read its manifest with only an emptiness test, so a file of blank lines left the array empty and `set -u` aborted on `objects[0]: unbound variable` -- a message about nothing to do with coverage. Lines are trimmed, an empty result says so, and each entry is checked to be an executable, which is the diagnostic its sibling gate gives for the same manifest. Both are standalone-run cases; through coverage.sh check_coverage_objects.sh gets there first. Fourth, `if(AF_COVERAGE AND NOT DEFINED USE_COMPILER_CACHE)` is also false for a value a *previous* configure left in the cache. A tree first configured without AF_COVERAGE carries USE_COMPILER_CACHE:BOOL=ON from CompileCache.cmake's own option(), so reconfiguring it with -DAF_COVERAGE=ON skipped the morph#426 default silently -- the exact failure mode that block exists to prevent. It is indistinguishable from a deliberate override, so it is not overridden; it now warns, names the two ways out, and leaves check_coverage_roots.sh to turn it into an error if the cache did lend the build another worktree's paths. Six new self-test cases cover the four, including the two directions of the stale-mapping rule and the top-level-header key. Refs #404, #426 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
…wrong
Nothing here had ever changed the framework's own C++ and required a test to
fail. Coverage answers "did a line run"; it cannot answer "would anything have
noticed if it were wrong", and those come apart exactly where a test drives code
without asserting on the result -- which is a suspicion this repository has
already had to confirm twice by hand, in codecov.yml's notes on rule_model.cpp
and on crm's attachActionLog pair.
The first score, over include/morph/core and include/morph/forms, driven by
morph_tests' 1279 passing cases:
999 mutants, 640 killed, 359 survived -- 64.06%
against a library measuring 95.69% lines and 91.19% branches. The gap between
those numbers is the finding. It is recorded rather than defended; morph#408
owns what to do about it.
The tool is Mull 0.34.0, and the risk this ticket was filed on -- that no
mutation tool builds against the pinned compilers -- turned out to be false.
Mull ships prebuilt packages for LLVM 13 through 22, so both CI's clang 20 and
this workstation's clang 22 are covered, by different packages: mull-ir-frontend
is an LLVM pass plugin and is loadable only by the clang major it was built for,
which scripts/mutation.sh checks before it starts a 30-minute build rather than
after. Measured on mull-runner-22 / LLVM 22.1.2 against clang 22.1.8; the score
has not been re-measured on clang 20, because the mutant set is generated from
IR and two clang majors do not emit the same IR.
Two things about running it are documented because both fail silently. Mull's
config is read twice -- by the frontend at compile time to decide what to
instrument, and by the runner to decide what to execute -- so a binary built
under one scope and run under another reports "No mutants found. Mutation score:
infinitely high" and exits 0. The script owns the config file so the two cannot
disagree, and refuses a binary with no .mull_mutants section. And Mull's SQLite
reporter aborts on this project ("string or blob too big", the mutants quoting
whole multi-line call sites), which Mull treats as fatal -- after the 46-minute
run and before printing the score. IDE only, and the score is derived from that
report rather than from stdout.
The 359 survivors are triaged in scripts/mutation_survivors.json rather than
counted. 11 are equivalent mutants, listed individually with a reason each: the
golden-ratio hash_combine arithmetic in registry.hpp and bridge.hpp, whose value
reaches only an unordered_map bucket index, and five reserve() capacity hints,
whose argument changes allocation and nothing else. 27 are diagnostic side
channels -- 19 logging, 8 metrics -- kept separate because they are absences of
assertion rather than equivalences, and because observability already has a test
file while logging has no sink installed at all. The remaining 321 are the
finding, and three shapes are named: the server's reply deleted from 25 dispatch
sites with the suite still passing, 30 boundary comparisons exercised only away
from their boundaries, and 13 completions never delivered with nothing waiting.
One of them is now a test, which is what proves the loop closes rather than that
a report can be generated. wire.hpp's kMaxEnvelopeBytes cap survived `>` becoming
`>=` -- the cap moving down a byte and rejecting a legal envelope -- while the
case immediately above it in test_wire_hardening.cpp was already named "decode
accepts an envelope at the size limit boundary" and asserted a 1 KiB payload.
The new case decodes an envelope of exactly kMaxEnvelopeBytes and one of exactly
kMaxEnvelopeBytes + 1. With the mutant applied it fails --
"input exceeds maximum size (8388608 > 8388608 bytes)" -- and without it the
suite passes 21409 assertions in 1280 cases.
Closes #405
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
…ailures Four more from a second review of this branch, and the first is the one that matters: the manifest-aware cases added in the previous commit were appended *below* self_test()'s `if failures: return 1`, so they ran, printed their errors, incremented the counter -- and the function then fell through to "all self-test checks passed" and exited 0. Breaking the OPTIONAL_SUBSYSTEMS logic and re-running the self-test produced a green drift-guard step. The check now sits last, where a case added after it cannot be silently swallowed, and says so. Second, the subset-build path printed "not gated" and gated anyway. Per-subsystem floor breaches were already in `failures` by the time that branch ran, so a plain `cmake --preset clang-coverage` still exited 1 on floors it cannot reach -- which is the defect the previous commit set out to fix, half fixed. Floor breaches now accumulate separately from structural findings, because the two are true of different things: a stale table or an allowlist entry that no longer matches the code is a fact about the tree and fatal whatever was built, while a floor is a comparison against a number measured under CI's configure. On a subset build the breaches are printed as "would fail under CI's configure" and the structural findings still fail. Two cases pin both directions. Third, a line in test_check_coverage_roots.sh's header lost its `#` and was executed, so every run -- including drift-guard.yml's -- emitted "line 30: 6: command not found" on stderr. Fourth, the morph#426 default did not cover the route CI actually takes. cmake/CompileCache.cmake returns early when CMAKE_C[XX]_COMPILER_LAUNCHER is already set, and ci.yml's linux-coverage job passes -DCMAKE_C_COMPILER_LAUNCHER=sccache on its configure line -- so USE_COMPILER_CACHE decided nothing there while the new status message announced the cache was off. Caching on that job is safe (one checkout, no second worktree for a path to come from) and is the caller's decision, so this reports the situation instead of overriding it, and names check_coverage_roots.sh as what catches it if the same route is taken on a machine with several worktrees. Refs #404, #426 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
mull-runner exits non-zero whenever mutants survive, which is the normal outcome and the whole thing this script reports. Under `set -e` that ended the script immediately after the runner and before the score was computed -- and the failure was invisible, because the log then ends on mull's own "Surviving mutants: 125 / Total execution time" lines, which read exactly like a successful finish. Two full runs looked fine and printed no score. The runner's status is captured instead, and the report file decides: a genuine runner error (a missing library, a crashed baseline) leaves no report and fails here with the status quoted, while an ordinary non-zero with a report present is survivors and is what the score is computed from. Verified end to end: `bash scripts/mutation.sh net` now exits 0 and prints "mutation score (net): 77.88% (440 killed, 125 survived, 565 mutants)". Refs #405 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sprint round 1, lane
T tests— the lane's whole queue in one PR so CI runs oncefor it rather than seven times. This repository's self-hosted runners serialise;
a PR per ticket had five
CIruns queued behind each other with none inprogress.
Each ticket keeps its own commits, so a red job stays attributable and any one
can be reverted alone.
Closes #372
Closes #403
Closes #402
Closes #404
Closes #405
Closes #426
Closes #419
The headline: the framework's first mutation score is 64.06%
999 mutants over
include/morph/coreandinclude/morph/forms, driven bymorph_tests' 1279 passing cases — 640 killed, 359 survived. Against a librarythat measures 95.69% lines and 91.19% branches.
That 30-point gap between "executed" and "would be noticed if it were wrong" is
the finding, and it is what #405 existed to produce. The survivors are triaged in
scripts/mutation_survivors.jsonrather than counted: 11 equivalent with areason each, 27 diagnostic side channels (logging and metrics — absences of
assertion, not equivalences, and kept separate for that reason), and 321
genuinely unasserted. Three shapes recur, and the first is the one to read
twice:
reply(...)deleted from ~25 dispatch sites inremote.hppwith the suite still passing;
The loop is proven closed end to end.
wire.hpp:514'sjson.size() > kMaxEnvelopeBytessurvived>→>=— while the case directlyabove it in
test_wire_hardening.cppis already named "decode accepts anenvelope at the size limit boundary" and asserts a 1 KiB payload. A new case
decodes exactly
kMaxEnvelopeBytesand exactly one byte more; with the mutantapplied by hand it fails (
input exceeds maximum size (8388608 > 8388608 bytes)), and reverted the suite passes 21409 assertions in 1280 cases.What is in it
-Wshadow-header. Addsscripts/check_automoc_includes.shand a fixture corpus.scripts/coverage.shprofiled three test binaries out of nine. It now profiles every coverage-instrumented one, including the clientladder_kanban_headlessthat kanban spawns, and an unfixed gap says so instead of silently shrinking the denominator.include/morphgains a coverage component of its own — the framework was scored by no target at all.91.19% over 2588 arms, 179 partial lines.scripts/mutation.sh— the reproducible campaign, tool pinned (Mull 0.34.0, LLVM 22.1.2, clang 22.1.8), plus the survivor triage above.scripts/check_coverage_roots.shfails if it ever does again.test_coverage_gaps.cpp's marker repointed at the branches its cases actually take.#426: why the cache is disabled rather than prefix-mapped
Both fixes were on the table. The prefix map was rejected on measurement, not
preference:
absolute source root, so two worktrees key differently and miss anyway —
measured: a prefix-mapped object built in a third worktree carries that
worktree's own paths, i.e. it compiled locally. The "correct and fast"
branch of the trade does not exist.
(
SF:lib.hpp), llvm-cov's positional source filters match nothing — andllvm-cov's behaviour when a filter matches nothing is to emit every file.
coverage.shwould silently widen to demos,gui/shells,_depsand thetest sources it deliberately excludes. Buying correct paths by breaking the
filter whose silent shrinkage this issue is about is the worse trade.
So
AF_COVERAGE=ONnow defaultsUSE_COMPILER_CACHEto OFF, overridable for asingle checkout. CI's
linux-coverageis untouched: it passes-DCMAKE_C[XX]_COMPILER_LAUNCHER=sccache, which makesCompileCache.cmakereturn early, so it keeps sccache and pays no build-time cost.
The mechanism was reproduced directly in two minutes without a coverage build:
compile one TU in worktree A through
fastcache-cc, then the byte-identical TUin worktree B, and B's binary exports
SF:/…/wtA/lib.hpp. Same TU with nolauncher exports B's own path — which identifies the launcher, not the compiler.
A related argument settled with numbers.
llvm-cov'sfunctions have mismatched datacount, measured in a tree with zero foreignroots and one profile, over a growing
-objectlist: 0, 183, 413, 998, 2184at 1, 2, 4, 8 and 16 objects. It scales with object count because a header-only
template library instantiates the same function differently per binary. It is
not evidence of this defect in either direction, and #403's numbers do not
need re-taking —
check_coverage_roots.shrun against the exact tree #403's A/Bwas measured in reports 607 files and zero foreign roots.
Verification, all actually run
ctest --preset clang-coverage— 2436/2436 passedbash scripts/coverage.sh— exit 0.check_coverage_objects: 16 profiled /18 checked / 3 unfixed gaps.
check_coverage_roots: 607 files, all under thecheckout. Branch gate:
ok: 91.19% over 2588 arms, 179 partial lines.bash scripts/mutation.sh net— exit 0,mutation score (net): 77.88% (440 killed, 125 survived, 565 mutants),an exact reproduction of an earlier independent 125/565.
/code-review high origin/master..lane/t-testsran twice; 8 findings, allreal, all fixed in
92366f61and94c855a4. Two were self-inflicted trapsworth naming: new self-test cases appended below a
return 1, so a brokengate printed "all self-test checks passed" and exited 0 (reproduced); and the
branch gate failing the documented plain
cmake --preset clang-coverageflowwhile blaming a fixed issue for two options simply being off.
/simplifydid not run. It forks into the primary checkout and edits with--fix. Saying so rather than letting silence read as "found nothing".Two silent failure modes in the mutation tooling are documented because they
were hit: Mull's SQLite reporter aborts on this project (
string or blob too big) and Mull treats that as fatal — after the 46-minute run and before thescore; and
mull-runnerexits non-zero whenever mutants survive, soset -eended the script one line before the score while the log read like a clean
finish.
Not claimed
The mutation score is not claimed to carry over to CI's clang 20 — the
mutant set is generated from IR, and Mull ships a
-20package whose set willdiffer. Local coverage (92.52% over 22044 lines) and Codecov master (90.44% over
15546) are not comparable yet, because this branch already profiles file
sets —
include/morph/netabove all — that master's uploaded report does notcontain. The comparison to make is local-vs-Codecov on the same commit, once
this merges.
🤖 Generated with Claude Code
https://claude.ai/code/session_0154xzWuBMPveLcdeUgydifb