coverage: the clang-coverage preset uses the shared compiler cache, which serves objects across worktrees and silently drops files from the report

[Yaraslaut]

Found by the N ladder lane while trying to measure morph#411 and morph#412,
and routed through the manager because the compiler cache is shared by every
lane. Structurally confirmed against master be478df1 before filing.

What is wrong

A clang-coverage build uses the shared compiler cache, the cache can serve an
object compiled in a different worktree, and nothing normalises the source
paths that llvm-cov records. The result is a coverage report whose rows carry
two different absolute prefixes for the same logical file.

Measured by N ladder, from a build configured entirely inside
/home/yaraslau/repo/morph-wt/368:

368/examples/crm/include/crm/models/account_model.hpp
372/examples/crm/src/models/account_model.cpp     <- from ANOTHER worktree

The .cpp objects came out of lane 372's worktree through
fastcache-cc (FASTCACHE_ADDR=127.0.0.1:6674).

The three structural facts that make it possible

All read on master be478df1:

1. The coverage preset does not disable the cache. CMakePresets.json's
   clang-coverage sets CMAKE_BUILD_TYPE=Debug and AF_COVERAGE=ON and
   nothing else, while cmake/CompileCache.cmake:48 declares
   USE_COMPILER_CACHE with default ON.
2. Nothing normalises paths. grep -rn "coverage-prefix-map\|file-prefix-map\|fprofile-abs-path"
   over cmake/ and CMakeLists.txt returns nothing, so clang embeds absolute
   source paths verbatim into the instrumented objects.
3. scripts/coverage.sh filters by relative path — SOURCES=(include/morph),
   examples/<rung>/<sub>. A row recorded as .../372/examples/crm/src/...
   cannot match examples/crm/src/models, so it is not merely mis-attributed,
   it vanishes from the report entirely.

What it costs, measured

Two independent consequences, both observed:

• Whole file sets disappear. In that run, crm's and lims's entire
  src/models/*.cpp set is absent from the combined report, while ledger's and
  kanban's are present.

• llvm-cov drops functions it cannot reconcile, reporting
  1071 functions have mismatched data. The same file measured two ways:

  file	via coverage.sh's 8-object invocation	against ladder_common_tests alone
  examples/common/gui/event_poller.hpp	92 lines / 31 missed — 66.30%	106 lines / 1 missed — 99.06%
  presenter.hpp	86.54%	100.00%

A 33-point swing on the same file, from the cache alone.

Why this matters beyond one lane

Every coverage ticket in the current sprint measures locally before proposing a
target: morph#403, morph#402, morph#404, morph#406, morph#409, morph#410,
morph#411, morph#412. No local coverage number is trustworthy while this
holds, which means a target derived from one is a target derived from noise.
morph#403 in particular is about making the coverage denominator honest; it
cannot succeed while the numerator is contaminated.

What this does NOT claim

CI is probably unaffected, and the Codecov figures may still be real. CI
uses sccache in a single checkout, so there is no second worktree for a path to
come from. The 90.39% (examples/common) and 87.67% (crm) figures that
morph#411 and morph#412 quote are from the Codecov API and are not challenged
here. This issue is about local measurement, and about a hazard CI would
inherit the moment it built from more than one checkout.

Verification status

Reproduced (by N ladder): the two-prefix report, the missing rung sources,
the 1071 functions have mismatched data line, and the two per-file deltas
above. Structurally confirmed (by the manager, independently): the preset's
cache variables, USE_COMPILER_CACHE's ON default, the absence of any
prefix-map flag, and coverage.sh's relative SOURCES filters.

Not verified: that -DUSE_COMPILER_CACHE=OFF alone produces a report
matching CI's numbers. A cache-free rebuild was in progress when this was filed
and its result should be recorded here.

Not verified: whether the same cross-worktree reuse corrupts anything other
than coverage. A non-coverage object carries paths only in debug info, so the
blast radius is probably limited to coverage and to debugger source resolution —
but that is reasoning, not measurement.

What would close this

1. A clang-coverage configure cannot silently consume a foreign worktree's
   objects. Either AF_COVERAGE=ON forces USE_COMPILER_CACHE=OFF, or the
   build adds -ffile-prefix-map/-fcoverage-prefix-map so recorded paths are
   worktree-independent and cache hits become correct rather than merely fast.
   State which was chosen and why.
2. A coverage report produced from any worktree contains no path outside the
   configured source directory — checkable mechanically, and worth a check in
   scripts/coverage.sh itself, since a silently shrinking report is precisely
   the failure mode that file's own comments warn about twice (morph#141,
   morph#179).
3. llvm-cov's functions have mismatched data count is zero, or the
   remaining count is explained. A nonzero count means data is being discarded.
4. A local cache-free run and CI's Codecov figures are compared and the
   difference stated, so future local measurements have a known relationship to
   the number the project gates on.

Blocks the measurement clauses of morph#403, morph#402, morph#404,
morph#406, morph#409, morph#410, morph#411 and morph#412.

[Yaraslaut]

Correction from the manager: this does not reproduce, and my framing above was too strong

The T tests lane ran the decisive check in its own coverage tree and the
effect is absent. I filed this issue asserting the consequence
("no local coverage number is trustworthy"); that was one tree's observation
generalised, and it is not supported. Relabelling triage: unverified and
correcting the record here rather than editing the text above, so the reasoning
stays visible.

The check that settles it, and why it is the right one

llvm-cov export over all 16 objects with no source filter at all — the
unfiltered coverage mapping, before anything can be dropped for failing to match
a relative path. Result: 607 files, every one under
/home/yaraslau/repo/morph-wt/372. Zero foreign roots.

grep '^SF:' build/clang-coverage/coverage.lcov | grep -v '^/home/yaraslau/repo/morph-wt/372/'
(no output)

That is exactly the artefact this issue's mechanism must appear in: a record
whose path came from another worktree would be present in the unfiltered export
and absent from the filtered one. There are none. And the build did use the
cache — configure logged [cache] Enabling fastcache-cc at 127.0.0.1:6674 — so
this is not a tree that accidentally avoided the mechanism.

The 33-point swing does not reproduce either, and its shape is informative

On examples/common/gui/event_poller.hpp:

measurement	N ladder	T tests
single object (ladder_common_tests)	99.06%	99.06%
multi-object	66.30%	100.00%

The two trees agree exactly wherever one object is involved, and diverge only
where several are — and T tests' figure moves up with more objects, not
down. That points at the multi-object run rather than at the tree, the cache or
the toolchain.

A likelier explanation for the 66.30%, and it is testable

llvm-cov resolves each function record by (name, structural hash) across the
objects it is given, and warns N functions have mismatched data when two
objects carry a same-named function with a different hash. In T tests' tree
that count is 1972 with 16 objects, up from 1076 with 9 — it scales with the
number of objects, not with cache use, because a header-only template library
instantiates the same functions differently in each binary.

So the 1071 functions have mismatched data line quoted in the original report
is expected at that object count and is not evidence of cross-worktree
contamination. If a run additionally passes objects built at different times —
a stale build directory, or one rebuilt between the two measurements — records
drop for hash mismatch and a file reads low with no foreign path involved at
all. That is the leading hypothesis now.

One argument in the original filing was wrong

I wrote that a missing file cannot distinguish "binary never passed to
llvm-cov" from "path did not match the filter". In a controlled A/B they
are distinguishable, and T tests' was one: a single build tree, a single
filter, a single profile set, with only the -object list changing. A
path-matching failure affects both arms identically and therefore cannot produce
a difference between them. And the observed movement was files appearing
(include/morph/net's seven files; examples/kanban/src/headless/main.cpp once
its binary was registered) — a filter mismatch can never add a file.

morph#403's evidence therefore stands unaffected.

What this issue now needs, in order

1. Establish whether the effect exists outside one tree. Run the unfiltered
   llvm-cov export in the worktree that produced the original report and look
   at the roots. If they are all local, the two-prefix report has a different
   cause and this issue should be reframed around that cause or closed.
2. Only if foreign roots are actually present: proceed to the fix, choosing
   between forcing USE_COMPILER_CACHE=OFF under AF_COVERAGE and adding
   prefix-map flags — a real trade between correctness and build time for every
   lane, which is why it stays a separate ticket rather than folding into
   morph#403.

The three structural facts in the original filing are unchanged and still true:
the coverage preset does not disable the cache, no prefix-map flag exists, and
coverage.sh filters by relative path. They make the mechanism possible.
Nothing yet shows it occurring.

Unblocking

I had recorded this as blocking morph#402, #404, #406, #409, #410, #411 and
#412. That was premature on one unreproduced observation and is being removed.

[Yaraslaut]

Resolved: the defect is real. Reverting my triage: unverified relabel.

The unfiltered export was run in the worktree that produced the original report,
and foreign roots are present. My previous comment generalised one lane's
negative result the same way the original filing generalised one lane's positive
result; this comment is the measurement that settles it.

The check, in the tree that shows it

llvm-cov export over all 9 objects, no source filter, the same
merged.profdata coverage.sh used:

total SF records                              688
under .../morph-wt/368/                       442
under .../morph-wt/372/                       246

The 246 are not all third-party:

count	root	kind
56	372/build/clang-coverage/_deps/lightweight-src/…	third-party
55	372/build/…/moc_*.cpp	AUTOMOC output
135	372/examples/…	first-party
28	372/include/morph/…	first-party

The sharpest fact: examples/crm/src/models/*.cpp — all eight files, the
entire src side of codecov.yml's crm component — appears in the export
only under the 372 root and never under 368. coverage.sh filters
positionally by the relative path examples/crm/src/models, so every one of
those records is dropped. That is why crm's model .cpp files are absent from
the report entirely.

examples/common appears under both roots, and the split is diagnostic: the
368 root carries 17 headers; the 372 root carries the same headers again
plus every .cpp (gui/app_context.cpp, gui/error_text.cpp,
gui/event_poller.cpp, testkit/fault_proxy.cpp, testkit/qml_surface.cpp).
That foreign set is exactly the file set missing from the filtered report and
exactly the set that reappeared when the same tree was measured against
ladder_common_tests alone.

Why the other lane saw none — and why that is not a contradiction

A cache hit only occurs when another worktree has already compiled the
identical TU with an identical command line. The tree that shows foreign roots
configured with CI's exact line:

--preset clang-coverage -DMORPH_BUILD_NET=ON -DMORPH_BUILD_OFFLINE_SQLITE=ON \
  -DMORPH_BUILD_QT=ON -DMORPH_BUILD_LADDER=ON -DMORPH_LADDER_RUNGS=all

and another lane had built the same thing. A lane whose flags differ gets misses
and compiles locally, so its objects carry only its own paths. That is a tree
which got no hits, not a tree in which the mechanism is absent. A negative
there is consistent with the defect rather than evidence against it — which is
what my previous comment failed to distinguish.

The two hypotheses are not competing, and staleness is ruled out

The functions have mismatched data count is not evidence of this defect on
its own, and the lane that raised it has withdrawn that argument: 1071 at 9
objects there, 1076 at 9 objects in the other tree — consistent, and scaling
with object count as a header-only template library would predict.

But the two mechanisms are causally linked, not alternatives. The same header
appears under both roots, so llvm-cov sees two same-named functions with
different structural hashes and drops them — and the cause of the two hashes is
the two path roots. event_poller.hpp reading 99.06% single-object and 66.30%
multi-object is that drop.

Staleness is excluded twice over:

1. build/clang-coverage was configured once, built once to completion (exit 0,
   zero error:), ctest ran once (2376/2376 passed, 2291 profraw), and
   coverage.sh ran once. Both measurements used the same binaries and the same
   merged.profdata with no rebuild between them.
2. Decisively: the crm run was single-object (ladder_crm_tests alone, no
   filter) and already printed both roots side by side. With one object there is
   no cross-object hash collision available as an explanation.

The mechanism, stated precisely

No -fdebug-prefix-map / -fcoverage-prefix-map / -ffile-prefix-map is set
anywhere in the build. A cache entry is keyed on content, but the object it
returns embeds an absolute source path. A hit across worktrees therefore
yields an object whose coverage mapping names the other tree. The coverage
preset does not opt out of the launcher, and coverage.sh filters by relative
path, so the mis-rooted records are silently dropped rather than
mis-attributed.

Two independent fixes, either sufficient

1. Set a coverage prefix map in the coverage preset, so recorded paths are
   worktree-independent and a cache hit becomes correct as well as fast.
2. Have the coverage preset opt out of the launcher
   (-DUSE_COMPILER_CACHE=OFF), trading build time for correctness.

Both live in CMakePresets.json / cmake/. The choice is a real trade-off for
every lane and should be made deliberately, which is why this stays separate
from morph#403 rather than folding into it.

What is unchanged

• CI is not affected. Single checkout, sccache, no second worktree for a
  path to come from. The Codecov figures morph#411 and morph#412 quote are still
  not challenged and may be a real regression.
• morph#403's evidence stands. Its A/B moved in the direction of files
  appearing as objects were added, and a filter mismatch cannot add a file.