Plan: LLVM source-based code coverage for ponyc

[SeanTAllen]

This supersedes the earlier rough outline (discussion #4848). It corrects a
load-bearing error in that outline and lays out a staged, de-risked plan.

What this delivers

ponyc --coverage ./my-tests produces a binary that, when run, records which
Pony expressions and branches executed. Standard LLVM tooling turns that into a
per-line, per-region, per-branch report:

ponyc --coverage ./my-tests
./my-tests                                    # writes default.profraw
llvm-profdata merge default.profraw -o cov.profdata
llvm-cov show ./my-tests -instr-profile=cov.profdata

This is LLVM source-based coverage (the llvm-cov show ecosystem), chosen
over gcov-style coverage. Source-based gives per-region and true branch
coverage and embeds its mapping in the binary; the cost is that ponyc has to
produce that mapping itself (see below). gcov would be a smaller change but
gives only line-level coverage derived from debug info, and we want the richer
report.

The correction to the earlier outline

The earlier outline proposed adding only LLVM's InstrProfilingLoweringPass
plus linking the profiling runtime, estimated at "50-150 lines." That is not
sufficient, and the reason matters for the whole plan.

InstrProfilingLoweringPass is a lowering pass. Its own header says it
"lowers the profile instrumented code generated by FE or the IR based
instrumentation pass." It lowers llvm.instrprof.* intrinsics that something
else already inserted. ponyc's IR contains no such intrinsics today, so the
pass would be a no-op and the profile would be empty.

llvm-cov show needs two things that ponyc does not currently produce:

1. llvm.instrprof.increment intrinsics placed at the entry of each
   counted region, so the runtime knows what to count.
2. Coverage mapping — the __llvm_covmap / __llvm_covfun sections that
   record, for each function, which counter covers which source span and how
   counters combine for branches (e.g. else-count = body-count − if-count).

Both are produced by the compiler frontend. In clang this is
clang/lib/CodeGen/CoverageMappingGen.cpp (~2700 lines) plus
CodeGenPGO.cpp (~1600 lines), and it is tied to clang's C/C++ AST, so it is
not reusable. ponyc must grow its own equivalent against the Pony AST, using
LLVM's coverage-mapping writer API (llvm/ProfileData/Coverage/).

So the real shape is: a new coverage-emission component in ponyc's codegen
(the bulk of the work), plus the pass wiring and runtime linking the outline
already identified (the easy part). The outline described the easy part and
called it the whole job.

DWARF debug info is not the foundation here. ponyc's existing DWARF is what a
gcov-style approach would have reused; source-based coverage carries its own
source locations in the coverage mapping and does not derive them from DWARF.

Prior art that de-risks the runtime half

The runtime-and-linking half of this work is not new ground. ponyc already
builds itself with coverage instrumentation via the use=coverage Make option
(CMakeLists.txt compiles ponyc with -fprofile-instr-generate -fcoverage-mapping under PONY_USE_COVERAGE), and #5528 added CI smoke tests
that confirm the instrumented binary emits default.profraw. Separately,
genexe.cc already links a coverage runtime through embedded LLD: its
PONY_COVERAGE block adds -lgcov to programs when ponyc was built with gcov
coverage, because libponyrt then carries gcov constructors. Neither path is the
feature proposed here — both instrument ponyc/libponyrt, not user Pony
programs, and the new work is the coverage-mapping emission against the Pony AST.
What they establish is narrower than "the profile runtime is ready": they prove
coverage-style runtimes already link into Pony programs (libgcov via embedded
LLD; clang's profile runtime via the compiler driver in ponyc's own build) and
that vendored compiler-rt sources build (the crt objects come from
compiler-rt/lib/builtins). The vendored compiler-rt/lib/profile is not
built anywhere today — whether it compiles and links via embedded LLD is
exactly what Stage 0 pins down. So this is real risk reduction on the linking
approach, not proof the runtime is in hand.

Note the naming adjacency: the build-time use=coverage / PONY_USE_COVERAGE
(coverage of ponyc) is a different thing from the proposed --coverage
compiler flag (coverage of a user's program). Docs must keep the two distinct
so users and contributors don't conflate them.

How the pieces fit

At compile time, for each Pony function ponyc emits:

• a function-name global and a structural hash (matches the profile back to the
  mapping),
• llvm.instrprof.increment(name, hash, num_counters, index) at each region
  entry,
• a coverage-mapping record describing each region's source span and counter.

Then:

• InstrProfilingLoweringPass lowers the intrinsics into the counter data
  sections (__profc_*, __profd_*, __llvm_prf_names).
• The linked profiling runtime (built from the vendored
  lib/llvm/src/compiler-rt/lib/profile/) writes default.profraw at exit.
• llvm-profdata + llvm-cov read the binary's coverage mapping plus the
  profile to produce the report.

Design decisions

The flag

Add bool coverage to pass_opt_t (src/libponyc/pass/pass.h) and a
--coverage option in src/libponyc/options/options.c, following the existing
OPT_STRIP / OPT_LINT_LLVM boolean-flag pattern.

A bare --coverage (not --coverage=<format>) is the right v1 surface. We have
chosen source-based; gcov is a possible future format. It is easier to add a
format selector later than to remove one, so we start with the bare flag.

Interaction with --release and optimization (decided: error)

genopt.cc forces strip_debug = true and builds the O3 pipeline for
--release; otherwise it builds the O0 pipeline. Optimization degrades
coverage accuracy (inlining and region merging blur which source region ran).
strip_debug is unlikely to remove the coverage-mapping sections (they are not
debug metadata) — but this is unverified and only matters if optimized coverage
is pursued later; it should be confirmed as part of that future work, not
assumed now. An optimized build also produces misleading coverage regardless.

Options for --coverage + --release:

• (a) Error. Clearest semantics; coverage is a debug/test-build concern.
• (b) Allow, but warn and disable optimization. Convenient, but silently
  does something other than what --release asked.
• (c) Allow optimized coverage. clang supports -O2 -fcoverage-mapping;
  results are valid but coarser. Most work, least clear for v1.

Decided: (a) error for v1. --coverage combined with --release is a hard
error. This avoids the optimized-accuracy rabbit hole and keeps semantics
obvious; optimized coverage is possible later work.

Independently: when --coverage is set, the lowering pass must run. It is
registered via registerOptimizerLastEPCallback — the same extension point the
existing strip pass uses. That extension point does fire in the O0 pipeline
(buildO0DefaultPipeline invokes the optimizer-last callbacks, the same reason
--strip works in debug builds), so registering it there suffices and keeps the
wiring consistent with the strip pass rather than special-casing it.

Which constructs get regions

This is the bulk of the work and the main correctness surface. Counted regions
are needed for, at minimum:

• function bodies (entry counter),
• if / elseif / else arms,
• match cases, including guards,
• while, repeat, for bodies,
• try / else / then blocks,
• short-circuit and / or operands.

Pony's implicit control flow also needs handling: partial calls (?), return,
error, break, continue create gap/skipped regions that the mapping must
represent so counts after them are correct. Getting this complete and correct is
where bugs in coverage output come from, so it is built and validated
incrementally (see stages), not all at once.

Source files and the stdlib

A Pony program is whole-program compiled into one LLVM module spanning many
files and packages. The coverage mapping supports multiple file IDs per
function, so ponyc maintains a file-id registry and tags each region with its
file. v1 emits mapping for all compiled code, including the stdlib; users filter
the report with llvm-cov's -ignore-filename-regex. A ponyc-side stdlib
filter is possible future work. (Without filtering, reports are dominated by
stdlib noise — this is documented, not silently left as a surprise.)

Tooling availability — building the LLVM tools (decided)

The workflow needs llvm-cov and llvm-profdata. ponyc's CI currently builds
LLVM with llvm_tools=false (an option added solely to keep the old build cache
small by omitting the ~1.6 GB of tool binaries ponyc never links). That reason
is obsolete: the libs are cached in GHCR now, where the size doesn't matter.

Decided: remove the llvm_tools option entirely as part of this work and
build the tools everywhere. That gives CI llvm-cov/llvm-profdata so the
coverage tests can run end to end (see testing), and it incidentally fixes the
runtime-bitcode build, which needs llvm-link and fails today with the tools
off. The removal touches: Makefile, make.ps1, lib/CMakeLists.txt, every
workflow libs-build invocation (llvm_tools=false / -LlvmTools false across
pr, nightlies, release, ponyc-tier2/tier3, ponyc-weekly-checks,
update-lib-cache, and the Windows variants), and the two coupling notes that
document the flag (AGENTS.md, .github/workflows/AGENTS.md). Because the edit
changes hashed build files (Makefile, make.ps1, lib/CMakeLists.txt), the
libs-cache key rotates, so new tools-on artifacts don't collide with the cached
tools-off ones — the change busts the cache cleanly.

This resolves tooling for CI. Shipping the tools to end users (so a released
ponyc carries llvm-cov) is a separate packaging question and remains future
work; for now a user viewing coverage reports supplies matching LLVM tools.

Staged implementation

Each stage is independently mergeable and independently validated. Stage 0 is a
throwaway spike; the rest are real.

Stage 0 — Spike (throwaway, de-risks everything)

Prove the end-to-end pipeline before building the region emitter. Hand-emit, for
a trivial Pony program, function-entry instrprof.increment intrinsics and a
minimal coverage mapping covering just function bodies. Wire the lowering pass,
build and link the profiling runtime, run the program, and confirm
llvm-cov show reports function execution against the .pony source.

What the spike pins down (all specific to the vendored LLVM major, currently
22):

• the exact intrinsic, section, and global names, and the coverage-mapping
  encoding the CoverageMappingWriter produces,
• that the lowering pass runs and emits the counter sections,
• that the runtime builds from the vendored source, links via LLD, and writes a
  non-empty .profraw,
• that llvm-cov reads our mapping.

If any of these fight us, we learn it here, cheaply, before investing in region
emission. The spike is deleted once Stage 1 reimplements it properly.

Stage 1 — Plumbing + function-level coverage (real)

• Remove the llvm_tools option and build the tools everywhere (see
  "Tooling availability" above). This lands first because the Stage 1 end-state
  check and all later coverage tests need llvm-cov. Edit the hashed build files
  (Makefile, make.ps1, lib/CMakeLists.txt) so the libs-cache key rotates,
  strip llvm_tools=false / -LlvmTools false from every workflow libs build,
  and update the two coupling notes (AGENTS.md, .github/workflows/AGENTS.md).

• --coverage flag and pass_opt_t.coverage.

• New C++ codegen component (e.g. src/libponyc/codegen/gencoverage.cc/.h,
  C++ because the LLVM coverage APIs are C++) exposing C-callable hooks the
  C codegen calls. It owns the file-id registry, per-function counter
  allocation, intrinsic emission, and the module-level coverage-mapping global.

• Function-entry counters and mapping only (no inner regions yet).

• genopt.cc: register InstrProfilingLoweringPass via
  registerOptimizerLastEPCallback gated on opt->coverage — the same
  extension point the strip pass uses (the pipeline is built and run wholesale,
  so passes are injected through these callbacks, not appended afterward).
  Handle the --release interaction per the decision above.

• New src/profile_rt/CMakeLists.txt building the profiling runtime from
  lib/llvm/src/compiler-rt/lib/profile/ (the right platform files per OS),
  following the src/crt/CMakeLists.txt pattern, shipped alongside libponyrt.

• Install/packaging (not uniform across platforms — read carefully): a
  build artifact that ponyc links must be both shipped by make install and
  locatable at link time, or it works in the build tree and vanishes from a
  released ponyc (the binary-relative-artifact failure mode the gencshim "Clang
  resource directory" coupling warns about). The precedents differ by platform:

  • Linux/macOS: the Makefile install target copies shipped artifacts into
    the ponyc lib dir (as it does for the crt objects), and the ELF linker
    locates them via find_ponyc_crt_dir over the program lib search path.
    Follow that precedent for the profile runtime.
  • Windows: there is no crt precedent — src/crt is Linux-gated and the
    COFF linker has no find_ponyc_crt_dir; it resolves libs by bare name over
    /LIBPATH entries. So Windows needs its own packaging + discovery: a root
    CMakeLists.txt install() rule (the pattern the clang-headers/clang-dir
    installs already use for cmake-driven Windows installs) plus a /LIBPATH
    entry for the ponyc lib dir in link_exe_lld_coff. Treat this as its own
    sub-step, not parity with crt.

  Verify a make install (and Windows make.ps1 install) tree can link a
  --coverage program.

• genexe.cc: link the profiling runtime in each of the three embedded-LLD
  linker paths — link_exe_lld_elf (Linux/BSDs), link_exe_lld_macho (macOS),
  link_exe_lld_coff (Windows). There is no external-linker path; every
  non-LLD target is already a hard error. The existing PONY_COVERAGE -lgcov
  block in link_exe_lld_elf is the model for where this slots in on ELF.

End state: llvm-cov show reports which functions ran. This is already useful
and proves the kept (non-spike) path.

Stage 2 — Control-flow regions (real, incremental)

Add region emission one construct family at a time, validating each with
llvm-cov show before moving on: if/elseif/else, then match (+ guards),
then loops, then try, then short-circuit and/or, then the
implicit-control-flow gap regions (?, return, error, break, continue).
Per-construct counterfactual tests (see below). This staging keeps each change
small and its coverage output checkable.

Stage 3 — Polish and ship

• Enforce the --coverage + --release error (the decided behavior).
• Docs: a coverage page describing the workflow and the LLVM-tools requirement;
  --help text; mention in relevant build/usage docs.
• An examples/ entry: a small program plus a README showing the
  compile/run/report workflow (per the project convention that feature plans
  include an example, and that examples directories keep a README).
• CHANGELOG entry and a .release-notes/ file (user-facing "added"; VERSION is
  not 0.0.0).

Testing strategy

Because the llvm_tools removal puts llvm-cov/llvm-profdata in every CI
build (see "Tooling availability"), the coverage tests run end to end in CI —
no special tool-acquisition step, no two-tier split. Tests at three levels:

• Fast codegen check (CodegenTest in libponyc.tests): compile a snippet
  with coverage on and inspect the module — instrprof.increment intrinsics
  present, a __llvm_covmap global present, counter count matches the
  construct. Catches instrumentation regressions without running anything.
• End-to-end smoke test (CI, full-program): compile with --coverage, run
  the binary, then llvm-profdata merge + llvm-cov export and assert the
  pipeline produces a non-empty report. This is the Add use=coverage smoke tests to CI #5528-style smoke test for
  this feature; it exercises the compile → run → profdata → cov path.
• Per-construct correctness (CI, Stage 2 onward): for each construct, a test
  where one path runs and another does not, asserting via llvm-cov export that
  the unexecuted region reports zero. This is the assertion that actually proves
  the region mapping is correct, and now that CI has the tools it is
  regression-protected rather than local-only.

The version coupling still matters: the llvm-cov used must match the
coverage-mapping format ponyc emits — which it does by construction now, since
both come from the same vendored LLVM.

Risks and open questions

• Coverage-mapping format coupling. ponyc emits a versioned format that
  llvm-cov must match. Pinned in the spike; revisited on every LLVM upgrade
  (worth a note in the upgrade-llvm process).
• Region-mapping correctness for Pony's control flow. Pony's implicit
  partiality and early exits are the most error-prone part; counterfactual
  tests per construct are the guard.
• Platform linking of the profiling runtime. The runtime has
  platform-specific files and the link must work across all three embedded-LLD
  paths (link_exe_lld_elf, link_exe_lld_macho, link_exe_lld_coff). Linux
  first; macOS and Windows validated before claiming support. On Windows,
  whether InstrProfilingPlatformWindows.c needs SDK symbols beyond what
  vcvars.default_libs / kernel32 already provide is a spike unknown (it does
  not trip the vcvars discovery coupling — adding a library to the link consumes
  existing /LIBPATH entries without changing how vcvars_get populates them).
• Resolved decisions: (1) --coverage + --release is a hard error for v1;
  (2) CI gets llvm-cov by removing the llvm_tools option and building the
  tools everywhere (the option's cache-size purpose is obsolete under GHCR
  caching), done by editing the hashed build files so the libs-cache key rotates
  cleanly.

Explicitly out of scope (future work)

• MC/DC (modified condition/decision coverage) — needs additional mapping
  metadata; separate effort.
• gcov-style coverage as an alternate --coverage format.
• A pony-cov tool that understands Pony semantics (stdlib filtering, grouping
  by package) and/or wraps the LLVM tools.
• CI coverage-service adapters (Codecov, Coveralls) via llvm-cov export -format=lcov.
• Optimized (--release) coverage.