Rough outline: LLVM source-based code coverage for ponyc

[SeanTAllen]

Rough outline for adding code coverage support to ponyc via LLVM's source-based coverage instrumentation.

Approach

ponyc already uses LLVM 21.1.8 as its backend. LLVM has built-in source-based code coverage that inserts edge/block counters at the IR level and embeds source mapping into the binary. The infrastructure is already there — ponyc just needs to wire it up.

The alternative approaches (ptrace-based like kcov, hardware trace via Intel PT, source-level instrumentation) are all significantly more work and less capable. LLVM's coverage gives per-expression granularity, branch coverage, and ~5-15% runtime overhead.

What ponyc already has

• Debug info generation (src/libponyc/codegen/gendebug.cc, codegen.c): Full DWARF emission is enabled. DIBuilder is properly initialized, compile units are created, method/function debug info is generated, debug locations are set on instructions. This is the foundation LLVM's coverage mapping needs.

• Modern pass pipeline (src/libponyc/codegen/genopt.cc): Uses LLVM's PassBuilder with extension point callbacks (registerOptimizerEarlyEPCallback, registerOptimizerLastEPCallback, etc.), all already using the LLVM 21 ThinOrFullLTOPhase parameter. Adding the InstrProfilingLoweringPass slots in naturally.

• Extensible compiler flags (src/libponyc/pass/pass.h): The pass_opt_t struct already has release, strip_debug, verify, lint_llvm — adding coverage fits the existing pattern.

• Vendored compiler-rt source (lib/llvm/src/compiler-rt/lib/profile/): The profiling runtime source is already in the tree. ponyc already builds CRT objects (crtbeginS.o, etc.) from the vendored compiler-rt/lib/builtins/ via src/crt/CMakeLists.txt — the same approach can build the profiling runtime library.

• Embedded LLD linker (src/libponyc/codegen/genexe.cc): On Linux and macOS, ponyc drives LLD directly as a library. Adding a -lclang_rt.profile or linking the profiling .a is straightforward — the link step already handles platform-specific CRT objects and library flags.

What needs to change in ponyc

~6-7 files, estimated 50-150 lines of changes:

1. src/libponyc/pass/pass.h — Add bool coverage to pass_opt_t
2. CLI parsing (src/libponyc/options/options.c) — Add --coverage flag, following the pattern of OPT_STRIP / OPT_LINT_LLVM
3. src/libponyc/codegen/codegen.c — When coverage is enabled, ensure debug info module flags are set and potentially add coverage-specific module flags
4. src/libponyc/codegen/genopt.cc — Add InstrProfilingLoweringPass via the existing registerOptimizerLastEPCallback hook (see LLVM 21 API details below)
5. src/libponyc/codegen/genopt.cc — When --coverage is active, prevent strip_debug from being force-set in release builds (coverage requires debug info to function)
6. src/libponyc/codegen/genexe.cc — Link the compiler-rt profiling runtime. On Linux and macOS (embedded LLD paths), add the profiling library to the args. On Windows (COFF/MSVC path) and the external-linker fallback paths, add the equivalent flags
7. Build system (src/crt/CMakeLists.txt or a new src/profile_rt/CMakeLists.txt) — Build the profiling runtime from the vendored lib/llvm/src/compiler-rt/lib/profile/ source, following the pattern already used for CRT objects

LLVM 21 API specifics

The key LLVM 21 types for the pass integration:

#include <llvm/Transforms/Instrumentation/InstrProfiling.h>
#include <llvm/Transforms/Utils/Instrumentation.h>

// In the optimise() function, add via registerOptimizerLastEPCallback:
if(c->opt->coverage) {
  PB.registerOptimizerLastEPCallback(
    [&](ModulePassManager &mpm, OptimizationLevel level,
        ThinOrFullLTOPhase phase) {
      InstrProfOptions Options;
      Options.DoCounterPromotion = (level.getSpeedupLevel() >= 2);
      mpm.addPass(InstrProfilingLoweringPass(Options));
    }
  );
}

Alternatively, ponyc could pass PGOOptions (from llvm/Support/PGOOptions.h) to the PassBuilder constructor with PGOAction::IRInstr, which is how clang drives it. The manual registerOptimizerLastEPCallback approach gives more control and keeps the change minimal.

Intended workflow

# Compile with coverage
ponyc --coverage ./my-tests

# Run tests (produces default.profraw)
./my-tests

# Process and view coverage
llvm-profdata merge default.profraw -o coverage.profdata
llvm-cov show ./my-tests -instr-profile=coverage.profdata

A standalone Pony tool could later wrap the reporting step to produce Pony-specific output, but llvm-cov handles the heavy lifting out of the box.

Open question: debug info under optimization

Lines 1039-1044 in genopt.cc currently force-strip debug info in release builds:

// There is a problem with optimised debug info in certain cases. This is
// due to unknown bugs in the way ponyc is generating debug info. When they
// are found and fixed, an optimised build should not always strip debug
// info.
if(c->opt->release)
  c->opt->strip_debug = true;

This comment has been there for a while. Before proceeding, we should revalidate whether optimized builds with debug info are in fact still problematic. If the underlying issues have been fixed (or were fixed incidentally over time), coverage could work with optimized builds too. If not, coverage would be limited to debug builds — which is likely fine for test binaries anyway.

When --coverage is active, the implementation must either skip the strip_debug = true override or error out if --coverage and --release are combined. Stripping debug info would destroy the coverage mapping data.

What this gives us

• Line-level and branch-level coverage for any Pony program
• Works with test binaries to show which code paths tests actually exercise at runtime (not just which lines appear in the binary)
• Low runtime overhead (~5-15%, further reducible via LLVM 21's sampling mode)
• Uses battle-tested LLVM infrastructure rather than building custom instrumentation
• Platform support wherever ponyc + compiler-rt are available (Linux, macOS, Windows)

What this doesn't cover (potential follow-up)

• MC/DC (Modified Condition/Decision Coverage) for complex boolean expressions — LLVM 21's MC/DC support is mature, but it requires frontend work to emit MC/DC region mapping metadata, so it's a separate effort on top of the basic implementation
• A Pony-native reporting tool that understands Pony semantics (e.g., filtering stdlib, grouping by package)
• Integration with CI coverage services (Codecov, Coveralls) — would need output format adapters
• Static analysis of which lines appear in the binary vs. don't (complementary to runtime coverage, simpler tool, no ponyc changes needed)
• Sampled instrumentation mode for reduced overhead on long-running programs (LLVM 21 supports this via InstrProfOptions::Sampling, but it's not needed for the initial implementation)

[SeanTAllen]

Replaced by #5536.

That plan corrects a load-bearing error in this outline: wiring up
InstrProfilingLoweringPass and linking the runtime, on their own, produce
nothing — the pass only lowers llvm.instrprof.* intrinsics that ponyc
doesn't emit, and llvm-cov show needs a coverage mapping that ponyc has to
generate itself (DWARF isn't the foundation for source-based coverage; it's
what a gcov approach would have reused). #5536 lays out the real shape — a
coverage-mapping component in ponyc's codegen plus the plumbing — as a staged,
reviewed plan. Closing this in favor of it.