Implementation plan: compiling C shims alongside Pony (v1)

[SeanTAllen]

Implements discussion #5390. Drop a .c next to your .pony; ponyc compiles it
in-process with embedded clang and links the object in. No second build system.

Settled scope

• C only (.c). .cpp/.cc deferred (C++ runtime linking). Flag family is
  designed C/C++-shared so C++ is additive later.
• System headers in — reuse the sysroot detection the embedded linker already does.
• Dedicated pass (PASS_C), clang diagnostics routed as ponyc errors.
• No caching — recompile every build.

Divergences from current ponyc

Area	Today	This change
use flag storage	program-wide (program_t->libs)	new schemes stored per-package on package_t
package contents	only .pony parsed	.c in the package's top dir compiled + linked
LLVM build	LLVM_ENABLE_PROJECTS "lld"	"lld;clang" + clang static libs linked into ponyc
programs ponyc emits	C-ABI, libponyrt only	may include compiled-C objects

---

Stage 1 — build & link clang (standalone PR, merge-gated behind Stage 2)

Goal: ponyc builds with clang's libraries linked in and the resource dir shipped.
No feature yet; no behavior change.

1. lib/CMakeLists.txt:183: "lld" → "lld;clang".
2. Root CMakeLists.txt: add find_package(Clang CONFIG ...) mirroring the LLVM/LLD
   discovery at :27-28; define PONYC_CLANG_LIBS. Prefer clang's exported CMake
   targets (clangFrontend/CodeGen/Driver/Parse/Sema/Analysis/AST/Basic/Lex/
   Serialization/Edit/Support) so transitive deps resolve; fall back to a hardcoded
   list like PONYC_LLD_LIBS (:389) if the export isn't usable.
3. src/ponyc/CMakeLists.txt:75: add ${PONYC_CLANG_LIBS} to target_link_libraries.
4. Expose the clang version to ponyc's C++ for the resource-dir path: include
   clang/Basic/Version.h (CLANG_VERSION_MAJOR/MINOR) or add a compile def
   mirroring LLVM_VERSION at CMakeLists.txt:416.
5. Ship clang's resource dir. It installs to build/libs/lib/clang/<ver>/include.
   Ensure the ponyc install/release packaging carries it so it sits at a known
   offset from the ponyc binary (same idea as the bundled packages/ dir).
6. Windows: make.ps1 always rebuilds from source — confirm the clang project
   builds on that path too.

Verify: ponyc builds & links; make test-core green (debug + release); binary
runs. Measure & report: make libs build-time delta, ponyc binary-size delta,
CI wall-clock delta. These are the costs the discussion's "one line" understates.

---

Stage 2 — the feature

2a. Directive schemes — src/libponyc/pkg/use.c, pkg/program.cc

• Register cinclude:, cdefine: in handlers[] (use.c:32-40) with
  allow_guard=true, allow_name=false — so ... if macosx works for free.
  (cstd: and cflag: are both dropped from v1: cstd: is sugar over -std —
  clang's default dialect compiles normal C glue; cflag:, a raw escape hatch,
  reintroduces a flag-precedence footgun against the locked target flags. Add either
  back if demand shows.)
• Handlers get the package via ast_nearest(use, TK_PACKAGE) → ast_data and store
  per-package (not program-wide). cinclude: resolves relative paths against the
  package dir exactly like use_path (program.cc:168-173); it's additive (a search
  list), so multiple includes never conflict — no check needed.
• Value grammar — do not reuse quoted_locator. lib:/path: run their locator
  through quoted_locator (program.cc:114-135), which rejects any value containing
  INVALID_LOCATOR_CHARS ("' \()$|&<>..., program.cc:109-111). C defines need exactly those — cdefine:VERSION="1.2"has quotes, include paths have spaces — socinclude:/cdefine:must bypass it: store the value raw and pass it verbatim as one clang argv element. Safe, because clang takes an argv array, not a shell command — no injection surface.cdefine:still parses the macro *name* (text before=) for the duplicate check; the value after =` is opaque.
• cdefine: duplicate check: error if two active cdefine: directives define
  the same macro name — whether the values differ (FOO=1 vs FOO=2) or match
  (FOO=1 twice). A macro name is a declaration; redeclaring it is a mistake, like
  let a = 1 twice — deterministic ordering would otherwise let one silently shadow
  the other. Macro name = text before =; on append, scan the package's c_defines
  for a same-name entry (the strlist_find pattern use_library uses). Point the
  error at the redefining directive, ideally at the original too (error frame); name
  the prior value when it differs. This diverges from use "lib:", which dedups
  exact duplicates — linking a lib twice is harmless, redeclaring a macro is not.
  Guards resolve cross-platform cases for free: a guarded-out directive never reaches
  the handler (guard evaluated first, use.c:138-142), so cdefine:FOO=1 if linux /
  cdefine:FOO=2 if windows aren't a conflict — only active-target defines are checked.

2b. Per-package storage — src/libponyc/pkg/package.c:45-60

• Add to package_t: strlist_t* c_includes, c_defines, c_sources.
• create_package (:535-583) initializes every field by hand — there is no
  zero-init — so each new field needs an explicit = NULL there.
• Free in package_free (:1074-1081), not package_done (which only frees
  global package state). Add strlist_free for each of the three strlists, or they
  leak per package.

2c. .c discovery — src/libponyc/pkg/package.c:201-267

• In parse_files_in_dir, collect files ending in .c into a separate bucket
  and append their full paths to c_sources ((package_t*)ast_data(package)).
  They must not join the entries[] array that gets fed to parse_source_file
  (:262) — that's the Pony parser and would choke on C. The extension logic is
  safe (strrchr+strcmp(".c") won't match .cc/.cpp; a bare .c hidden file
  is already skipped by the name[0]=='.' guard). Keep the deterministic sort.
  Single-directory only — vendored C in subdirs is untouched, as intended.
• No-source-files guard — leave it (decision, reverses an earlier note).
  package.c:1056-1061 errors "no source files in package" when
  ast_child(package) == NULL. A .c adds no Pony AST, so a directory with only .c
  and no .pony stays an error. Keep that. A shim is C that travels with a Pony
  package; every real shim sits next to .pony, which satisfies the guard. Making a
  C-only directory a valid package is a new concept nobody asked for — don't introduce
  it quietly (easier to give than take away). Worth a small message tweak — "no Pony
  source files in package" — so someone who dropped only a .c gets a clear reason.

2c-bis. Existing full-program test collision (silent shadowing, not a link break)

Five test/full-program-tests/ programs ship additional.c next to main.pony,
built into a <name>-additional library by test/full-program-tests/CMakeLists.txt
(GLOB_RECURSE *.c *.cc; lib name from the first path component) and linked via
use "lib:<name>-additional": c-callback, ffi-call-in-initializer,
ffi-different-returns, ffi-return-arg-reachable, identity-digestof-object. Once
.c discovery lands, ponyc would also compile each additional.c as a shim object.
Correction to an earlier claim in this plan: this is not a link failure. A shim
.o vs a use "lib:" library is object-vs-library — the object's symbols win and the
library is never pulled (a static archive's members aren't pulled once the symbol is
defined; a shared lib's symbols are overridden). The build still passes; it just
silently links the shim instead of the lib. That's the real harm: these tests exist to
cover the use "lib:"-links-a-built-lib path, and they'd quietly stop exercising it —
a silent loss of coverage, worse than a loud failure. (Only .c collides in v1; .cc
isn't discovered yet, so C++ fixtures are safe until the C++ stage — note it for then.)

These are external-lib FFI tests, not shims, and the "use "lib:" links a
separately-built lib" path must stay covered. So keep them as external-lib tests
and move their .c out of the scanned package dir:

• git mv <test>/additional.c <test>/c-src/additional.c for the five.
• The cmake needs no change for the move: GLOB_RECURSE still finds the subdir
  .c, and the lib name derives from the first path component (<test>), so
  <test>-additional is unchanged and the use "lib:" line in main.pony is
  untouched. ponyc's single-dir discovery no longer sees the .c.
• New shim full-program tests are the inverse problem: their .c must be built
  only by ponyc, never by the cmake glob (else it's double-built and collides).
  Scope GLOB_RECURSE to skip the shim tests (a shims/ subtree the glob excludes,
  or place them where the glob doesn't reach). This is the cmake change.

2d. The pass — src/libponyc/pass/pass.h, new genc.c

• Add PASS_C to the pass_id enum (pass.h:209-232) between PASS_FINALISER and
  PASS_REACH — before the codegen group, not among it. REACH is the boundary where
  the shared AST-pass driver (ast_passes_program, run by both real builds and the
  test harness) hands off to the split codegen drivers (codegen() /
  codegen_gen_test()). Below REACH keeps PASS_C on the shared side: one call site,
  fail-fast, and --pass c stops for free (generate_passes returns before codegen
  when limit < PASS_REACH, pass.c:376). It needs nothing reach produces.
• Add its name ("c", unused today) in pass_name (pass.c:50-77); re-check the
  PASS_ALL <= AST_FLAG_PASS_MASK static assert (ast.c:64) — headroom fine
  (mask 0x1F=31, PASS_ALL 20→21).
• Update PASS_HELP (pass.h:236-258) — a hardcoded --pass help string flagged
  "update when pass_id changes." Insert " =c\n" in pass order (before =reach).
  Otherwise the static help (options.c:175) diverges from the dynamic listing
  (print_passes, options.c:194-214), which picks up the new pass automatically.
• KNOWN COUPLING (AGENTS.md): inserting PASS_C before REACH renumbers REACH and
  everything after it. The Pony mirror tools/lib/ponylang/pony_compiler/.../pass.pony
  uses hardcoded integers — add PassC and renumber Reach 14→15, Paint→16,
  LLVMIR→17, Bitcode→18, ASM→19, Obj→20, All→21, plus the union/primitive entries.
  Run make test-pony-compiler test-pony-lint test-pony-lsp test-pony-doc.
• New genc.c: bool genc(ast_t* program, pass_opt_t* opt) — takes (program, opt),
  not compile_t: clang builds its own target from opt->triple/cpu/features, writes
  the .o to the output dir, reads each package's flags off ast_data, and records the
  object paths on program_t. Needing nothing from the LLVM compile_t is exactly what
  lets it run before codegen. Walk the package chain (ast_child(program) then the
  ast_sibling chain — all packages); for each package_t with c_sources, compile
  each source with clang in-process:
  • target (from pass_opt_t, identical to Pony codegen): opt->triple,
    -mcpu=opt->cpu, target-features opt->features, PIC if opt->pic,
    -O0/-O2 per opt->release, debug per debug build, ABI opt->abi.
  • user flags (per package): -D<c_defines>, -I<c_includes> (resolved). No
    -std/cflag: in v1.
  • headers: clang resource dir (builtins) + ponyc-derived system include dirs (2f).
  • Emit-object to a .o in the output dir (suffix_filename pattern, codegen.c:1233
    — it currently takes compile_t; factor a variant taking opt, or build the path
    from opt directly).
  • Route clang's DiagnosticConsumer into opt->check.errors (via errorf with
    file/line) so clang errors become ponyc errors. [UNVERIFIED — verify in the spike,
    2f]. The whole error story and the compile-error tests (which assert on
    errors_t contents) rest on intercepting clang diagnostics programmatically. It's
    the API libclang/clangd are built on, so it's believed-true — but it's never been
    run here (clang is unbuilt until Stage 1). Don't let Stage 2 stand on it unverified.
  • Wrap each TU's emit-object in an LLVM CrashRecoveryContext (the guard libclang
    uses for in-process compilation). A clang ICE / report_fatal_error / assertion ends
    in abort() (llvm/.../ErrorHandling.cpp), and in-process that's ponyc dying —
    "Aborted (core dumped)," no message, no hint a shim was involved, filed as a ponyc
    bug. With the recovery context the operation fails instead of the process: report
    "internal error compiling <shim>.c" through errors_t, fail closed, with
    attribution. A few lines around the per-TU call. Triggers are rare (pathological C,
    clang bugs) but the failure shape is the worst available.
  • Record each .o path on program_t; add program_c_object_count/_at accessors
    mirroring program_lib_count/_at (program.cc:390-412).
• Single call site: invoke genc from ast_passes_program (pass.c:335) after the
  ast_passes call, gated if(opt->limit >= PASS_C). Both the real build (main.c:56
program_load → ast_passes_program) and the test harness (util.cc:575) run
  ast_passes_program, so one call site covers both — no edits to codegen() or
  codegen_gen_test(), no --pass c early-return.
  • Compile-error tests see clang errors here (test_expected_errors runs
    ast_passes_program); the existing error-count/substring assertions work unchanged.
  • Front-end tools (lsp/doc/lint) stop at limit ≤ FINALISER (< PASS_C), so the gate
    skips genc — they never invoke clang. Deliberate: shim C errors do not
    surface in-editor in v1. genc emits objects, which is wrong for an editor (.o
    spam per keystroke, possibly no output dir, half-edited code); in-editor errors
    need a diagnostics-only clang mode (-fsyntax-only, no codegen). That mode, if
    ever wanted, is a separate LSP path independent of where PASS_C sits — so this
    choice neither forecloses in-editor errors nor forces a future enum renumber.
  • Object lifetime / --pass obj contract: genc runs for any limit ≥ PASS_C, so
    --pass c, --pass obj, etc. each produce shim .os in the output dir. They're
    cleaned up only on a successful PASS_ALL link (alongside the Pony object,
    genexe.cc:2198-2202). Under any non-link mode (--pass obj/asm/ir/c) the shim
    .os persist — same as the Pony .o under --pass obj, which is the point of
    those modes (hand them to your own linker). A failed link leaves them too
    (cleanup is success-only). Stated so it's a contract, not an accident.

2d-bis. Determinism (reproducible builds)

The -I/-D flag order and the .c→.o compile/link order are deterministic by
construction — the plan's job is to state it and protect it, not add machinery:

• cinclude:/cdefine: accumulate as pass_scope visits TK_USE nodes (scope.c:365)
  in AST order = sorted-file order (parse_files_in_dir qsort) + source order within a
  file. They append to per-package strlists, which preserve insertion order
  (ponyint_list_append tail-appends, list.c:25; iteration is head→tail). genc reads
  them in that order. This is the same deterministic path use "lib:" already uses.
• .c discovery uses the same qsort, so multiple .c compile and their .o append to
  the link in a stable order.
• Coupling (point of breakage): the guarantee rests on (a) parse_files_in_dir's
  sort and (b) appending in visit order + reading in insertion order. A comment at the
  discovery/append sites should say so.
• Test: factor the clang-arg assembly so a unit test can hand it a package with
  several cinclude:/cdefine: directives and assert the emitted order is stable.

2e. Link append — src/libponyc/codegen/genexe.cc

• There is no shared pre-dispatch — link_exe (:2020) just routes to the three
  platform linkers. All three must be edited: ELF (file_o at :1341), MachO
  (:1770), COFF (:1919). After the existing file_o + user-libs sequence, append
  program_c_object_count/_at paths to the LLD arg vector. COFF differs
  structurally (file_o precedes the lib search paths), so place the append with
  platform-specific care there.

2f. System include paths — src/libponyc/codegen/genexe.cc

• The reusable piece is the sysroot string — resolve_sysroot (:836) +
  opt->sysroot. Note find_libc_crt_dir and its neighbors find library/CRT
  dirs, not include dirs, so there's nothing there to factor for headers; deriving
  the system include dirs from the sysroot (<sysroot>/usr/include,
  /usr/include/<triple>, the macOS SDK path) is new code — the compile-side
  mirror of the link-side path work, sharing only the sysroot detection. These
  helpers are static in genexe.cc, so expose them or relocate the sysroot logic to
  a shared spot genc.c can call.
• DECIDED: hand-roll (i), like LLD. We don't drive clang's Driver: the clang
  we embed is vanilla upstream, and the per-distro toolchain knowledge that makes a
  distro's own clang reliable lives in that distro's downstream patches, which we
  don't carry. The upstream GCCInstallationDetector is generic best-effort and can
  disagree with the libc our linker already picked (musl vs glibc), so we own the
  include-path discovery off the same sysroot the linker uses — keeping compile and
  link consistent by construction. macOS SDK / Windows MSVC discovery is the corner
  to watch.
• Spike (before Stage 2, once Stage 1's clang libs link): a ~50-line probe that
  (1) compiles a deliberately broken shim.c in-process with a custom
  DiagnosticConsumer and confirms the error arrives structured (severity/file/line) in
  errors_t; (2) compiles a good shim.c and confirms a linkable .o comes out;
  (3) pokes Windows MSVC/SDK include discovery. Converts the §2d [UNVERIFIED]
  diagnostic-interception assumption to verified before genc.c is written — week-one
  knowledge, not week-ten.

2g. Tests (how to run)

• Compile-error tests — make test-pony-compiler. New fixture: place a .c on
  disk (a temp package dir, or a magic package mapped to a temp dir) since
  inline-source fixtures can't carry a .c. Assert via test_expected_errors that a
  bad shim yields the expected clang-derived error text. This fixture mechanism is
  the new piece of discussion open-question Prevent names differing only by case #3.
• cdefine: duplicate test — make test-pony-compiler. This one is cheap: the
  check fires in pass_scope (the use handler), needs no .c on disk and no clang, so
  it's plain inline source with two same-name cdefine: directives, run to the scope
  pass, asserting the error — one case for differing values, one for identical values
  (both error). Add a passing case: distinct names, and guarded cross-platform defines
  that don't clash.
• Full-program success test — make test-core (debug and release). A package
  with a tiny shim + Pony calling it over FFI; compile + link + run; assert output.
  Goes through the real codegen/genexe path, exercising the link append.
• Counterfactual: break each new assertion to confirm it fires.
• Deferred (named, not silent): no full-program expected-build-failure runner mode.
  Compile errors are now gtest-assertable via test_expected_errors; link-class failures
  (e.g. duplicate symbols across two shims) stay untestable in the full-program runner —
  the same gap as today. Deferring is fine; naming it so it isn't a silent omission.

2h. Examples

examples/ffi-struct and examples/ffi-callbacks ship struct.c/callbacks.c next to
their .pony, link a hand-built lib via use "lib:..." + use "path:./", and their
READMEs document a manual C build step. v1 discovery would compile the .c as a shim and
silently shadow the hand-built lib (object-vs-library, same mechanism as 2c-bis).

• Keep both as external-lib FFI examples — they teach FFI + use "lib:" linking of a
  prebuilt lib, which stays a real, distinct thing users do (not everything is a shim).
  Apply the same fix as the full-program tests: relocate struct.c/callbacks.c into a
  subdir (e.g. c-src/) so single-dir discovery skips them. use "lib:"/use "path:./"
  stay as-is; update each README's manual build step to point at the new .c location.
• Add a dedicated shim example (e.g. examples/cshim): a .c next to its .pony,
  called over FFI, with use "cinclude:..."/use "cdefine:..." — ponyc compiles it, no
  manual build, no use "lib:". This is the feature's showcase. Update
  examples/README.md (pony-examples-readme conventions) for the new example and the
  relocated FFI sources.

2i. Release notes + docs

• .release-notes/compile-c-shims.md — ## title matching the PR title, user-facing
  description with a code example. VERSION is 0.64.0 (released), so this is required.
• Document the bare-object semantics (the release note is the user-facing home):
  shim .cs are compiled to objects and linked directly, not archived — so (a) C
  constructors (__attribute__((constructor))) run, and (b) two shims defining the same
  external symbol are a loud duplicate-definition link error (object-vs-object),
  whereas a shim shadowing a use "lib:" library is silent (object-vs-library).
  Include the migration warning: a .c you previously built into a lib and linked
  via use "lib:" is now auto-compiled as a shim and silently shadows that lib — move it
  to a subdir or drop the lib. (This is the user-facing twin of the 2c-bis/2h migrations.)
• Cross-platform pattern (docs): use guards condition the flags (cinclude:/
  cdefine: … if macosx); discovery has no per-file guard, so per-OS shim code uses
  whole-file #ifdef — every .c is always compiled, and a platform-specific file
  #ifdefs its whole body to an empty object elsewhere. That is the design; say so.
• cdefine: vs ponyc -D (docs): cdefine:FOO is a C-preprocessor macro (clang
  -D), distinct from ponyc's own -D/--define build flags that drive Pony's ifdef.
  Same word, different machines — disambiguate it.
• Tutorial coverage of the new use schemes lives in ponylang/pony-tutorial — a
  separate-repo follow-up, noted in the PR.

---

Merge sequencing

Stage 1 is reviewed but held; it merges immediately before Stage 2 so main never
carries an unused clang build. If the feature is abandoned, neither merges and
there's nothing to back out.

Decisions (resolved)

1. System include paths — hand-rolled from the sysroot, like LLD (2f). Not clang's driver.
2. cstd: and cflag: — both dropped from v1 (2a). cstd: is sugar over -std;
   cflag: reintroduces a flag-precedence footgun against the locked target flags.
3. Resource-dir packaging — a Stage-1 build/release task (not a fork): verify the
   artifacts carry lib/clang/<ver>/include at a known offset from the binary on
   every platform.
4. Merge sequencing — Stage 1 held until Stage 2 is ready.
5. PASS_C placement — after FINALISER, before REACH (2d). Build-time C errors only;
   no in-editor (LSP) shim diagnostics in v1 (would need a separate diagnostics-only path).
6. C-only packages — not valid (2c). A package needs at least one .pony; reverses
   an earlier note that relaxed the guard.
7. cdefine: duplicates error (2a) — any same-name redefinition, conflicting or
   identical value, like let a twice. cinclude: is additive, no check. Directive
   values bypass quoted_locator, so quotes/spaces (-DVERSION="1.2") work.
8. Examples (2h) — keep ffi-struct/ffi-callbacks as external-lib FFI examples
   (relocate their .c like the tests, dodge discovery); add a new examples/cshim to
   showcase the shim feature.

Assumptions to verify (Stage-1 spike, before Stage 2 — 2f)

• [UNVERIFIED] diagnostic interception. The error story and the compile-error
  tests rest on clang's DiagnosticConsumer → errors_t working in-process. Believed-
  true (libclang/clangd do it), never run here. The spike confirms it — plus a good
  shim → linkable .o, plus Windows include discovery — before genc.c is written.