Derive header prerequisites with -MMD -MP instead of listing them by hand

[Kenneth-Javier]

In #1284 you asked for the engine list in test_makefile_deps.py to be derived from the Makefile rather than kept by hand, so that "there is no list to keep in sync". This proposes the same idea one level down: let the compiler derive each engine's header prerequisites.

Why it matters. Each engine rule in c/Makefile lists its headers on one hand-written line. Anyone adding a header edits that line, and #1284 showed what happens when someone doesn't: make reports success and leaves a stale binary. Its test now catches the drift, but the line itself keeps moving. On dev over the last 30 days, 30 of the 132 commits touching c/Makefile changed an engine prerequisite line, by 17 different authors. Every open PR that touches the same rule conflicts on it. My own #1261 resolved that same line in 9 of its 20 merges from dev.

None of this is hard, but all of it takes attention. Each conflict needs someone to stop, reconcile two versions of one long line and wait for CI again, and each forgotten header is a bug nobody sees until a stale binary misbehaves. That cost lands on contributors and on the maintainer reviewing their merges, over and over, for work that is purely mechanical. Once the prerequisites are generated, the line no longer exists, so there is nothing to conflict on and nothing to forget.

The change. Add -MMD -MP to the compile flags and -include the generated .d files. The engine rules then only name their sources, objects and .build-config. The compiler records every header actually included, -MP keeps make from failing when a header is deleted, and the .d files join clean.py.

Checked with the project's MinGW gcc on a minimal copy of the patterns used here:

case	result
compile and link in one command, like colibri$(EXE)	.d written with the right headers
a test that #includes ../engine.c	the engine's headers follow through the include
a header changes, no hand-written list	make rebuilds
a header is deleted	make still succeeds, thanks to -MP

Not checked, and where I'd expect friction: the nvcc and MSVC-hosted CUDA builds; Clang on macOS, which supports the flags, but I haven't run it; Makefile.deepseek-v4.units; and what test_makefile_deps.py should become. It might check that the .d files exist and cover each engine's sources, rather than compare two hand lists.

If this is a direction you'd want, I'm happy to prototype it as a PR. If the hand lists are deliberate for a reason I'm not seeing, that's useful to know too.

[JustVugg]

Yes, this is a direction I want, and a prototype PR is welcome.

Your numbers hold on dev: over the last 30 days, 27 non-merge commits by 16 authors changed an engine's NAME$(EXE): line, and 20 merges carry a hand-resolved hunk on one. The latest was tonight: #1356 and your #1261 conflicted on exactly the colibri line.

I checked the one-command case with GCC 13 and it behaves as you describe, including a test that includes ../engine.c. One case does not, and it matters because it silently brings back the stale binary this is meant to remove. When a single command compiles more than one .c, GCC writes only the last translation unit's dependencies, with or without an explicit -MF:

$ gcc -MMD -MP eng.c helper.c -o eng2 && cat eng2.d
eng2: helper.c

eng.c includes a.h, and a.h is not in the file. Four recipes in c/Makefile compile that way today: test_segment_runtime, test_segment_conformance, test_edge_runtime and dsv4_decode_loader_test.

What I'd ask of the PR:

1. The multi-source recipes either compile to separate objects or keep an explicit list, and the new test fails if another one appears.
2. -MMD -MP go into CFLAGS. CFLAGS is part of BUILD_CONFIG, so the first make after the change rebuilds everything and writes the .d files. In a separate variable, a binary built before the change would stay "up to date" against headers edited after it.
3. First slice: engines and tests built by gcc and clang only. The nvcc recipes, the MSVC-hosted CUDA build and Makefile.deepseek-v4.units keep their lists for now, as you already flagged.
4. macOS proven in CI, with Apple clang and the system make 3.81 (make: .build-config is never written under GNU Make 3.81, so every colibri build relinks #1732 is the reminder that it is still out there). I have no clang on this box, so that case is unverified on my side.
5. clean.py removes the .d files, and test_makefile_deps.py becomes: after a build, each engine's .d exists and covers every #include "..." of its source. The other tests that read the Makefile look at rule names, not headers; test_glm53_metal_source.py slices the text between two rule names, so those names should stay.

On timing: this touches every engine rule, so it will conflict once with every open PR that edits the Makefile. Right after 1.12.1 is the cheapest moment for that, so whenever you are ready.

[Kenneth-Javier]

Thanks, and good catch on the multi-source recipes. My probe only covered one .c per command, and I've reproduced exactly what you describe. I'll take the prototype on this spec: -MMD -MP in CFLAGS, gcc/clang only in the first slice, the multi-TU recipes split or kept explicit behind a guard test, and test_makefile_deps.py rewritten as a post-build coverage check. PR to follow.

[Kenneth-Javier]

PR is up: #1758. It covers the five points above: header lists gone from 202 rules, the multi-source recipes split into objects with a test that fails on a new one, and macOS (Apple clang, Xcode's make) green in CI. The description lists what was verified where, and the little that isn't.

[tayfuryldz]

This looks cleaner than maintaining another dependency list by hand. The one thing I'd test in the real Makefile before switching is the compile+link-in-one-command case, because dependency-file naming gets awkward when there isn't a normal per-source object target.

If the generated dependency path is deterministic for every engine and clean.py removes it, -MMD -MP seems like a better source of truth than the long prerequisite lines. I'd also keep one regression test that touches a header and verifies the engine rebuilds.