feat(compiler): deterministic compilation and byte-diff drift check#44
Merged
Conversation
Emit lifted Var fields, override methods, and fn interface implementations in a deterministic order. Each was driven by a collection keyed on identity-hashed objects (Var and MethodInfo maps/sets, Type sets), whose iteration order varies per process, so two compiles of the same source produced different member order. Sort lifted Vars by name, override methods by signature, and fn Magic.Function interfaces by type name. Member order is semantically free, so a stable content-derived order is safe.
AssemblyBuilder.Save stamps a random module MVID and the current time into every emitted DLL. Once member emission is deterministic these are the only bytes that differ between two compiles of the same source, and .NET Framework exposes no API to set them. Patch the saved file: zero the PE TimeDateStamp and set the MVID to a hash of the file content, keeping it unique per assembly and stable per source.
Recompile the committed bootstrap and stdlib DLLs with the deterministic compiler and regenerate the magic-unity-dual variant. Every DLL changes once as member order and assembly headers settle onto their canonical form; subsequent compiles are byte-stable.
Exercise normalize-assembly! directly: normalize a committed DLL to its canonical form, dirty the PE timestamp, re-normalize, and assert the bytes are recovered exactly. Recovering the full bytes from only a timestamp change also proves the MVID is re-derived from content at the correct #GUID heap offset, guarding the PE / CLI offset parsing against regressions the behavioral suite cannot catch.
…ts (#43) Compilation is deterministic, so check-drift now byte-diffs the committed DLLs against a rebuild instead of hashing sources. This catches stale DLLs whatever the trigger (stdlib, compiler, or C# runtime change), which the manifests could not see. Deletes both manifests, magic.drift, and the -revert-byproducts machinery. The two Export runtime DLLs embed a git-describe SourceRevisionId, so they are restored from HEAD instead of diffed.
#43) Loading refresh.clj consumes process-global gensyms before the stdlib namespaces compile, so the manifest-code removal renumbers G__N names in 11 DLLs. Assembly-internal renames only.
Moves the drift-check rationale from bb.edn comments into docs/, including the contributor edge cases.
Byte-identity across machines required three fixes, each isolated by IL-diffing the same namespace compiled on macOS and Linux: emission sorts compare raw UTF-16 units instead of the culture-sensitive String.CompareTo; *file* is bound to the load-relative source path so def :file metadata matches the JVM instead of baking in the checkout directory; the global gensym counter is reset per file-writing compile unit so emitted names carry no boot or driver-script state. Also fixes refresh.clj top-level detection for a (ns alone on its line, which had exempted clojure.repl from refresh since the beginning. A Linux mono 6.12 container reproduces all 73 reference DLLs byte-for-byte from a macOS mono 6.14 tree.
skydread1
force-pushed
the
feat/deterministic-compilation
branch
from
July 14, 2026 01:34
3aacf96 to
76dbb03
Compare
…me normalization (#43) Emission is machine-independent, so CI runs bb build + the full bb check-drift byte-diff on the committed DLLs. load-one picks source vs DLL by mtime and a fresh checkout is ambiguous, so build / build-magic / refresh-stdlib set each committed DLL 1s newer than its source when the source still hashes to its dll-sources.edn entry, and 1s older when it was edited, so a real edit still recompiles. dll-sources.edn is one source-SHA manifest covering the stdlib and bootstrap sets uniformly, re-recorded by refresh-stdlib and build-bootstrap; bb check-drift skip-dlls stays as the fallback for an environment that fails to reproduce the bytes, checking the manifest alone. Docs updated for the cross-machine guarantees.
Every DLL re-emits once: ordinal member order, load-relative :file strings, unit-scoped gensym numbers. clojure.repl is compiled by refresh for the first time.
skydread1
force-pushed
the
feat/deterministic-compilation
branch
from
July 14, 2026 01:54
76dbb03 to
f9e5b02
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #43
.clj.dllon any machine. Member emission order (lifted Var cache fields,reify/proxy/deftypeoverride methods, theMagic.Functioninterface set) sorts by content-derived keys compared ordinally (the CLR's default string compare is culture-sensitive, so order followed the OS collation),*file*is bound load-relative so the checkout path stays out of the bytes, the gensym counter is reset per file-writing compile unit, and the assembly MVID and PE timestamp are rewritten from content after save. A Linux mono 6.12 container reproduces all 73 reference DLLs byte-for-byte from a macOS mono 6.14 tree.bb check-driftbyte-diffs the committed DLLs against a rebuild, in CI too. A stale DLL is flagged whatever the trigger (stdlib source, compiler, or C# runtime change), the case source hashes were blind to (fix(stdlib): restore clojure.spec.alpha regex ops and fix keys* :req-un on empty input #42 shipped that way). Thestdlib-manifest.edn/bootstrap-manifest.ednsource hashes are replaced by onedll-sources.edn, kept as the portable signal behindbb check-drift skip-dlls, the fallback for an environment that fails to reproduce the bytes.-revert-byproductsis gone: deterministic rebuilds leave no byproduct churn.magic.test.deterministicanddocs/deterministic-compilation.md.