Skip to content

[PROOF / DO NOT MERGE] Release determinism CI is broken on main#19859

Closed
T-Gro wants to merge 2 commits into
mainfrom
prove-release-fails-from-main
Closed

[PROOF / DO NOT MERGE] Release determinism CI is broken on main#19859
T-Gro wants to merge 2 commits into
mainfrom
prove-release-fails-from-main

Conversation

@T-Gro
Copy link
Copy Markdown
Member

@T-Gro T-Gro commented May 29, 2026

Purpose

Evidence-only draft. This branch contains no product changes — only a CI configuration flip from Debug to Release for the Determinism job, with 8 parallel iterations to maximize the probability of catching probabilistic races on a single CI run.

Hypothesis

Multiple non-determinism races exist on current main, not visible because the existing CI Determinism_Debug job disables the optimizer (SetOptimizeOff for Debug):

  1. DetupleArgs / TLR (F# compiler produces non-deterministic metadata #Strings heap layout #19732) — parallel optimizer iterates Val sets by racy Val.Stamp (Interlocked.Increment globally assigned)
  2. IlxGen.TypeDefsBuilder.AddTypeDef — counter-based type ordering races under parallel codegen
  3. GrabExtraBindingsToGenerateConcurrentStack drain order is non-deterministic
  4. NiceNameGenerator — global singleton counter across other callers
  5. FileIndex — assignment race under parallel parse

If any of these fire in any of the 8 iterations, we see hash mismatches on FSharp.Compiler.Service.dll etc.

Why this matters

PR #19810 fixes (1) but cannot land confidently while (2)-(5) remain hidden. This PR establishes the baseline: Release determinism CI currently fails. Subsequent PRs (#19810 + follow-ups) must drive it back to green by fixing the product, not by reverting CI to Debug.

What to look at

  • Determinism_Release iter1..iter8 job results
  • Search build logs for != (hash mismatch) and Files have different content

Related: #19732, #19810

This draft demonstrates that the current main branch is non-deterministic
in Release configuration. 8 parallel iterations maximize the probability
that pre-existing races (TypeDefsBuilder counter, ConcurrentStack drain,
parallel optimizer Val.Stamp iteration order) trigger at least once.

DO NOT MERGE. Evidence-only PR for #19732 / #19810.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions
Copy link
Copy Markdown
Contributor

✅ No release notes required

@T-Gro
Copy link
Copy Markdown
Member Author

T-Gro commented May 31, 2026

✅ PROOF VALIDATED

Build 1440659:

  • 6 of 8 iterations PASSED (no race fired)
  • iter4 and iter7 FAILED with:
    Determinism failed for the following binaries:
    	FSharp.Compiler.Service.dll (old hash: 0C1A3BE3C55CCF242B84FB3D2DE10944; new hash: F6DD537FB21EB456F27BE5BB66E3107D)
    

This confirms:

  1. Main is non-deterministic in Release. The race is probabilistic (fires ~25% per build).
  2. The Determinism_Release CI job catches it reliably with multiple iterations (maxParallel: 8).

Fixes are in PR #19810.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

1 participant