Assertion failed '!"Inconsistent profile data"' in 'Xunit.Sdk.CollectionTracker`1+BufferedEnumerator[int]:MoveNext():bool:this' during 'Optimize control flow'

[JulieLeeMSFT]

Summary:
JIT assert 'Inconsistent profile data' in fgprofile.cpp:4821 during 'Optimize control flow' (Tier1) under fullpgo_random_gdv_edge stress mode. Affects arm, arm64, x64 Linux and x64 Windows.

Failed in (1):

• runtime-coreclr libraries-pgo 20260329.1

Console Log: Console Log

• Source: runtime-coreclr libraries-pgo / net11.0-linux-Release-arm-fullpgo_random_gdv_edge-(Debian.13.Arm32.Open)AzureLinux.3.Arm64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-arm32v7 / System.Linq.Parallel.Tests

Failed tests:

- net11.0-linux-Release-arm-fullpgo_random_gdv_edge-(Debian.13.Arm32.Open)AzureLinux.3.Arm64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-arm32v7
- net11.0-linux-Release-arm64-fullpgo_random_gdv_edge-AzureLinux.3.Arm64.Open
- net11.0-linux-Release-x64-fullpgo_random_gdv_edge-AzureLinux.3.Amd64.Open
- net11.0-windows-Release-x64-fullpgo_random_gdv_edge-Windows.10.Amd64.Open
  - System.Linq.Parallel.Tests

Error Message:

Assert failure(PID 37 [0x00000025], Thread: 43 [0x002b]): Assertion failed '!"Inconsistent profile data"' in 'Xunit.Sdk.CollectionTracker`1+BufferedEnumerator[int]:MoveNext():bool:this' during 'Optimize control flow' (IL size 33; hash 0x7873684e; Tier1)

    File: /__w/1/s/src/coreclr/jit/fgprofile.cpp:4821

Stack Trace:

clrjit!Compiler::fgDebugCheckProfileWeights
clrjit!Compiler::fgDebugCheckProfile
clrjit!Phase::PostPhase
clrjit!Phase::Run
clrjit!Compiler::compCompile
clrjit!Compiler::compCompileHelper
clrjit!Compiler::compCompileAfterInit
clrjit!jitNativeCode
clrjit!CILJit::compileMethod
coreclr!invokeCompileMethod
coreclr!UnsafeJitFunctionWorker
coreclr!UnsafeJitFunction
coreclr!MethodDesc::JitCompileCodeLocked
coreclr!TieredCompilationManager::CompileCodeVersion
coreclr!TieredCompilationManager::OptimizeMethod
coreclr!TieredCompilationManager::DoBackgroundWork

Analysis:
Same underlying assert as #118595 but triggered in different phase (Optimize control flow vs Lowering nodeinfo) and different stress mode (fullpgo_random_gdv_edge vs OSR). Likely same root cause in profile weight consistency checking.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[mangod9]

some analysis of the failure:

Why the Assert Fires

The phase sequence is:

 Line 4348:  CHECK_PROFILE enabled           ← profile checks run after every phase
   ...
 Line 4619:  PHASE_OPTIMIZE_FLOW             ← modifies flow graph
               → PostPhase: profile check    ← ASSERT fires here!
   ...
 Line 4645:  PHASE_REPAIR_PROFILE            ← repair runs, but too late

The profile consistency check runs in PostPhase after optimize flow but before the repair phase. The optimization
breaks the profile, and the check catches it before the repair can fix it.

The Specific Gap

fgUpdateFlowGraph (the core of "Optimize control flow") only marks fgPgoConsistent = false in limited cases:

 - Line 1246: Jump-to-empty-block optimization
 - Line 1621: Switch branch optimization
 - Line 2758: Branch optimization
 - Line 4785: OSR only (if (modified && opts.IsOSR()))

But general block removal, block compaction, and jump threading don't flag the profile as inconsistent, even
though they can invalidate weights. With real profile data this rarely causes visible imbalance; with random edge
counts the margins are tighter and any transformation can tip the balance.

Fix Options (Beyond Relaxing the Assert)

Fix 1: Flag inconsistency for all flow modifications (simplest, correct)

In fgUpdateFlowGraph around line 4785:

 // Current: only OSR
 if (modified && opts.IsOSR())
 {
     fgPgoConsistent = false;
 }
 
 // Proposed: all modifications that touch profile-weighted blocks
 if (modified && fgPgoConsistent)
 {
     JITDUMP("fgUpdateFlowGraph: Flow modified, profile may be inconsistent.\n");
     fgPgoConsistent = false;
 }

This is safe — the repair phase at line 4645 will run ProfileSynthesis::Run() to restore consistency. The cost is
that synthesis runs more often, but it already runs whenever ANY of the 16+ fgPgoConsistent = false sites fire.

Fix 2: Defer profile checking past the repair phase

Disable CHECK_PROFILE before the optimize flow phase and re-enable it after repair:

 // Before optimize flow
 activePhaseChecks &= ~PhaseChecks::CHECK_PROFILE;
 DoPhase(this, PHASE_OPTIMIZE_FLOW, &Compiler::optOptimizeFlow);
 // ... intervening phases ...
 DoPhase(this, PHASE_REPAIR_PROFILE_POST_MORPH, &Compiler::fgRepairProfile);
 activePhaseChecks |= PhaseChecks::CHECK_PROFILE;

This acknowledges that between optimization and repair, the profile is expected to be temporarily inconsistent.

Fix 3: Fix each transformation to maintain weights (most correct, most invasive)

When fgCompactBlock merges blocks or fgRemoveBlock removes blocks, properly redistribute weights to
successor/predecessor blocks. This is the "ideal" fix but touches many code paths and risks introducing new bugs.

Recommendation

Fix 1 is the right approach. It's a one-line change, correctly models reality (flow modifications invalidate
profiles), and the repair infrastructure already exists. The JIT's design is deferred repair — the bug is just
that fgUpdateFlowGraph fails to flag inconsistency for its general block removal/compaction paths (only OSR and
specific sub-optimizations are flagged).

The JitRandomEdgeCounts mode exposes this gap because random counts lack the natural redundancy of real profile
data, making flow transformations more likely to create visible imbalances.

[AndyAyersMS]

The model we're following is to flag each transformation that's unable to maintain profile data instead of having some blanket "we changed something" bail out. That way we can accumulate which transformations are the key culprits and know where to focus if we want to reduce the number of cases where profile becomes inconsistent.

So we should track down the specific transformation involved.

[kg]

Summary:

JIT assertion in fgprofile.cpp during Optimize control flow at Tier1 under PGO stress.

Failed in (1):

• runtime-coreclr libraries-pgo 20260408.1

Console Log: Console Log

Source: runtime-coreclr libraries-pgo / net11.0-linux-Release-arm-fullpgo_random_gdv_edge-(Debian.13.Arm32.Open)AzureLinux.3.Arm64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-arm32v7 / System.Linq.Parallel.Tests

Failed tests:

- net11.0-linux-Release-arm-fullpgo_random_gdv_edge-(Debian.13.Arm32.Open)AzureLinux.3.Arm64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-arm32v7

- net11.0-linux-Release-arm64-fullpgo_random_gdv_edge-AzureLinux.3.Arm64.Open

- net11.0-linux-Release-x64-fullpgo_random_gdv_edge-AzureLinux.3.Amd64.Open

- net11.0-windows-Release-x64-fullpgo_random_gdv_edge-Windows.10.Amd64.Open

  - System.Linq.Parallel.Tests

Error Message:

Assert failure(PID 37 [0x00000025], Thread: 43 [0x002b]): Assertion failed '!"Inconsistent profile data"' in 'Xunit.Sdk.CollectionTracker`1+BufferedEnumerator[int]:MoveNext():bool:this' during 'Optimize control flow' (IL size 33; hash 0x7873684e; Tier1)

Stack Trace:

    File: /__w/1/s/src/coreclr/jit/fgprofile.cpp:4821
    clrjit!Compiler::fgDebugCheckProfileWeights
    clrjit!Compiler::fgDebugCheckProfile
    clrjit!Phase::PostPhase
    clrjit!Phase::Run
    clrjit!Compiler::compCompile
    coreclr!TieredCompilationManager::DoBackgroundWork

Analysis:

JIT assertion in fgprofile.cpp:4821 during the "Optimize control flow" phase at Tier1 compilation under PGO stress modes (fullpgo_random_gdv_edge). Profile consistency check fires after fgUpdateFlowGraph. Both Linux (exit 134/SIGABRT) and Windows (exit -1073740286) are the same assertion. Assigned to @AndyAyersMS.

• Generated by ci-pipeline-monitor/scripts/generate_report.py *

[JulieLeeMSFT]

Summary:

JIT asserts with "Inconsistent profile data" during Tier1 compilation under fullpgo_random_gdv_edge stress mode. Manifests as SIGABRT (exit 134) on Linux and STATUS_FAIL_FAST_EXCEPTION (exit -1073740286) on Windows. Affects System.Linq.Parallel.Tests and System.Text.Json.Tests.

Failed in (1):

• runtime-coreclr libraries-pgo 20260408.1

Console Log: Console Log

Source: runtime-coreclr libraries-pgo / net11.0-linux-Release-arm-fullpgo_random_gdv_edge-(Debian.13.Arm32.Open)AzureLinux.3.Arm64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-arm32v7 / System.Linq.Parallel.Tests

Failed tests:

- net11.0-linux-Release-arm-fullpgo_random_gdv_edge-(Debian.13.Arm32.Open)AzureLinux.3.Arm64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-arm32v7

- net11.0-linux-Release-arm-instrument_if_optimizing-(Debian.13.Arm32.Open)AzureLinux.3.Arm64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-arm32v7

- net11.0-linux-Release-arm64-fullpgo_random_gdv_edge-AzureLinux.3.Arm64.Open

- net11.0-linux-Release-arm64-instrument_if_optimizing-AzureLinux.3.Arm64.Open

- net11.0-linux-Release-x64-fullpgo_random_gdv_edge-AzureLinux.3.Amd64.Open

- net11.0-windows-Release-x64-fullpgo_random_gdv_edge-Windows.10.Amd64.Open

- net11.0-windows-Release-x64-instrument_if_optimizing-Windows.10.Amd64.Open

- net11.0-windows-Release-x64-jitosr_stress_random-Windows.10.Amd64.Open

- net11.0-windows-Release-x86-instrument_if_optimizing-Windows.10.Amd64.Open

  - System.Linq.Parallel.Tests

  - System.Net.NameResolution.Functional.Tests

  - System.Text.Json.Tests

Error Message:

Assert failure(PID 37 [0x00000025], Thread: 43 [0x002b]): Assertion failed '!"Inconsistent profile data"' in 'Xunit.Sdk.CollectionTracker`1+BufferedEnumerator[int]:MoveNext():bool:this' during 'Optimize control flow' (IL size 33; hash 0x7873684e; Tier1)

    File: /__w/1/s/src/coreclr/jit/fgprofile.cpp:4821

Stack Trace:

clrjit!Compiler::fgDebugCheckProfileWeights
clrjit!Compiler::fgDebugCheckProfile
clrjit!Phase::PostPhase
clrjit!Phase::Run
clrjit!Compiler::compCompile
coreclr!CEEInfo::doAssert
clrjit!assertAbort
KERNELBASE!RaiseFailFastException

Analysis:

JIT profile weight consistency check fails under PGO stress modes. Assigned to AndyAyersMS. Same root cause as #118595 but different phase. Tracked by #126381.

Generated by ci-pipeline-monitor/scripts/generate_report.py