Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assertion failed 'change >= 0' #100350

Closed
BruceForstall opened this issue Mar 27, 2024 · 4 comments · Fixed by #100385
Closed

Assertion failed 'change >= 0' #100350

BruceForstall opened this issue Mar 27, 2024 · 4 comments · Fixed by #100385
Assignees
Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-clean-ci-optional Blocking optional rolling runs JitStress CLR JIT issues involving JIT internal stress modes

Comments

@BruceForstall
Copy link
Member

https://dev.azure.com/dnceng-public/public/_build/results?buildId=620358&view=ms.vss-test-web.build-test-results-tab

libraries-jitstress test failure
System.Text.RegularExpressions.Tests Work Item
seemingly every platform

----- start Wed 03/27/2024  7:21:47.72 ===============  To repro directly: =====================================================
pushd C:\h\w\A5880934\w\9C2708EB\e\
"C:\h\w\A5880934\p\dotnet.exe" exec --runtimeconfig System.Text.RegularExpressions.Tests.runtimeconfig.json --depsfile System.Text.RegularExpressions.Tests.deps.json xunit.console.dll System.Text.RegularExpressions.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\A5880934\w\9C2708EB\e>"C:\h\w\A5880934\p\dotnet.exe" exec --runtimeconfig System.Text.RegularExpressions.Tests.runtimeconfig.json --depsfile System.Text.RegularExpressions.Tests.deps.json xunit.console.dll System.Text.RegularExpressions.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Text.RegularExpressions.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.RegularExpressions.Tests (found 327 of 358 test cases)
  Starting:    System.Text.RegularExpressions.Tests (parallel test collections = on [4 threads], stop on fail = off)
    System.Text.RegularExpressions.Tests.RegexCompileToAssemblyTests.CompileToAssembly_SimpleUseInDebug [SKIP]
      Condition(s) not met: "IsDebugAndRemoteExecutorSupported"

Assert failure(PID 7060 [0x00001b94], Thread: 904 [0x0388]): Assertion failed 'change >= 0' in 'System.Text.RegularExpressions.CompiledRegexRunner:Regex5982_TryMatchAtCurrentPosition(System.Text.RegularExpressions.RegexRunner,System.ReadOnlySpan`1[ushort]):ubyte' during 'Profile incorporation' (IL size 77119; hash 0x346865ee; Tier0-MinOpts)

    File: D:\a\_work\1\s\src\coreclr\jit\fgprofilesynthesis.cpp:1267
    Image: C:\h\w\A5880934\p\dotnet.exe

----- end Wed 03/27/2024  7:29:52.87 ----- exit code -1073740286 ----------------------------------------------------------
@BruceForstall BruceForstall added JitStress CLR JIT issues involving JIT internal stress modes area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI labels Mar 27, 2024
Copy link
Contributor

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Mar 27, 2024
@BruceForstall BruceForstall added the blocking-clean-ci-optional Blocking optional rolling runs label Mar 27, 2024
@AndyAyersMS
Copy link
Member

The old regex IL could produce some pretty messy flow. I think this is exposing lack of a cyclic cap for the irreducible (improper) loop cases:

ccp backedge BB163 (6.707601e+29) -> BB164 likelihood 1
Cyclic weight 6.707601e+29 > 0.999(cap) -- will reduce to cap
For loop at BB164 cyclic weight is 0.999 cyclic probability is 1000 [capped]

Computing block weights
Synthesis solver: flow graph has 265 improper loop headers

iteration 0: BB164 :: old 0 new 2.545923e+07 change 2.545923e+07

...
iteration 1: BB117 :: old 8.508028 new 6.324828e+35 change 6.324828e+35
...
iteration 9: BB990 :: old 1.423293e+306 new inf change inf

Seems like any realistic flow we'll see should have counts that top out at 10^12 or so, so perhaps I'll just stop if we see something that large.

@AndyAyersMS
Copy link
Member

Another option would be (assuming we were in repair mode) switch to blend, perhaps a hard blend, and then retry ... worth considering if we value consistency over accuracy.

@AndyAyersMS
Copy link
Member

FYI this was triggered by stress putting in random profile data, which was wildly inconsistent, and triggered a repair. Normally we would not see profile data like this in a Tier0 method.

JitStress -- incorporating random profile data

AndyAyersMS added a commit to AndyAyersMS/runtime that referenced this issue Mar 28, 2024
In large methods with lots of irreducible loops we may find
reconstructed counts reaching very large values.

Since profile counts in practice won't ever be much larger than say
10^12, detect when reconstructed counts exceed this value, and stop
the algorithm.

We may eventually decide to rerun in "hard blend" mode where we
intentionally limit the edge likelihood ranges. But this should do for
now.

Closes dotnet#100350.
AndyAyersMS added a commit that referenced this issue Mar 28, 2024
In large methods with lots of irreducible loops we may find
reconstructed counts reaching very large values.

Since profile counts in practice won't ever be much larger than say
10^12, detect when reconstructed counts exceed this value, and stop
the algorithm.

We may eventually decide to rerun in "hard blend" mode where we
intentionally limit the edge likelihood ranges. But this should do for
now.

Closes #100350.
@dotnet-policy-service dotnet-policy-service bot removed the untriaged New issue has not been triaged by the area owner label Mar 28, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Apr 28, 2024
matouskozak pushed a commit to matouskozak/runtime that referenced this issue Apr 30, 2024
In large methods with lots of irreducible loops we may find
reconstructed counts reaching very large values.

Since profile counts in practice won't ever be much larger than say
10^12, detect when reconstructed counts exceed this value, and stop
the algorithm.

We may eventually decide to rerun in "hard blend" mode where we
intentionally limit the edge likelihood ranges. But this should do for
now.

Closes dotnet#100350.
Ruihan-Yin pushed a commit to Ruihan-Yin/runtime that referenced this issue May 30, 2024
In large methods with lots of irreducible loops we may find
reconstructed counts reaching very large values.

Since profile counts in practice won't ever be much larger than say
10^12, detect when reconstructed counts exceed this value, and stop
the algorithm.

We may eventually decide to rerun in "hard blend" mode where we
intentionally limit the edge likelihood ranges. But this should do for
now.

Closes dotnet#100350.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-clean-ci-optional Blocking optional rolling runs JitStress CLR JIT issues involving JIT internal stress modes
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants