JIT: fold isinst using reaching VNs and compareTypesForCast by EgorBo · Pull Request #128500 · dotnet/runtime

EgorBo · 2026-05-22T18:45:27Z

This PR extends optAssertionIsSubtype to support PHI nodes via optVisitReachingAssertions. Also it makes logic slightly less conservative than direct == for class handles when we iterate assertions.

Example: the following always-true cast is currently not folded by the JIT:

static void Test(int len)
{
    object bc;
    if (len > 100)
        bc = new DerivedClass1();
    else
        bc = new DerivedClass2();

    if (bc is BaseClass) // <--- always true, but we don't fold it.
        Console.WriteLine("bc is a BaseClass");
    else
        Console.WriteLine("bc is not a BaseClass");
}

class BaseClass {}
class DerivedClass1 : BaseClass {}
class DerivedClass2 : BaseClass {}

Codegen:

 ; Assembly listing for method Program:Test(int) (FullOpts)
 G_M29168_IG01:
        sub      rsp, 40
-G_M29168_IG02:
-       cmp      ecx, 100
-       jg       SHORT G_M29168_IG04
-G_M29168_IG03:
-       mov      rcx, 0x7FFEA76B8E60      ; DerivedClass2
-       call     CORINFO_HELP_NEWSFAST
-       jmp      SHORT G_M29168_IG05
-G_M29168_IG04:
-       mov      rcx, 0x7FFEA76BC140      ; DerivedClass1
-       call     CORINFO_HELP_NEWSFAST
-G_M29168_IG05:
-       mov      rdx, rax
-       mov      rcx, 0x7FFEA76B8CD8      ; BaseClass
-       call     CORINFO_HELP_ISINSTANCEOFCLASS
-       test     rax, rax
-       jne      SHORT G_M29168_IG07
-G_M29168_IG06:
-       mov      rcx, 0x26775762D58      ; 'bc is not a BaseClass'
-       call     [System.Console:WriteLine(System.String)]
-       jmp      SHORT G_M29168_IG08
-G_M29168_IG07:
+G_M29168_IG02:
        mov      rcx, 0x26775762D98      ; 'bc is a BaseClass'
        call     [System.Console:WriteLine(System.String)]
-G_M29168_IG08:
        nop
-G_M29168_IG09:
+G_M29168_IG03:
        add      rsp, 40
        ret
-; Total bytes of code 104
+; Total bytes of code 26

Extend optAssertionIsSubtype to: * resolve target/source class handles via the embedded-handle map (AOT/R2R-safe); * fold via compareTypesForCast == Must, not just handle equality; * inspect VN-level type info (e.g. VNF_JitNew exact type); * recurse through reaching VNs/assertions for PHI defs. This lets the JIT prove always-true isinst checks where the object comes from a PHI of typed allocations (e.g. new DerivedClass1() and new DerivedClass2() joined into object then tested 'is BaseClass'). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

EgorBo · 2026-05-22T18:46:27Z

@MihuBot

dotnet-policy-service · 2026-05-22T18:46:46Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Copilot

Pull request overview

This PR enhances CoreCLR JIT global assertion propagation so it can prove certain isinst/chkcast helper calls are redundant by combining (1) live assertions, (2) VN-level object type info, and (3) PHI reaching-VN analysis, using compareTypesForCast to validate cast success.

Changes:

Refactors optAssertionIsSubtype to return bool and to resolve the target type handle robustly (helper-arg handle or VN-encoded handle).
Introduces optAssertionVNIsSubtype to prove cast success via matching subtype/exact-type assertions, VN object-type recovery, and recursive PHI-edge analysis with a depth budget.
Updates the cast-helper handling in optAssertionProp_Call to consume the new boolean result and adjust debug logging.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/coreclr/jit/compiler.h	Updates assertion-prop helper signatures and declares the new VN-based subtype helper.
src/coreclr/jit/assertionprop.cpp	Implements VN + reaching-assertion based subtype proof and applies it to folding redundant cast helpers.

EgorBo · 2026-05-22T19:50:11Z

@MihuBot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

* valuenum.h: describe the pCls out-param and embedded-handle mapping. * assertionprop.cpp: fix self-referential comment (cls -> castTo). * Add regression test under JIT/opt/AssertionPropagation covering PHI-of-derived (folds), PHI-with-null (must not fold to true), direct alloc, and PHI-with-unrelated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

EgorBo · 2026-05-22T22:38:34Z

PTAL @AndyAyersMS @dotnet/jit-contrib, SPMI diffs are empty due to new JIT-EE calls, jit-diffs

AndyAyersMS

I had the impression that optAssertionVNIsSubtype was not very useful. Am I wrong?

`GenTree::BashToConst<T>` repurposes an arbitrary node into a constant but never resets `AsIntCon()->gtCompileTimeHandle`, so the field retains stale data from the node's prior life. `optConstantAssertionProp` then re-applies `GTF_ICON_CLASS_HDL` on top of the bashed node based on the VN's handle flags, leaving a CNS_INT with `iconValue = realClassHandle`, the class-handle flag set, but a stale `gtCompileTimeHandle`. When `fgValueNumberTreeConst` later sees this node it overwrites the existing entry in the EmbeddedHandleMap with that stale handle (via `AddOrUpdate`), poisoning lookups. Consumers such as `IsVNTypeHandle` and `VNEvalForCast` then read the garbage back, and `compareTypesForCast` crashes when dereferencing the bogus type handle. Three small fixes: 1. `GenTree::BashToConst<T>`: zero `gtCompileTimeHandle` alongside the existing `gtFieldSeq = nullptr`, matching the "node is now a fresh constant" semantics. 2. `fgValueNumberTreeConst`: only register an entry in the EmbeddedHandleMap when `gtCompileTimeHandle != 0`, so a re-flagged bashed node cannot overwrite a valid mapping placed by a real `gtNewIconEmbHndNode` that shares the same iconValue. 3. `fgValueNumberSpecialIntrinsic`: fix an inverted bail-out that proceeds to `getRuntimeTypePointer` with a 0 handle when the map lookup fails (`&&` + `clsHandle != 0` made the bail unreachable because `clsHandle` is initialized to 0). With fix #2, "no entry" becomes more common, and this site must bail explicitly. Reproduced on System.Memory.Tests / System.Linq.Tests with DOTNET_TieredCompilation=1 against the PR's `IsVNTypeHandle` consumer. Tests pass after the fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

EgorBo · 2026-05-23T01:51:30Z

I had the impression that optAssertionVNIsSubtype was not very useful. Am I wrong?

It seems it leads to +1,720 bytes regressions and 730 missing contexts if I remove it (in Libraries.pmi) so not sure. jit-diff hit just a few uses because these patterns often appear with PGO and jit-diff doesn't have it.

BTW, fixed #128061 bug along the way, BashToConst didn't zero out gtCompileTime handle.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Copilot AI review requested due to automatic review settings May 22, 2026 18:45

Copilot started reviewing on behalf of EgorBo May 22, 2026 18:45 View session

github-actions Bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label May 22, 2026

dotnet-policy-service Bot assigned EgorBo May 22, 2026

MihuBot mentioned this pull request May 22, 2026

[JitDiff X64] [EgorBo] JIT: fold isinst using reaching VNs and compareTypesF ... MihuBot/runtime-utils#1922

Open

Copilot AI reviewed May 22, 2026

View reviewed changes

Comment thread src/coreclr/jit/assertionprop.cpp Outdated

cleanup

1f03a1f

MihuBot mentioned this pull request May 22, 2026

[JitDiff X64] [EgorBo] JIT: fold isinst using reaching VNs and compareTypesF ... MihuBot/runtime-utils#1923

Open

clean up

4967816

Copilot AI review requested due to automatic review settings May 22, 2026 20:29

Copilot started reviewing on behalf of EgorBo May 22, 2026 20:29 View session

EgorBo and others added 2 commits May 22, 2026 22:30

Add JIT header comment to IsVNTypeHandle overload

9b12b3c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix comment

0018f35

Copilot AI reviewed May 22, 2026

View reviewed changes

Comment thread src/coreclr/jit/valuenum.h Outdated

Comment thread src/coreclr/jit/assertionprop.cpp Outdated

Comment thread src/coreclr/jit/assertionprop.cpp Outdated

EgorBo and others added 2 commits May 22, 2026 23:18

Update IsInstReachingVN.csproj

3039a29

Copilot AI review requested due to automatic review settings May 22, 2026 21:22

Copilot started reviewing on behalf of EgorBo May 22, 2026 21:22 View session

EgorBo marked this pull request as ready for review May 22, 2026 21:25

Copilot AI reviewed May 22, 2026

View reviewed changes

Comment thread src/tests/JIT/opt/AssertionPropagation/IsInstReachingVN.cs

Comment thread src/tests/JIT/opt/AssertionPropagation/IsInstReachingVN.csproj

Comment thread src/tests/JIT/opt/AssertionPropagation/IsInstReachingVN.cs

Update IsInstReachingVN.cs

9e37596

EgorBo requested a review from AndyAyersMS May 22, 2026 22:38

AndyAyersMS approved these changes May 22, 2026

View reviewed changes

This was referenced May 23, 2026

LibraryImportGenerator.Unit.Tests crashing on linux-x64 mono interpreter #100800

Open

Multiple Helix work items fail on maccatalyst/tvos CoreCLR Release #126460

Open

Copilot AI review requested due to automatic review settings May 23, 2026 01:45

Copilot started reviewing on behalf of EgorBo May 23, 2026 01:45 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

Comment thread src/tests/JIT/opt/AssertionPropagation/IsInstReachingVN.csproj

Comment thread src/tests/JIT/opt/AssertionPropagation/IsInstReachingVN.cs

Comment thread src/tests/JIT/opt/AssertionPropagation/IsInstReachingVN.cs

EgorBo merged commit 6d63129 into dotnet:main May 23, 2026
143 of 145 checks passed

EgorBo deleted the jit-isinst-fold-reaching-vns branch May 23, 2026 11:30

EgorBo mentioned this pull request May 23, 2026

[ci-scan] Known Build Error: JIT AV in optAssertionPropMain during type comparison folding under jitstress on arm64 #128061

Closed

Conversation

EgorBo commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EgorBo commented May 22, 2026

Uh oh!

dotnet-policy-service Bot commented May 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

EgorBo commented May 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EgorBo commented May 22, 2026

Uh oh!

AndyAyersMS left a comment

Choose a reason for hiding this comment

Uh oh!

EgorBo commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

EgorBo commented May 22, 2026 •

edited

Loading

EgorBo commented May 23, 2026 •

edited

Loading