Avoid unnecessary allocations of jump stubs

[jkotas]

Fixes #116748

[Copilot]

Pull Request Overview

This PR adds checks to prevent additional jump stub allocations once an overflow condition is detected, deferring to a full retry instead of creating more stubs.

• Introduces m_fJumpStubOverflow guard in the Rel32 path to force delta = 0
• Adds a similar overflow check in the Rel28 stub allocation path, declaring jumpStubAddr and conditionally skipping stub creation

Comments suppressed due to low confidence (1)

src/coreclr/vm/jitinterface.cpp:11788

• The new overflow path for Rel28 stub allocation isn't covered by existing tests. Consider adding a unit or integration test that sets m_fJumpStubOverflow and verifies no additional jump stubs are allocated.

                TADDR jumpStubAddr;

[Copilot]

In the Rel28 stub allocation path, the branch for m_fJumpStubOverflow sets jumpStubAddr = 0 but does not reset delta. This could lead to an invalid relocation calculation using jumpStubAddr = 0. Consider assigning delta = 0 here or skipping the rest of the relocation logic to match the behavior in the Rel32 branch.

[jakobbotsch]

Is my understanding of what happens in the test correct?

1. We have a first JIT attempt that sets m_fAllowRel32 = false
2. The second JIT attempt fails to allocate a jump stub and sets m_fJumpStubOverflow = false
3. The third try allocates a new code heap without any address range constraints and puts everything in there

[jakobbotsch]

Or is 1 and 2 happening during the same attempt? And the slowness of the compilation was because of needing to find 284k jump stubs, which all go into the ReserveWithinRange path that can be quite slow?

[jkotas]

It is 3 step process in the worst-case scenario on x64 like what's exercised by this test. It is 2 step process on arm64 (there is no allowRel32 on arm64.)

It may be possible to make it 2 step process in the worst case on x64 as well if the JIT passed more reloc details to the EE. I do not think we care enough about the extremely large methods like this one to make it better.

The second JIT attempt fails to allocate a jump stub and sets m_fJumpStubOverflow = false

Right.

The third try allocates a new code heap without any address range constraints and puts everything in there

The third allocates a code heap with enough spare space for jump stubs that we may need. Jump stubs are allocated in the same code heap as regular code and we always try to find space for jump stubs in the existing code heaps first.