Fix false-positive assertion in CheckRegDisplaySP due to stale cached stack limit

[Copilot]

On Linux, the main thread's stack grows on demand — the kernel commits new pages below the initial boundary as SP moves lower. m_CacheStackLimit is set once at thread init via pthread_attr_getstack() and never refreshed. Under JIT stress modes (jitminopts, tailcallstress, interpmode1) that drive deeper recursion, GC/profiler stack walks encounter frames with SP below the stale cached limit, triggering a false-positive _ASSERTE in CheckRegDisplaySP.

Description

In CheckRegDisplaySP (#ifdef DEBUG_REGDISPLAY, checked/debug builds only), replace the stale GetCachedStackLimit() with GetStackLowerBound(), which queries the current committed stack boundary dynamically:

// Before
_ASSERTE(...|| PTR_VOID(pRD->SP) >= pRD->_pThread->GetCachedStackLimit());

// After — dynamic query avoids stale cached value on grown stacks
#ifndef DACCESS_COMPILE
_ASSERTE(...|| PTR_VOID(pRD->SP) >= pRD->_pThread->GetStackLowerBound());
#else
_ASSERTE(...|| PTR_VOID(pRD->SP) >= pRD->_pThread->GetCachedStackLimit());
#endif

GetStackLowerBound() is not available in DAC builds (#ifndef DACCESS_COMPILE), so DAC retains the original GetCachedStackLimit() path. No release build impact — the entire function is inside #ifdef DEBUG_REGDISPLAY.

Fixes #124839

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[jkotas]

PAL is caching the cache limits as well:

runtime/src/coreclr/pal/src/thread/thread.cpp

Lines 2396 to 2405 in 6adf63b

	CPalThread::GetCachedStackBase()
	{
	_ASSERT_MSG(this == InternalGetCurrentThread(), "CPalThread::GetStackBase called from foreign thread");
	if (m_stackBase == NULL)
	{
	m_stackBase = GetStackBase();
	}
	return m_stackBase;

. This is still going to use cached stack limit as far as I can tell.

[jkotas]

I think we will need to pass a bool flag all the way to the PAL to fetch non-cached stack limits.

Also, we make sure to use the cached stack limits most of the time to make this check cheap.

[jkotas]

NO_FIXED_STACK_LIMIT is working around the same problem, just for Musl. If we find a better way to fix this, we should delete NO_FIXED_STACK_LIMIT.

[janvorli]

It's been a long time since I've implemented the stack limits stuff for Unix, so I don't remember all the details, but I am not sure the copilot analysis is correct. I'll experiment locally to see if it is right or not.

[janvorli]

My feeling was right. On non-MUSL linux, the pthread_attr_getstack returns hard stack limit. Going over (or I should say below) that limit results in stack overflow. On MUSL based linux, there is no hard limit and the stack grows until it cannot grow anymore due to hitting virtual memory range that's already mapped by something or due to OOM when trying to get physical memory page to backup the next stack page.
That's why on MUSL based linux we define the NO_FIXED_STACK_LIMIT.

Looking at the failed legs listed in the actual issue, they were all failing when running with interpreter. So I think it is an issue when we somehow ended up passing a wrong SP to the CheckRegDisplaySP. Could have been due to some issues in the interpreter that were fixed since, as I cannot see recent failures due to this issue.

[mangod9]

ok should we close the issue then and wait for a repro? Looks like there was another hit a few weeks ago: #125760 (comment), but doesnt look like the dumps are still available for it.

[janvorli]

I think it would be reasonable to close it and wait for a repro with dump.