Fix caught foreign native exception handling take 2

[janvorli]

The previous fix was reverted, as it didn't take into account hardware
exceptions stemming from managed code. This has broken debugger tests.

When a native exception occurs on a foreign thread, it is propagated
through managed frames and then flows into the foreign native frames
and is handled there, the runtime still reports the exception as unhandled
via AppDomain.CurrentDomain.UnhandledException event and also prints
out the unhandled exception message. Even though the process then continues
running just fine.

This change fixes the behavior so that only managed exceptions
thrown by the .NET runtime are reported as unhandled in such case.
Foreign exceptions, like e.g. C++ exceptions, are not reported as
unhandled and they are just propagated into the foreign native
frames. If that exception is not handled by the native frames,
then C++ reports them as unhandled.

I have also added a test that fails without this fix.

Close #115654

[Copilot]

Pull Request Overview

This PR refines unhandled-exception reporting so that only managed exceptions thrown by the .NET runtime trigger the AppDomain unhandled-event, while foreign/native exceptions are left to native handlers. It also adds test cases to verify this behavior on both the main and secondary threads, and adjusts core exception-logging logic accordingly.

• Update expected exit codes in unhandledTester.cs to account for hardware (NullReference) exceptions.
• Add C# branches for mainhardware, secondaryhardware, and secondaryunhandled in unhandled.cs and corresponding native/thread helpers in ExceptionInterop.
• Modify exceptionhandling.cpp to skip logging for exceptions not originating from managed code.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/tests/baseservices/exceptions/unhandled/unhandledTester.cs	Adjusted expectedExitCode logic and added hardware test invocations
src/tests/baseservices/exceptions/unhandled/unhandled.cs	Added branches to throw NullReferenceException on main and secondary threads, plus secondaryunhandled
src/tests/baseservices/exceptions/exceptioninterop/ExceptionInteropNative.cpp	Introduced InvokeCallbackOnNewThread and native catch/rethrow helper
src/tests/baseservices/exceptions/exceptioninterop/ExceptionInterop.cs	Imported InvokeCallbackOnNewThread and added PropagateAndCatchCppException test
src/coreclr/vm/exceptionhandling.cpp	Added isNotHandledByRuntime and gated logging to managed-code exceptions only

Comments suppressed due to low confidence (3)

src/tests/baseservices/exceptions/unhandled/unhandledTester.cs:141

• The new "secondaryunhandled" branch in unhandled.cs isn’t invoked by TestEntryPoint, leaving that code path untested. Add a RunExternalProcess("secondaryunhandled", "unhandled.dll"); call immediately after the secondaryhardware invocation.

RunExternalProcess("secondaryhardware", "unhandled.dll");

src/tests/baseservices/exceptions/exceptioninterop/ExceptionInterop.cs:198

• [nitpick] For consistency with C# naming conventions, rename the parameter from callBack to callback.

public static extern void InvokeCallbackOnNewThread(delegate*unmanaged<void> callBack);

src/coreclr/vm/exceptionhandling.cpp:4018

• [nitpick] The isNotHandledByRuntime initialization mixes preprocessor directives and logical operators, which can be hard to follow. Consider restructuring the expression or aligning the #ifdef block so the OR-conditions remain visually coherent.

bool isNotHandledByRuntime =

[jkotas]

I got a bit confused by invalidRevPInvoke local variable name. I am not sure what's invalid about it.

Would isReversePInvoke be a better name for it?

[janvorli]

That's a historical name that was created when the Interop::ManagedToNativeExceptionCallback was introduced. It is confusing to me too. I am planning to restructure this code a bit as part of a fix to the #116676, so I will address it as part of that.

[janvorli]

cc: @tommcdon - I have verified that this change fixes the failure in the debugger tests.

[janvorli]

/ba-g the wasm build failure is unrelated to this change, it happens in onstackreplacement.cpp, but the log doesn't show any specific failure.