Ensure that Native AOT libraries are not unloaded

[rcj1]

Ensure that NAOT libraries are not unloaded by opening them with the RTLD_NODELETE flag. Mac and glibc support this flag, and musl libc does not support unloading libraries.
Fixes #64629

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR updates NativeAOT’s Unix PAL to prevent unloading the NativeAOT module by using RTLD_NODELETE, and updates the NativeAOT SharedLibrary smoke test to actually call dlclose on Unix.

Changes:

• Add best-effort “pin in memory” behavior on Unix by calling dlopen(..., RTLD_NODELETE) in PalGetModuleHandleFromPointer.
• Update the NativeAOT SharedLibrary smoke test to call dlclose(handle) on Unix (matching the intent of exercising an unload attempt).

Show a summary per file

File	Description
src/tests/nativeaot/SmokeTests/SharedLibrary/SharedLibrary.cpp	Switch Unix path to call dlclose to exercise the “attempted unload” scenario.
src/coreclr/nativeaot/Runtime/unix/PalUnix.cpp	Attempt to apply RTLD_NODELETE to the module containing a given pointer to prevent unload on supporting libcs.

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 1

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

[jkotas]

info.dli_fname is not guaranteed to resolve back to our library. The library could have been replaced atomically during an update, loaded from a transient location, ...

A few ideas to make this more robust:

• Use -z nodelete linker command line switch instead of this where possible. -z nodelete is supported by glibc Linux.
• Add | RTLD_NOLOAD to make sure that we do not load a new library that happens to match by name. It may fail to pin our library, but it won't cause active damage at least.

[rcj1]

Unfortunately, RTLD_NOLOAD and RTLD_NODELETE are not compatible on Mac, in combination RTLD_NODELETE is ignored.

[rcj1]

If we use RTLD_NOLOAD, we will have to just hold onto a reference.

[Copilot]

Copilot's findings

Suppressed comments (3)

src/tests/nativeaot/SmokeTests/SharedLibrary/SharedLibrary.cpp:80

• The test currently calls dlclose/FreeLibrary but doesn't verify the library stays loaded afterwards. Since the goal is to ensure NativeAOT shared libraries are not unloadable, add a post-unload check (GetModuleHandle on Windows; dlopen with RTLD_NOLOAD on Unix) so the test fails if the module actually unloads without crashing immediately.

    // to unload the library does not to crash at least.
#ifdef TARGET_WINDOWS
    FreeLibrary(handle);
#else
    dlclose(handle);

src/coreclr/nativeaot/Runtime/unix/PalUnix.cpp:866

• The pinning path calls dlopen(..., RTLD_NOLOAD) but ignores whether it succeeded. If dlopen returns null, the module may still be unloadable despite pinModule=true. Consider checking the return value and falling back to a plain dlopen (which should just bump the refcount for an already-loaded image) to make pinning reliable.

        if (pinModule && info.dli_fname != nullptr)
        {
            // NativeAOT runtime state cannot be safely unloaded.
            // Keep the extra reference for the lifetime of the process.
            dlopen(info.dli_fname, RTLD_LAZY | RTLD_NOLOAD);

src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets:294

• _targetOS is normalized to 'linux' for linux-musl and linux-bionic as well (see SingleEntry.targets). If this flag is intended for glibc only, checking for '$(_linuxLibcFlavor)' == 'glibc' won't work (glibc is represented by an empty _linuxLibcFlavor). Consider excluding the known non-glibc flavors instead (musl/bionic).

      <!-- NativeAOT shared libraries should not be unloaded -->
      <LinkerArg Include="-Wl,-z,nodelete" Condition="'$(_targetOS)' == 'linux' and '$(NativeLib)' == 'Shared'" />

• Files reviewed: 7/7 changed files
• Comments generated: 0 new

[Copilot]

Copilot's findings

Suppressed comments (2)

src/tests/nativeaot/SmokeTests/SharedLibrary/SharedLibrary.cpp:81

• dlclose(handle) is now executed on every non-Windows Unix target, but the runtime-side pinning in this PR only runs on HOST_OSX (and the linker -z,nodelete is Linux-only). This means platforms like FreeBSD/OpenBSD may actually unload the NativeAOT shared library here, which the comment above explicitly says is not supported, potentially turning this smoke test into a platform-specific crash/regression.

Consider limiting the dlclose attempt to the platforms where we have a non-unload guarantee in this PR (Apple + Linux), and keep the handle open elsewhere until/unless we add equivalent pinning.

#ifdef TARGET_WINDOWS
    FreeLibrary(handle);
#else
    dlclose(handle);
#endif

src/coreclr/nativeaot/Runtime/unix/PalUnix.cpp:869

• pinModule is currently only acted on for HOST_OSX, and even there it uses RTLD_NOLOAD without RTLD_NODELETE (and without checking the result). This makes the pinModule contract inconsistent across Unix platforms and doesn’t match the PR description’s claim of using RTLD_NODELETE where supported.

It would be more robust to honor pinModule on all non-WASM Unix platforms by taking an extra reference via dlopen(..., RTLD_NOLOAD) and, when available, also setting RTLD_NODELETE. That way Linux/glibc and other ELF Unixes get the same behavior, and the code doesn’t silently ignore pinModule on most Unix targets.

#if defined(HOST_OSX)
        if (pinModule && info.dli_fname != nullptr)
        {
            // NativeAOT runtime state cannot be safely unloaded.
            // Keep the extra reference for the lifetime of the process.

• Files reviewed: 7/7 changed files
• Comments generated: 0 new