[API Proposal]: Allow runtime-less methods as NativeAOT library entry points

[TheLeftExit]

Background and motivation

As many other users who asking to run things in DllMain under NativeAOT, I'm using NativeAOT to inject code into a game, and I very much want to stay within the .NET ecosystem and avoid writing a separate C++ library shim just for the injection bit.

From what I've read, NativeAOT doesn't support runtime initialization under loader lock. However, I've also seen samples of runtime-less code, which is limited, but still usable for simple tasks with enough boilerplate (https://github.com/MichalStrehovsky/zerosharp).

A minimal hook injection involves calling a few LibraryImport-ed Win32 APIs (GetModuleHandle, VirtualAlloc/Protect), taking an address of an UnmanagedCallersOnly function, and writing to some pointers. All of this is doable without a managed runtime, and it probably doesn't violate loader lock rules.

To facilitate this scenario while sticking to the rules imposed by NativeAOT and the OS, I'm proposing two attributes. The first:

• Would apply to an UnmanagedCallersOnly method,
• Would restrict the method's code to no-runtime-compatible operations and calls to other "runtimeless" methods,
• Would compile the method directly from IL to bytecode (or minimize whatever thunks are usually inserted for reverse P/Invokes, to bring the code down to the bare metal as much as possible),
• Would allow unmanaged callers to invoke this function directly, without initializing the runtime if it hasn't yet been initialized,

The second:

• Would apply to a method marked with the first attribute,
• Would only apply to a single, parameterless method in the module,
• Would, allow insertion of a call to this method into what's currently a no-op DllMain.

API Proposal

namespace System.Runtime.InteropServices;

[AttributeUsage(AttributeTargets.Method)]
public sealed class UnmanagedMethod : Attribute;

[AttributeUsage(AttributeTargets.Method)]
public sealed class UnmanagedModuleInitializer : Attribute;

API Usage

public static unsafe class Example
{
    [DllImport("kernel32.dll")]
    private static extern uint GetModuleHandleW(char* moduleName);

    [DllImport("kernel32.dll")]
    private static extern uint VirtualProtect(uint address, uint size, uint newProtect, out uint oldProtect);

    [UnmanagedCallersOnly(CallConvs = [typeof(CallConvStdcall)])]
    [UnmanagedMethod]
    [UnmanagedModuleInitializer]
    public static void OnStartup()
    {
        // Call to PeekMessageA in gta_sa.exe's main loop
        var originalInstructionPtr = GetModuleHandleW(null) + (0x748A57 - 0x400000);
        var newFunctionPtr = (uint)(delegate* unmanaged[Stdcall]<void*, nint, uint, uint, uint, int>)&NotPeekMessageA;
        VirtualProtect(originalInstructionPtr, 6, 0x04, out var oldProtect);
        *(byte*)originalInstructionPtr = 0xE8;
        *(uint*)(originalInstructionPtr + 1) = newFunctionPtr - (originalInstructionPtr + 5);
        VirtualProtect(originalInstructionPtr, 6, oldProtect, out _);

    }

    [UnmanagedCallersOnly(CallConvs = [typeof(CallConvStdcall)])]
    private static int NotPeekMessageA(void* lpMsg, nint hWnd, uint wMsgFilterMin, uint wMsgFilterMax, uint wRemoveMsg)
    {
        return 0; // Install/proc custom synchronization context, then return from real PeekMessageA
    }
}

Alternative Designs

It might also be possible to reduce the number of attributes - for instance, add another parameter to UnmanagedCallersOnly instead of designating a separate UnmanagedMethod attribute.

Risks

Users will now be able to cause undefined behavior if their runtimeless code violates loader locks in other ways (e.g., LoadLibrary).

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

[huoyaoyuan]

I'm using NativeAOT to inject code into a game, and I very much want to stay within the .NET ecosystem and avoid writing a separate C++ library shim just for the injection bit.

Engineering-wise, it's much more actionable to write a separated NativeAOT library for injection. The bootstrap library can then link to no-GC runtime.

All of this is doable without a managed runtime, and it probably doesn't violate loader lock rules.

The limitation can be more complicated depending on the involved P/Invokes. Additionally, there's no reliable way to verify it.

• Would restrict the method's code to no-runtime-compatible operations and calls to other "runtimeless" methods,

This is quite a niche scenario and unlikely to get such level of support. Again, the verification of "runtimeless" is not established at all.

• Would, allow insertion of a call to this method into what's currently a no-op DllMain.

This isn't the right direction since we are not interested to establish (unreliable) DllMain support. Instead, unblocking DllExport("DllMain") by customization should be used instead.

---

This should not be an API proposal since the whole mechanism hasn't been designed. Thinking in another direction, the real blocker is that there's no way to delay the bootstrap of managed runtime. If there is hypothetical manual bootstrap method, then the workflow can be:

[DllExport]
[UnmanagedcallersOnly]
public static void DllMain()
{
    SetupHook(&Hook);
}

[UnmanagedcallersOnly]
public static void Hook()
{
    BootstrapCLR();
    string a = ""; // using managed type
}

Or, if the NativeAOT compiler can delay the initialization to usage of any managed type (under an unsupported switch), then it can be easily workarounded. Just link to a custom runtime without the empty DllMain and it will just work under method separation. Linking custom runtime would become an explicit confirmation of "I know this is unsupported".

Mechanisms for building verifications would not be considered, as it can easily become indeed unverifiable involving OS internals. "Write your own analyzer" is the viable approach then.

[huoyaoyuan]

In other words, DllMain injection isn't an reliable end-to-end scenario that the team is interested to support. Thus, dedicated options and analyzing model won't be considered. Small tweaks to allow workarounds may be considered.

[TheLeftExit]

@huoyaoyuan Thanks for taking a look. Now that I think of it, considering the direction earlier DllMain-related issues went, something like a documented switch was never going to be an option.

It's just that I've looked at the considered .csproj properties and linker options (https://github.com/dotnet/runtime/tree/main/src/coreclr/nativeaot/BuildIntegration), and it's difficult to understand what's referred to as custom runtimes in other issues, and how to get started on using one of those. If there's something I can do with linker options or other stuff at the .csproj level, I'm okay with that. Maybe a brief FAQ or a "so you want to go into the unsupported area - here's what you need to know" doc in this repo would help, otherwise the resources on customizing the NativeAOT build process are scarce. Or maybe it's the fact that I haven't looked into how native apps compile, and it would've been more intuitive with that knowledge.

Either those docs, or tweaks to simplify customization, would be great.

(I though "API proposal" fits best since I couldn't find a "feature proposal" template, guess not)

[jkotas]

We are not interested in creating support for managed code that is safe to run inside DllMain. It is very niche scenario, and it is very complicated and expensive to do well.

All code running in DllMain has to be written in C or similar lower-level language. I am sorry,

[jkotas]

as custom runtimes in other issues, and how to get started on using one of those

Custom runtime in these discussions typically means that you take C# language and the AOT compiler, and you provide your own class library and runtime. https://github.com/MichalStrehovsky/SeeSharpSnake is an example. These examples show what's possible, but they do not have a lot of practical use cases. When you cannot use the rich .NET class library, you are giving up most of the .NET value prop.

[TheLeftExit]

For future readers - I think I got what I wanted with custom linker args, it just required a static C library instead of an "unmanaged C# island": https://github.com/TheLeftExit/DllMain