NativeAOT on Graviton2 As armv8.2-a Fails to Execute

[chrisoverzero]

Description

When NativeAOT-compiling an application for deployment onto AWS Lambda, specifying IlcInstructionSet with the known, documented supported instruction sets for Graviton2 processors fails to execute with the error message:

The required instruction sets are not supported by the current CPU.

Reproduction Steps

For a minimal reproduction, I'm using the code from maxday's lambda-perf cold start–testing repository, but targeting .NET 8's NativeAOT. The meaningful parts of that are:

public class Function
{
    private static async Task Main()
    {
        await LambdaBootstrapBuilder.Create(FunctionHandler, new SourceGeneratorLambdaJsonSerializer<LambdaFunctionJsonSerializerContext>())
            .Build()
            .RunAsync();
    }

    public static StatusResponse FunctionHandler()
    {
        return new StatusResponse(statusCode: 200);
    }
}

public record StatusResponse(int statusCode);

[JsonSerializable(typeof(StatusResponse))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
}

…and for .csproj settings:

    <TargetFramework>net8.0</TargetFramework>
    <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
    <PublishAot>true</PublishAot>
    <AssemblyName>bootstrap</AssemblyName>

    <IlcInstructionSet>armv8.2-a</IlcInstructionSet>

Expected behavior

Because Amazon's Graviton2 processor is documented as revision 8.2-a with support for features "fp16, rcpc, dotprod, crypto" and ilc doesn't support "fp16" or "crypto", I expected this:

<IlcInstructionSet>armv8.2-a,rcpc,dotprod</IlcInstructionSet>

…to compile and run. It compiled, but did not run. Figuring that maybe I was trying to add too much sauce, I tried:

<IlcInstructionSet>armv8.2-a</IlcInstructionSet>

…expecting this to compile and run. It compiled, but did not run.

Actual behavior

Upon execution, this error message is received:

The required instruction sets are not supported by the current CPU.

Regression?

No response

Known Workarounds

Leave off IlcInstructionSet, implicitly targeting revision 8.0. This does compile (natch) and run.

Configuration

I am cross-compiling for arm64 in the Docker image mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64:

~/.dotnet/dotnet publish -p:SysRoot=/crossrootfs/arm64 -p:LinkerFlavor=lld

dotnet --version is "8.0.100-rc.1.23402.12"

Other information

I admit that I'm not certain this is a problem with the NativeAOT compiler, since I don't know how to look into a binary and say "Aha, here is the incorrect instruction."

[ghost]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

When NativeAOT-compiling an application for deployment onto AWS Lambda, specifying IlcInstructionSet with the known, documented supported instruction sets for Graviton2 processors fails to execute with the error message:

The required instruction sets are not supported by the current CPU.

Reproduction Steps

For a minimal reproduction, I'm using the code from maxday's lambda-perf cold start–testing repository, but targeting .NET 8's NativeAOT. The meaningful parts of that are:

public class Function
{
    private static async Task Main()
    {
        await LambdaBootstrapBuilder.Create(FunctionHandler, new SourceGeneratorLambdaJsonSerializer<LambdaFunctionJsonSerializerContext>())
            .Build()
            .RunAsync();
    }

    public static StatusResponse FunctionHandler()
    {
        return new StatusResponse(statusCode: 200);
    }
}

public record StatusResponse(int statusCode);

[JsonSerializable(typeof(StatusResponse))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
}

…and for .csproj settings:

    <TargetFramework>net8.0</TargetFramework>
    <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
    <PublishAot>true</PublishAot>
    <AssemblyName>bootstrap</AssemblyName>

    <IlcInstructionSet>armv8.2-a</IlcInstructionSet>

Expected behavior

Because Amazon's Graviton2 processor is documented as revision 8.2-a with support for features "fp16, rcpc, dotprod, crypto" and ilc doesn't support "fp16" or "crypto", I expected this:

<IlcInstructionSet>armv8.2-a,rcpc,dotprod</IlcInstructionSet>

…to compile and run. It compiled, but did not run. Figuring that maybe I was trying to add too much sauce, I tried:

<IlcInstructionSet>armv8.2-a</IlcInstructionSet>

…expecting this to compile and run. It compiled, but did not run.

Actual behavior

Upon execution, this error message is received:

The required instruction sets are not supported by the current CPU.

Regression?

No response

Known Workarounds

Leave off IlcInstructionSet, implicitly targeting revision 8.0.

Configuration

I am cross-compiling for arm64 in the Docker image mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64:

~/.dotnet/dotnet publish -p:SysRoot=/crossrootfs/arm64 -p:LinkerFlavor=lld

dotnet --version is "8.0.100-rc.1.23402.12"

Other information

I admit that I'm not certain this is a problem with the NativeAOT compiler, since I don't know how to look into a binary and say "Aha, here is the incorrect instruction."

Author:	chrisoverzero
Assignees:	-
Labels:	area-NativeAOT-coreclr
Milestone:	-

[EgorBo]

I'll try on my M1, does it work if you don't specifcy any ISA at all? armv8.2-a should include neon (always enabled), lse (atomicts, e.g. all Interlocked.* APIs) and crc + rdma, the last two are unlikely to be hit?

[chrisoverzero]

[…] does it work if you don't specifcy any ISA at all?

Do you mean leaving off IlcInstructionSet? That does compile and run, yes. I'll update "Workarounds" to be clearer about that.

[am11]

It is working on M1 with RC1 daily build. You can also specify native (since you are on new enough SDK).

cpufeatures.{c,h}, cpuid.h and minipalconfig.h from https://github.com/dotnet/runtime/tree/main/src/native/minipal participate in CPU feature detection and they can be complied independently with this kind of a glue code:

#define TARGET_UNIX
#define TARGET_ARM64

#include <stdio.h>
#include <cpufeatures.h>

int main(void) {
  int actualFeatures = minipal_getcpufeatures();
  printf("actual features: %d\n", actualFeatures);

  // TODO: unpack and print individual feature flags defined in cpufeatures.h

  return 0;
}

[chrisoverzero]

You can also specify native […]

Even if I'm not building on ARM?

[am11]

Ah cross compilation 🙈
Maybe try using this build container without cross: #88971 (comment).

[am11]

I admit that I'm not certain this is a problem with the NativeAOT compiler, since I don't know how to look into a binary and say "Aha, here is the incorrect instruction."

@EgorBo, @MichalStrehovsky, what do you think about dumping all the supported instruction sets in this error path:

runtime/src/coreclr/nativeaot/Runtime/startup.cpp

Lines 186 to 190 in c6fdf52

	if ((g_cpuFeatures & g_requiredCpuFeatures) != g_requiredCpuFeatures)
	{
	PalPrintFatalError("\nThe required instruction sets are not supported by the current CPU.\n");
	RhFailFast();
	}

"Supported instruction sets are: ..."

[chrisoverzero]

Setting cross-compilation aside, this also does not work when compiled and run on the Graviton2 machine I allocated for testing:

sh-5.2$ ./src/bin/Release/net8.0/linux-arm64/publish/bootstrap 

The required instruction sets are not supported by the current CPU.
Aborted (core dumped)

What can I do to get you more information?

ETA: Oh, I just remembered native. Trying that, now that I'm on Graviton2 natively, does work. So there's something defined in ilc's "armv8.2-a" which shouldn't be present? Or maybe Amazon is mistaken about the specs of Graviton2? Either feels unlikely, but I'll go with whatever.

Or the feature detection is missing features?

sh-5.2$ grep ^Features /proc/cpuinfo
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs

[MichalPetryka]

all the supported instruction sets

Listing required ones too would make sense.

[chrisoverzero]

Trying the supported options one at a time, the ones which cause failures are:

• rcpc
• dotprod
• rdma

…but I think these are covered in the features line from my previous comment, as "lrcpc", "asimddp", and "asimdrdm". The names are different, but these look like they correspond to HWCAP_LRCPC, HWCAP_ASIMDDP, and HWCAP_ASIMDRDM from here.

I'm well over my head on this, so I don't know where those are #defined, or how they're used or detected or anything, really. I do not know what the next steps are.

[MichalPetryka]

Could you run some software that can list out supported CPU features on the machine (like for example https://github.com/google/cpu_features#quickstart)?

[chrisoverzero]

The output of list_cpu_features:

{
  "arch": "aarch64",
  "implementer": 65,
  "variant": 3,
  "part": 3340,
  "revision": 1,
  "flags": ["aes", "asimd", "asimddp", "asimdhp", "asimdrdm", "atomics", "cpuid", "crc32", "dcpop", "evtstrm", "fp", "fphp", "lrcpc", "pmull", "sha1", "sha2", "ssbs"]
}

[MichalStrehovsky]

ETA: Oh, I just remembered native. Trying that, now that I'm on Graviton2 natively, does work.

If you're native on Graviton right now, try adding <ItemGroup><IlcArg Include="--verbose" /></ItemGroup> with native.

It will dump the list of instruction sets it expanded native into at compile time. We can crosscheck with what's expected.

[MichalStrehovsky]

@EgorBo, @MichalStrehovsky, what do you think about dumping all the supported instruction sets in this error path:

Yes, I wanted to change this to PalPrintFatalError("\nThe required instruction sets are not supported by the current CPU (Guru Meditation %d %d).\n", g_cpuFeatures, g_requiredCpuFeatures);. We should do that.

[chrisoverzero]

If you're native on Graviton right now, try adding <ItemGroup><IlcArg Include="--verbose" /></ItemGroup> with native.

Done.

ILC: The 'native' instruction set expanded to +aes,+crc,+lse,+neon,+sha1,+sha2

[MichalStrehovsky]

I did a quick check in #89988 and this is a product-build-time issue. I think this is a general product issue, not Native AOT specific. If this #define is not set, we would not generate a check for this at all.

/__w/1/s/src/native/minipal/cpufeatures.c:359:2: error: No HWCAP_LRCPC
#error No HWCAP_LRCPC
 ^
1 error generated.

[MichalStrehovsky]

I'll try on my M1

It would likely not reproduce on M1 because Apple has its own way to detect these. This would be Linux specific.

[chrisoverzero]

Fantastic. I look forward to this landing in the daily build. Thanks to all for your efforts.

[chrisoverzero]

I confirm that with SDK version 8.0.100-rc.1.23407.1 and:

<PackageReference Include="Microsoft.DotNet.ILCompiler" Version="8.0.0-rc.1.23406.6" />
<PackageReference Include="runtime.linux-x64.Microsoft.DotNet.ILCompiler" Version="8.0.0-rc.1.23406.6" />

…the problem is resolved. I don't know what daily build will incorporate these versions into the SDK, but I can keep working with this. Thanks again.