AWS Lambda .NET 7 AOT (Native) does not start when provisioned concurrency is used

[vitalybibikov]

Describe the bug

When .NET 7 AOT lamdba is being deployed to al2 runtime (Custom runtime on Amazon Linux 2)
it's impossible for it to create Provisioned concurrency as it errors with:

Custom runtime on Amazon Linux 2

'Failed FUNCTION_ERROR_INIT_FAILURE(0/10)'

In logs I can see:

Expected Behavior

Provisioned concurrency works as expected, function initializes and starts.

Current Behavior

Reproduction Steps

We are having this issue in production for our functions ,but it is reproducable for a simple lambda.NativeAOT template:

1. Create new lambda project => dotnet new lambda.NativeAOT
2. Deploy it => dotnet lambda deploy-function
3. Create version, e.g. 1
4. Go to AWS Console => Lamdba => Configuration => Concurrency
5. Create Provisioned concurrency configuration
6. Wait

Possible Solution

No response

Additional Information/Context

Project configuration:

<PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <AWSProjectType>Lambda</AWSProjectType>
    <AssemblyName>bootstrap</AssemblyName>
    <!-- This property makes the build directory similar to a publish directory and helps the AWS .NET Lambda Mock Test Tool find project dependencies. -->
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    <!-- Generate Native AOT image during publishing to improve cold start time. -->
    <PublishAot>true</PublishAot>
    <!-- StripSymbols tells the compiler to strip debugging symbols from the final executable if we're on Linux and put them into their own file. 
    This will greatly reduce the final executable's size.-->
    <StripSymbols>true</StripSymbols>
  </PropertyGroup>

AWS .NET SDK and/or Package version used

<PackageReference Include="Amazon.Lambda.RuntimeSupport" Version="1.8.2" />
<PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.0" />

Targeted .NET Platform

.NET 7

Operating System and version

AmazonLinux2 (provided.al2)

[ashishdhingra]

Reproducible. Needs review with the team.

Got trim warning when deploying the function:

... docker run: /tmp/dotnet/.nuget/packages/amazon.lambda.serialization.systemtextjson/2.3.0/lib/net6.0/Amazon.Lambda.Serialization.SystemTextJson.dll : warning IL2104: Assembly 'Amazon.Lambda.Serialization.SystemTextJson' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [/tmp/source/LambdaNativeAOTTest.csproj]
... docker run: /tmp/dotnet/.nuget/packages/amazon.lambda.runtimesupport/1.8.2/lib/net6.0/Amazon.Lambda.RuntimeSupport.dll : warning IL2104: Assembly 'Amazon.Lambda.RuntimeSupport' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [/tmp/source/LambdaNativeAOTTest.csproj]

Also need to refer Assembly trimming as mentioned in blog post Building serverless .NET applications on AWS Lambda using .NET 7 for workaround for trimming issues and see if this error is related.

[normj]

I have reproduced the problem as well and think I know the fix which will require a patch to Amazon.Lambda.Runtime. Let me do some more confirmation and then I'll put out a PR.

[normj]

Version 1.8.4 of Amazon.Lambda.RuntimeSupport has been released that fixes the issue. Please update your reference to this package and redeploy.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.