Null Reference when deploying to k8s

[ilariosa]

I am trying to deploy a c# service that accesses s3 using aws-sdk-net in k8s but I keep on getting a null reference error.

error:

info: AWSSDK[0]
      Found credentials using the AWS SDK's default credential search
Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at Amazon.Extensions.NETCore.Setup.ClientFactory.CreateConfig(Type, AWSOptions) + 0x8a
   at Amazon.Extensions.NETCore.Setup.ClientFactory.CreateServiceClient(ILogger, Type, AWSOptions) + 0xfe
   at Amazon.Extensions.NETCore.Setup.ClientFactory.CreateServiceClient(IServiceProvider provider) + 0xf0

Referenced packages:

    <PackageReference Include="AWSSDK.Core" Version="3.7.400.13" />
    <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.7.301" />
    <PackageReference Include="AWSSDK.SecurityToken" Version="3.7.400.13" />
    <PackageReference Include="AWSSDK.SSO" Version="3.7.400.13" />
    <PackageReference Include="AWSSDK.SSOOIDC" Version="3.7.400.13" 

Startup code:

            var config = new ConfigurationBuilder()
                .AddEnvironmentVariables()
                .Build();

            builder.Services.AddDefaultAWSOptions(config.GetAWSOptions());
            builder.Services.AddAWSService<IAmazonS3>();

Platform version: eks.11

Environment Variables:

AWS_STS_REGIONAL_ENDPOINTS=regional
AWS_DEFAULT_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::***:role/****
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Is there something I am missing?

[dscpinheiro]

Did you start seeing this error after upgrading to the latest version of the AWSSDK.Extensions.NETCore.Setup package?

If possible (and you could share it), I'm also curious on what properties you have on the AWS section of your appsettings file (I'm wondering if there's a property our package is not handling correctly).

[ilariosa]

this is a new app, and also our first time to try deploying a c# service to k8s (all our typescript apps using aws sdk with same k8s setup are running without problem.

For the appsettings:

{
  "AWS": {
    "Region": "eu-west-1"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*"
}

The app is setup in k8s to use IAM role. even tried calling AssumeRoleWithWebIdentityCredentials.FromEnvironmentVariables() and can get credentials.

One thing I noticed though now are these warnings when building the docker image:

#10 16.18 /src/SampleApp/Startup.cs(46,13): warning IL2026: Using member 'Microsoft.Extensions.DependencyInjection.ServiceCollectionExtensions.AddDefaultAWSOptions(IServiceCollection, AWSOptions)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. The AWSSDK.Extensions.NETCore.Setup package has not been updated to support Native AOT compilations. [/src/SampleApp/SampleApp.csproj]
#10 16.18 /src/SampleApp/Startup.cs(46,51): warning IL2026: Using member 'Microsoft.Extensions.Configuration.ConfigurationExtensions.GetAWSOptions(IConfiguration)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. The AWSSDK.Extensions.NETCore.Setup package has not been updated to support Native AOT compilations. [/src/SampleApp/SampleApp.csproj]
#10 16.18 /src/SampleApp/Startup.cs(54,13): warning IL2026: Using member 'Microsoft.Extensions.DependencyInjection.ServiceCollectionExtensions.AddAWSService<T>(IServiceCollection, ServiceLifetime)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. The AWSSDK.Extensions.NETCore.Setup package has not been updated to support Native AOT compilations. [/src/SampleApp/SampleApp.csproj]
#10 16.18 /src/SampleApp/Startup.cs(55,13): warning IL2026: Using member 'Microsoft.Extensions.DependencyInjection.ServiceCollectionExtensions.AddAWSService<T>(IServiceCollection, ServiceLifetime)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. The AWSSDK.Extensions.NETCore.Setup package has not been updated to support Native AOT compilations. [/src/SampleApp/SampleApp.csproj]

here's a snippet for the project file:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <InvariantGlobalization>true</InvariantGlobalization>
    <PublishAot>true</PublishAot>
    <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
      <JsonSerializerIsReflectionEnabledByDefault>true</JsonSerializerIsReflectionEnabledByDefault>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AWSSDK.Core" Version="3.7.400.13" />
    <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.7.301" />
    <PackageReference Include="AWSSDK.SecretsManager" Version="3.7.400.13" />
    <PackageReference Include="AWSSDK.SecurityToken" Version="3.7.400.13" />
    <PackageReference Include="AWSSDK.SSO" Version="3.7.400.13" />
    <PackageReference Include="AWSSDK.SSOOIDC" Version="3.7.400.13" />
    <PackageReference Include="AWSSDK.SQS" Version="3.7.301.1" />

and also snippet of the DockerFile

.....
FROM build AS publish
ARG BUILD_CONFIGURATION=Release
RUN dotnet publish "./SampleApp.csproj" -c $BUILD_CONFIGURATION -o /app/publish /p:UseAppHost=true

FROM mcr.microsoft.com/dotnet/runtime-deps:8.0 AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["./SampleApp"]

[dscpinheiro]

Oh, you're using AOT... Unfortunately version 3.7.301 of the AWSSDK.Extensions.NETCore.Setup package is not Native AOT compatible (which explains the warnings you saw during the build). Do you still see the error without the PublishAot option?

We also have a preview version of AWSSDK.Extensions.NETCore.Setup that is compatible with AOT, but it'll require you to update the SDK packages too: https://www.nuget.org/packages/AWSSDK.Extensions.NETCore.Setup/4.0.0-preview.2 (this issue has a list of all breaking changes currently included in the V4 preview: #3362)

[ilariosa]

I tried it immediately after seeing the warnings, and it's indeed the cause of the null reference error! thanks!