SSMXmlRepository.GetAllElementsAsync fails to call SSM when XRay is enabled

[madmox]

Describe the bug

I have an ASP.NET application deployed on ECS Fargate, and I want to use SSM for data protection. I also use XRay for tracing. When the application starts, it fails because it can't read SSM (even though I setup the correct IAM permissions in the task's role).

Expected Behavior

The application should start normally.

Current Behavior

The following exceptions are thrown:

An error occurred while reading the key ring.

Stack Trace:
Amazon.XRay.Recorder.Core.Exceptions.EntityNotAvailableException: Entity doesn't exist in AsyncLocal
  File "/_/sdk/src/Core/Internal/Context/AsyncLocalContextContainer.netstandard.cs", line 45, col 17, in Entity AsyncLocalContextContainer.GetEntity()
  File "/_/sdk/src/Core/AWSXRayRecorderImpl.cs", line 968, col 13, in Entity AWSXRayRecorderImpl.GetEntity()
  File "/_/sdk/src/Handlers/AwsSdk/Internal/XRayPipelineHandler.cs", line 312, col 17, in void XRayPipelineHandler.ProcessBeginRequest(IExecutionContext executionContext)
  File "/_/sdk/src/Core/Internal/Context/TraceContextImpl.cs", line 71, col 17, in void TraceContextImpl.HandleEntityMissing(IAWSXRayRecorder recorder, Exception e, string message)
  File "/_/sdk/src/Handlers/AwsSdk/Internal/XRayPipelineHandler.cs", line 316, col 17, in void XRayPipelineHandler.ProcessBeginRequest(IExecutionContext executionContext)
  File "/_/sdk/src/Handlers/AwsSdk/Internal/XRayPipelineHandler.cs", line 695, col 17, in async Task<T> XRayPipelineHandler.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> CallbackHandler.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> ErrorCallbackHandler.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> MetricsHandler.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<IReadOnlyCollection<XElement>> SSMXmlRepository.GetAllElementsAsync()
  ?, in IReadOnlyCollection<XElement> SSMXmlRepository.GetAllElements()
  ?, in IReadOnlyCollection<IKey> XmlKeyManager.GetAllKeys()
  ?, in CacheableKeyRing KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
  ?, in CacheableKeyRing KeyRingProvider.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.ICacheableKeyRingProvider.GetCacheableKeyRing(DateTimeOffset now)
  ?, in IKeyRing KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, bool forceRefresh)

Error calling SSM to get parameters starting with /MyApp/DataProtection/: Entity doesn't exist in AsyncLocal

Stack Trace:
  ?, in void Logger.Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)+LoggerLog(LogLevel logLevel, EventId eventId, ILogger logger, Exception exception, Func<TState, Exception, string> ...
  ?, in void Logger.Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)
  ?, in void Logger<T>.Microsoft.Extensions.Logging.ILogger.Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)
  ?, in void LoggerExtensions.Log(ILogger logger, LogLevel logLevel, EventId eventId, Exception exception, string message, params object[] args)
  ?, in void LoggerExtensions.LogError(ILogger logger, string message, params object[] args)
  ?, in async Task<IReadOnlyCollection<XElement>> SSMXmlRepository.GetAllElementsAsync()
  ?, in void AsyncMethodBuilderCore.Start<TStateMachine>(ref TStateMachine stateMachine)
  ?, in Task<IReadOnlyCollection<XElement>> SSMXmlRepository.GetAllElementsAsync()
  ?, in bool ThreadPoolWorkQueue.Dispatch()
  ?, in void WorkerThread.WorkerThreadStart()
  ?, in void Thread.StartCallback()

Reproduction Steps

void ConfigureServices(WebApplicationBuilder builder)
{
    AWSXRayRecorder.InitializeInstance(builder.Configuration);
    AWSSDKHandler.RegisterXRayForAllServices();
    builder.Services.AddDataProtection()
        .SetApplicationName(applicationName: "MyApp")
        .PersistKeysToAWSSystemsManager(parameterNamePrefix: "/MyApp/DataProtection");
}

AWS .NET SDK and/or Package version used

Amazon.AspNetCore.DataProtection.SSM 3.0.0

Targeted .NET Platform

.NET 6

Operating System and version

Docker image mcr.microsoft.com/dotnet/aspnet:6.0

[madmox]

The problem is that AWSXRayRecorder.Handlers.AspNetCore middleware, which is in charge of calling AWSXRayRecorder.Instance.BeginSegment() / AWSXRayRecorder.Instance.EndSegment() in the context of a request, does not apply in the context of application startup (when SSM is actually called by SSMXmlRepository).

I'm not sure where I should call these methods manually though... not even sure there's an extension point available for me to do this (creating the segment around the PersistKeysToAWSSystemsManager() call does not solve the problem, it looks like the SSM call is not performed here, which seems reasonable for a service registration method).

[normj]

Because the library is polling in the background outside of the scope of a request that creates the X-Ray segment. That is what is triggering the Entity not found exception.

The easiest workaround is to set the AWS_XRAY_CONTEXT_MISSING environment variable to LOG_ERROR.

Here is the documentation for more information.
https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-dotnet-configuration.html#xray-sdk-dotnet-configuration-envvars

[madmox]

Thanks for the workaround, I'll probably resort to that and disable the related logs to avoid polluting our error notifications channel.

It's a shame there's no way to have a more specific way to address this problem though, there might be cases where it's still legitimate to throw an error. Also, one may want to instrument these calls too. Maybe provide extension points around the call to SSM?

[ik130]

Hi madmox, as stated by normj, this is currently behaving as designed. However, we will change this request from a bug to a feature request and check if we can prioritize it.