Instrumenting failed AWS SDK requests throws due to AWS's use of UriCreationOptions.DangerousDisablePathAndQueryCanonicalization

[tloten]

Please mark the type framework used:

• ASP.NET Core

Please mark the type of the runtime used:

• .NET Core 8.0

Please mark the NuGet packages used:

• Sentry 4.6.2
• Sentry.AspNetCore 4.6.2

After updating .Net 7 -> 8, we started getting a few errors when doing AWS S3 accesses:

System.InvalidOperationException
GetComponents() may not be used for Path/Query on a Uri instance created with UriCreationOptions.DangerousDisablePathAndQueryCanonicalization.

Specifically, it happens when S3 500's, and Sentry attempts to instrument/inspect the error. It doesn't seem to occur on 40x responses from S3.

In .Net 8+, the AWS SDK explicitly sets DangerousDisablePathAndQueryCanonicalization when constructing URIs. See this line in the AWS codebase for details. Sentry then later calls .GetComponents() on these URIs, which is evidently not allowed.

I've created a minimal repro here: https://gist.github.com/tloten/6e3c69fa19fba91773f944ada6de24ae

Full stack trace:

System.InvalidOperationException: GetComponents() may not be used for Path/Query on a Uri instance created with UriCreationOptions.DangerousDisablePathAndQueryCanonicalization.
  File "UriExt.cs", line 335, in string Uri.GetComponents(UriComponents components, UriFormat format)
    throw new InvalidOperationException(SR.net_uri_GetComponentsCalledWhenCanonicalizationDisabled);
  ?, in void SentryHttpFailedRequestHandler.DoEnsureSuccessfulResponse(HttpRequestMessage request, HttpResponseMessage response)
  ?, in void SentryHttpMessageHandler.HandleResponse(HttpResponseMessage response, ISpan span, string method, string url)
  ?, in async Task<HttpResponseMessage> SentryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  File "HttpClient.cs", line 530, in async Task<HttpResponseMessage> HttpClient.SendAsync(HttpRequestMessage request)+Core(?)
    response = await base.SendAsync(request, cts.Token).ConfigureAwait(false);
  ?, in async Task<IWebResponseData> HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
  ?, in async Task<T> HttpHandler<TRequestContent>.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> RedirectHandler.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> Unmarshaller.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> AmazonS3ResponseHandler.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> ErrorHandler.InvokeAsync<T>(IExecutionContext executionContext) x 2
  ?, in async Task<T> CallbackHandler.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> Signer.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> S3ExpressPreSigner.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> EndpointDiscoveryHandler.InvokeAsync<T>(IExecutionContext executionContext) x 2
  ?, in async Task<T> CredentialsRetriever.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> RetryHandler.InvokeAsync<T>(IExecutionContext executionContext) x 2
  ?, in async Task<T> CallbackHandler.InvokeAsync<T>(IExecutionContext executionContext) x 2
  ?, in async Task<T> AmazonS3ExceptionHandler.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> ErrorCallbackHandler.InvokeAsync<T>(IExecutionContext executionContext)
  ?, in async Task<T> MetricsHandler.InvokeAsync<T>(IExecutionContext executionContext)
  File "S3ImageAccessor.cs", line 89, col 13, in async Task S3ImageAccessor.UploadImage(...)   <--- our code here, everything above is AWS/Sentry/dotnet

[jamescrosswell]

Hi @tloten, thanks for the report. I wasn't able to reproduce the error you mentioned using your gist however. I'm just getting a System.Net.Http.HttpRequestException: nodename nor servname provided, or not known (foo.localhost:5236).

Are there some prerequisites perhaps? I have to have some AWS CLI tools or something installed on my machine first, for example?

[tloten]

Hi @jamescrosswell
Sorry about that - on my machine *.localhost resolves the same as localhost, I had no idea it was doing that!

Anyway, can work around that by adding ForcePathStyle = true to the AmazonS3Config. Ala

...
    // Point S3 at ourselves so we can simulate a 500 error happening in S3
    var s3Config = new AmazonS3Config()
    {
        ServiceURL = app.Urls.First(),
        ForcePathStyle = true,
        UseHttp = true,
    };
...

(I've updated the gist too if that's easier)

That just makes the S3Client use path based URLs (e.g. http://baseurl/bucket/key rather than http://bucket.baseurl/key) - again just a workaround to get the S3 client pointed at our dummy endpoint so we can simulate 500's.

Really unsure of the solution here - its quite odd that AWS SDK is using this DangerousDisablePathAndQueryCanonicalization option. And there doesn't seem to be a way to inspect a URI to find out if it's safe to call GetComponents either 🤔

[jamescrosswell]

Anyway, can work around that by adding ForcePathStyle = true to the AmazonS3Config. Ala

Cool, I can reproduce now! That's always the first step 😜