.NET Core IoTDataClient can't connect to IoT Custom Domains which do not contain the AWS Region in the URL

Describe the bug

The .Net Core AmazonIotDataClient can't connect to custom IoT domains which do not contain the AWS region in the domain name.
Please note this specifically does not work in dotnet core; works as expected with other IotDataClients (such as python or go).

IoT Domain Name: connect-iot.integration.openmethodscloud.com

Relevant Code

string iotEndpoint = GlobalProperty.IOT_ENDPOINT;
_awsIotDataConfig = new AmazonIotDataConfig();
_awsIotDataConfig.RegionEndpoint = RegionEndpoint.USWest2;
_awsIotDataConfig.ServiceURL = "https://" + iotEndpoint;

internal AmazonIotDataClient GetIotDataClient()
{
  string _accessKey = GlobalProperty.ACCESS_KEY;
  string _secretKey = GlobalProperty.SECRET_KEY;
  if(!string.IsNullOrEmpty(_accessKey) && !string.IsNullOrEmpty(_secretKey))
  {
    Console.WriteLine("Using Existing Credentials");
    return new AmazonIotDataClient(_accessKey, _secretKey, _awsIotDataConfig);
  }
  Console.WriteLine("Using Container Credentials");
  return new AmazonIotDataClient(_awsIotDataConfig);
}

AmazonIotDataClient iotDataClient = GetIotDataClient();
PublishResponse publishResponse = await iotDataClient.PublishAsync(iotRequest);

Expected Behavior

Using Container Credentials

UserCrypto 1|2022-06-08T18:58:01.509Z|INFO|UserCrypto is not supported.  This may be due to use of a non-Windows operating system or Windows Nano Server, or the current user account may not have its profile loaded. Unable to load shared library 'Crypt32.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libCrypt32.dll: cannot open shared object file: No such file or directory
EnvironmentVariablesAWSCredentials 2|2022-06-08T18:58:01.523Z|INFO|Credentials found using environment variables.
EnvironmentVariableInternalConfiguration 3|2022-06-08T18:58:01.528Z|INFO|The environment variable AWS_ENABLE_ENDPOINT_DISCOVERY was not set with a value.
EnvironmentVariableInternalConfiguration 4|2022-06-08T18:58:01.528Z|INFO|The environment variable AWS_MAX_ATTEMPTS was not set with a value.
EnvironmentVariableInternalConfiguration 5|2022-06-08T18:58:01.528Z|INFO|The environment variable AWS_RETRY_MODE was not set with a value.
EnvironmentVariableInternalConfiguration 6|2022-06-08T18:58:01.529Z|INFO|The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT was not set with a value.
EnvironmentVariableInternalConfiguration 7|2022-06-08T18:58:01.529Z|INFO|The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE was not set with a value.
EnvironmentVariableInternalConfiguration 8|2022-06-08T18:58:01.529Z|INFO|The environment variable AWS_USE_DUALSTACK_ENDPOINT was not set with a value.
EnvironmentVariableInternalConfiguration 9|2022-06-08T18:58:01.529Z|INFO|The environment variable AWS_USE_FIPS_ENDPOINT was not set with a value.
ProfileInternalConfiguration 10|2022-06-08T18:58:01.531Z|INFO|Unable to find a profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
DefaultConfigurationProvider 11|2022-06-08T18:58:01.533Z|INFO|Resolved DefaultConfigurationMode for RegionEndpoint [] to [Legacy].
EnvironmentVariablesAWSCredentials 12|2022-06-08T18:58:01.563Z|INFO|Credentials found using environment variables.
AWSSDKUtils 13|2022-06-08T18:58:01.629Z|DEBUG|Double encoded /topics/{topic} with endpoint https://connect-iot.integration.openmethodscloud.com/ for canonicalization: /topics/CTI%252F8dc862288a5a473ea538b4732f63aff2%252Fcti%252Fvbollapalli44mm355test2334
AmazonIotDataClient 14|2022-06-08T18:58:01.972Z|DEBUG|Received response (truncated to 1024 bytes): []

Current Behavior

Using Container Credentials

UserCrypto 1|2022-06-08T18:55:53.170Z|INFO|UserCrypto is not supported.  This may be due to use of a non-Windows operating system or Windows Nano Server, or the current user account may not have its profile loaded. Unable to load shared library 'Crypt32.dll' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libCrypt32.dll: cannot open shared object file: No such file or directory
EnvironmentVariablesAWSCredentials 2|2022-06-08T18:55:53.181Z|INFO|Credentials found using environment variables.
EnvironmentVariableInternalConfiguration 3|2022-06-08T18:55:53.191Z|INFO|The environment variable AWS_ENABLE_ENDPOINT_DISCOVERY was not set with a value.
EnvironmentVariableInternalConfiguration 4|2022-06-08T18:55:53.191Z|INFO|The environment variable AWS_MAX_ATTEMPTS was not set with a value.
EnvironmentVariableInternalConfiguration 5|2022-06-08T18:55:53.191Z|INFO|The environment variable AWS_RETRY_MODE was not set with a value.
EnvironmentVariableInternalConfiguration 6|2022-06-08T18:55:53.191Z|INFO|The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT was not set with a value.
EnvironmentVariableInternalConfiguration 7|2022-06-08T18:55:53.192Z|INFO|The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE was not set with a value.
EnvironmentVariableInternalConfiguration 8|2022-06-08T18:55:53.192Z|INFO|The environment variable AWS_USE_DUALSTACK_ENDPOINT was not set with a value.
EnvironmentVariableInternalConfiguration 9|2022-06-08T18:55:53.192Z|INFO|The environment variable AWS_USE_FIPS_ENDPOINT was not set with a value.
ProfileInternalConfiguration 10|2022-06-08T18:55:53.194Z|INFO|Unable to find a profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
DefaultConfigurationProvider 11|2022-06-08T18:55:53.197Z|INFO|Resolved DefaultConfigurationMode for RegionEndpoint [] to [Legacy].
EnvironmentVariablesAWSCredentials 12|2022-06-08T18:55:53.227Z|INFO|Credentials found using environment variables.
RegionFinder 13|2022-06-08T18:55:53.264Z|INFO|Unable to find exact matched region in endpoint connect-iot.integration.openmethodscloud.com
RegionFinder 14|2022-06-08T18:55:53.273Z|INFO|Unable to find fuzzy matched region in endpoint connect-iot.integration.openmethodscloud.com
AWSSDKUtils 15|2022-06-08T18:55:53.302Z|DEBUG|Double encoded /topics/{topic} with endpoint https://connect-iot.integration.openmethodscloud.com/ for canonicalization: /topics/CTI%252F8dc862288a5a473ea538b4732f63aff2%252Fcti%252Fvbollapalli44mm355test2334
AmazonIotDataClient 16|2022-06-08T18:55:53.635Z|ERROR|An exception of type HttpErrorResponseException was handled in ErrorHandler. --> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonIotDataClient 17|2022-06-08T18:55:53.652Z|ERROR|Received error response: [{"message":"Forbidden","traceId":"6c0c2d0f-cb1c-fd3c-7f10-1e397145fce7"}] --> Amazon.IotData.AmazonIotDataException: Forbidden
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
AmazonIotDataClient 18|2022-06-08T18:55:53.659Z|ERROR|AmazonIotDataException making request PublishRequest to https://connect-iot.integration.openmethodscloud.com/. Attempt 1. --> Amazon.IotData.AmazonIotDataException: Forbidden
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)

Reproduction Steps

Method Described Above

1. Create an IoT Custom Domain ( foobar.example.com ) with certificate
2. CNAME to IoT Endpoint in Route53 (or perferred DNS provider) for foobar.example.com
3. Use .NET core IotDataClient to publish to that endpoint.
4. Receive Forbidden Error

Alternate (faster) Way

1. Create IoT Custom Domain ( foobar.us-west-2.example.com ) with certificate
2. CNAME to IoT Endpoint in Route53 (or perferred DNS provider) for foobar.us-west-2.example.com
3. Configure .NET core IotDataConfig to any region that is NOT the regions endpoint ( us-east-1 ).
4. Use .NET core IotDataClient to publish to foobar.us-west-2.example.com
5. Observe that it still works even though it SHOULDN'T because the configured endpoint region is different.

Possible Solution

The SDK appears to be using the URL to determine the AWS Region as indicated by:

RegionFinder 13|2022-06-08T18:55:53.264Z|INFO|Unable to find exact matched region in endpoint connect-iot.integration.openmethodscloud.com
RegionFinder 14|2022-06-08T18:55:53.273Z|INFO|Unable to find fuzzy matched region in endpoint connect-iot.integration.openmethodscloud.com

instead of the region provided by

_awsIotDataConfig = new AmazonIotDataConfig();
_awsIotDataConfig.RegionEndpoint = RegionEndpoint.USWest2;
_awsIotDataConfig.ServiceURL = "https://" + iotEndpoint;

I would suggest the SDK uses the region provided in the AmazonIoTDAtaConfig();

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.IotData, 3.7.2.19

Targeted .NET Platform

ASP.NET Core Runtime 5.0.17

Operating System and version

Ubuntu, Alpine Linux

[ashishdhingra]

Hi @kyle-davison-openmethods,

Good morning.

Thanks for reporting the issue. From the logs shared by you, the error appears to be Forbidden error:

at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
at Amazon.Runtime.Internal.HttpHandler1.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) AmazonIotDataClient 17|2022-06-08T18:55:53.652Z|ERROR|Received error response: [{"message":"Forbidden","traceId":"6c0c2d0f-cb1c-fd3c-7f10-1e397145fce7"}] --> Amazon.IotData.AmazonIotDataException: Forbidden ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown. at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken) at Amazon.Runtime.Internal.HttpHandler1.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
--- End of inner exception stack trace ---
AmazonIotDataClient 18|2022-06-08T18:55:53.659Z|ERROR|AmazonIotDataException making request PublishRequest to https://connect-iot.integration.openmethodscloud.com/. Attempt 1. --> Amazon.IotData.AmazonIotDataException: Forbidden
---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
at Amazon.Runtime.Internal.HttpHandler1.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) --- End of inner exception stack trace --- at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream) at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception) at Amazon.Runtime.Internal.ExceptionHandler1.HandleAsync(IExecutionContext executionContext, Exception exception)
at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)
at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)

I'm unsure if this is related to region resolution. Is it possible to share the verbose detailed logs of your execution? (refer #1759 (comment) on how to enable it for .NET Core projects).

Thanks,
Ashish

[github-actions]

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[ashovlin]

Can you try setting the AuthenticationRegion property on the client config with the region where the domain is configured?

var config = new AmazonIotDataConfig
{
    ServiceURL = "<custom domain>",
    AuthenticationRegion = "<region>" // e.g. "us-east-1"
};

var client = new AmazonIotDataClient(config);

Docs: https://docs.aws.amazon.com/sdkfornet/v3/apidocs/index.html?page=Runtime/TClientConfig.html&tocid=Amazon_Runtime_ClientConfig

Background: Even for a custom domain, the SDK will sign the request with SigV4. The region is one of the inputs to request signing, and can be seen in the Authorization header of the request.

If an AuthenticationRegion is not provided, the SDK will attempt to parse it out of the ServiceURL which is failing in this case. AuthenticationRegion is preferred during request signing, over RegionEndpoint which is primarily for endpoint construction when not using a custom ServiceURL