Can't connect to AWS Elasticache using encryption in transit (required) + at rest

[tvdias]

Hello,

We're using AWS Elasticache with in-transit + at rest encryption, but having certificate validation issues.

In order to find the issue, we've added an health check to the service:

        builder.Services
            .AddHealthChecks()
            .AddRedis(_ =>
                {
                    return ConnectionMultiplexer.Connect(
                        builder.Configuration.GetValue<string>("RedisAddresses"),
                        (config) =>
                        {
                            var sslHost = builder.Configuration.GetValue<string>("RedisSslHost");
                            var user = builder.Configuration.GetValue<string>("RedisUser");
                            var password = builder.Configuration.GetValue<string>("RedisPassword");
                            var ssl = builder.Configuration.GetValue("RedisSslEnabled", true);
                            var enabledSslProtocols = builder.Configuration.GetValue("RedisEnabledSslProtocols", SslProtocols.Tls13);
                            var certificateRevocationCheckMode = builder.Configuration.GetValue("RedisCertificateRevocationCheckMode", X509RevocationMode.Online);
                            var skipServerCertificateValidation = builder.Configuration.GetValue("RedisSkipServerCertificateValidation", false);
                            var sslClientAuthenticationOptions = new SslClientAuthenticationOptions()
                            {
                                EnabledSslProtocols = enabledSslProtocols,
                                CertificateRevocationCheckMode = certificateRevocationCheckMode,
                                RemoteCertificateValidationCallback = RemoteCertificateValidationCallback
                            };

                            if (!string.IsNullOrEmpty(sslHost)) config.SslHost = sslHost;
                            if (!string.IsNullOrEmpty(user)) config.User = user;
                            if (!string.IsNullOrEmpty(password)) config.Password = password;
                            config.Ssl = ssl;
                            config.AbortOnConnectFail = false;
                            config.SslClientAuthenticationOptions = _ => sslClientAuthenticationOptions;

                            bool RemoteCertificateValidationCallback(object _, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
                            {
                                Console.WriteLine("sslPolicyErrors is {0}", sslPolicyErrors);

                                if (sslPolicyErrors.HasFlag(SslPolicyErrors.RemoteCertificateNameMismatch))
                                {
                                    Console.WriteLine("RemoteCertificateNameMismatch error occurred.");

                                    if (certificate is X509Certificate2 cert)
                                    {
                                        var cn = cert.GetNameInfo(X509NameType.SimpleName, false);
                                        Console.WriteLine("Certificate Common Name (CN): {0}", cn);
                                        Console.WriteLine("SslHost (sslHost): {0}", sslHost);
                                    }
                                }

                                // Added bc X509RevocationMode.Offline didn't work
                                if (sslPolicyErrors.HasFlag(SslPolicyErrors.RemoteCertificateChainErrors))
                                {
                                    Console.WriteLine("RemoteCertificateChainErrors error occurred.");
                                }

                                return skipServerCertificateValidation || sslPolicyErrors == SslPolicyErrors.None;
                            }
                        });
                },
                "Redis",
                HealthStatus.Unhealthy);

On the configuration, sslHost has the same value as RedisAddresses (but without the port). We also tried with and without the domain wildcard. RedisAddresses is the "Configuration endpoint" provided on the AWS console. The CN on the certificate is the same as the RedisAdresses domain, with a wildcard.

The error we get is

StackExchange.Redis.RedisConnectionException: The message timed out in the backlog attempting to send because no connection became available (5000ms) - Last Connection Exception: AuthenticationFailure on XXXXXX.cache.amazonaws.com:6379/Interactive, Initializing/NotStarted, last: NONE, origin: ConnectedAsync, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, global: 0s ago, v: 2.8.12.45748, command=PING, timeout: 5000, inst: 0, qu: 0, qs: 0, aw: False, bw: CheckingForTimeout, last-in: 0, cur-in: 0, sync-ops: 0, async-ops: 1, serverEndpoint: XXXXX.cache.amazonaws.com:6379, conn-sec: n/a, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: XXXXX(SE.Redis-v2.8.12.45748), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=4,Free=32763,Min=32,Max=32767), POOL: (Threads=11,QueuedItems=0,CompletedItems=2875,Timers=10), v: 2.8.12.45748 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)\n ---> StackExchange.Redis.RedisConnectionException: AuthenticationFailure on XXXXX.cache.amazonaws.com:6379/Interactive, Initializing/NotStarted, last: NONE, origin: ConnectedAsync, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, global: 0s ago, v: 2.8.12.45748\n ---> System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.\n   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)\n   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)\n   at StackExchange.Redis.PhysicalConnection.ConnectedAsync(Socket socket, ILogger log, SocketManager manager) in /_/src/StackExchange.Redis/PhysicalConnection.cs:line 1580\n   --- End of inner exception stack trace ---\n   --- End of inner exception stack trace ---\n   at HealthChecks.Redis.RedisHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in /_/src/HealthChecks.Redis/RedisHealthCheck.cs:line 75

sslPolicyErrors is RemoteCertificateNameMismatch
RemoteCertificateNameMismatch error occurred.
Certificate Common Name (CN): *.XXX.cache.amazonaws.com
SslHost (SslHost): clustercfg.XXX.cache.amazonaws.com

We haven't added any certificate to the service. The service is running on a kubernetes cluster deployed on this same AWS environment. Non c# services can connect to the cluster. The service is using a dotnet 8 FIPS image.

Can you provide some help/guidance/suggestion on how can we make the dotnet certificate validation work in this case?

Many thanks!

[xrascal]

I have encountered the same problem as you:
The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch
I have added the following configuration:
ssl=true, sslProtocols=tls12, sslHost=*. XXx.mache.amazonaws.com
And then the problem was solved. I am using DotNet 8.

[mgravell]

I wonder whether we should special-case this scenario, especially if we detect that we're talking to AWS. The tricky thing is: we're not AWS consumers, and historically: attempts to secure an endpoint that we can actually talk to for integration testing / validation have been unsuccessful (despite talking to AWS folks directly). We can dry to develop a patch in isolation, but I'd be nervous pushing such a fix without someone somewhere being able to validate it.

Hint hint.