Cannot connect to AWS RDS Aurora 3 (Mysql8) using SslCa=<file> and SslMode=VerifyCA

[frivard-coveo]

Software versions
MySqlConnector version: 2.3.5
Server type (MySQL, MariaDB, Aurora, etc.) and version: Aurora MySQL 3.04.0 (Compatible with MySQL 8.0.28)
.NET version: 8.0
(Optional) ORM NuGet packages and versions:

Describe the bug
Following our industry's leading security practices, I tried to upgrade from SslMode=Required to SslMode=VerifyCA. To do that I downloaded the AWS RDS "CA Bundle" at https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem (see doc) .

Unfortunately it did not work, it gives the exception The remote certificate was rejected by the provided RemoteCertificateValidationCallback

After a lot of debugging, I tried one more thing : instead of passing the SslCa file, I imported it into the Windows Trusted Root Certificate Store. And it worked; proofing that the CA bundle is valid and sufficient. That is a good confirmation, but is not a portable solution.

See the additional context for the rest of the info.

Exception

MySqlConnector.MySqlException (0x80004005): SSL Authentication Error  ---> System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.
    at System.Net.Security.SslStream.SendAuthResetSignal(ReadOnlySpan`1 alert, ExceptionDispatchInfo exception)
    at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
    at MySqlConnector.Core.ServerSession.InitSslAsync(ProtocolCapabilities serverCapabilities, ConnectionSettings cs, MySqlConnection connection, SslProtocols sslProtocols, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1434
    at MySqlConnector.Core.ServerSession.InitSslAsync(ProtocolCapabilities serverCapabilities, ConnectionSettings cs, MySqlConnection connection, SslProtocols sslProtocols, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 1472
    at MySqlConnector.Core.ServerSession.ConnectAsync(ConnectionSettings cs, MySqlConnection connection, Int64 startingTimestamp, ILoadBalancer loadBalancer, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ServerSession.cs:line 523
    at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 428
    at MySqlConnector.Core.ConnectionPool.ConnectSessionAsync(MySqlConnection connection, Action`4 logMessage, Int64 startingTimestamp, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 433
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 111
    at MySqlConnector.Core.ConnectionPool.GetSessionAsync(MySqlConnection connection, Int64 startingTimestamp, Int32 timeoutMilliseconds, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/ConnectionPool.cs:line 144
    at MySqlConnector.MySqlConnection.CreateSessionAsync(ConnectionPool pool, Int64 startingTimestamp, Activity activity, Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 919
    at MySqlConnector.MySqlConnection.OpenAsync(Nullable`1 ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlConnection.cs:line 419

Code sample

        var builder = new MySqlConnectionStringBuilder
        {
            Server = _mySqlConfig.Host,
            Port = _mySqlConfig.Port,
            Password = _mySqlConfig.Password,
            UserID = _mySqlConfig.Username,
            AllowLoadLocalInfile = true,
            AllowUserVariables = true,
            DefaultCommandTimeout = (uint)_mySqlConfig.CommandTimeout.TotalSeconds,
            SslMode = MySqlSslMode.VerifyCA,
            SslCa = _mySqlConfig.CACertificateFile,
            ConnectionTimeout = 999
        };

MySqlConnection connection = new MySqlConnection(builder.ToString());
await connection.OpenAsync(cancellationToken);

Expected behavior
I expect that the certificate present by AWS is validated by the CA bundle provided by AWS. It should be possible to do that via the SslCa option.

Additional context

The certificate chain presented by the AWS RDS server looks like (actual hostname redacted because... security...)

[Subject]  C=US, S=Washington, L=Seattle, O=Amazon.com, OU=RDS, CN=***redacted***.us-east-1.rds.amazonaws.com
[Issuer]  L=Seattle, CN=Amazon RDS us-east-1 Subordinate CA RSA2048 G1.A.10, S=WA, OU=Amazon RDS, O="Amazon Web Services, Inc.", C=US
[Serial Number]  00D7BEC99EA6A9A9F3E233BF8BDC696F97
[Not Before]  2024-03-15 9:37:26
[Not After]  2025-03-15 9:37:26
[Thumbprint]  424B43DAA5E6431865CB20664410539CC6A4F852


[Subject]  L=Seattle, CN=Amazon RDS us-east-1 Subordinate CA RSA2048 G1.A.10, S=WA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", C=US
[Issuer]  L=Seattle, CN=Amazon RDS us-east-1 Root CA RSA2048 G1, S=WA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", C=US
[Serial Number]  06EAB8D792ADBF818DF67EA9308A42CB
[Not Before]  2022-12-27 18:10:35
[Not After]  2032-12-27 19:10:35
[Thumbprint]  1D719047BFF523AE530227C8B5D57FACB87D8A7B

and the CA Bundle for us-east-1 looks like (the full global bundle is has 131 certs instead of 5, but we only care about 1 here):

[Subject]  CN=Amazon RDS Root 2019 CA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", S=Washington, L=Seattle, C=US
[Issuer]  CN=Amazon RDS Root 2019 CA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", S=Washington, L=Seattle, C=US
[Serial Number]  00C73467369250AE75
[Not Before]  2019-08-22 13:08:50
[Not After]  2024-08-22 13:08:50
[Thumbprint]  D40DDB29E3750DFFA671C3140BBF5F478D1C8096


[Subject]  CN=Amazon RDS us-east-1 2019 CA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", L=Seattle, S=Washington, C=US
[Issuer]  CN=Amazon RDS Root 2019 CA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", S=Washington, L=Seattle, C=US
[Serial Number]  2555
[Not Before]  2019-09-19 14:16:53
[Not After]  2024-08-22 13:08:50
[Thumbprint]  F0ED823ED14447BAB557FDF3E4927466988C1C78


[Subject]  L=Seattle, CN=Amazon RDS us-east-1 Root CA RSA2048 G1, S=WA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", C=US
[Issuer]  L=Seattle, CN=Amazon RDS us-east-1 Root CA RSA2048 G1, S=WA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", C=US
[Serial Number]  00F55231F162B663393E199B68E16819F5
[Not Before]  2021-05-25 18:34:57
[Not After]  2061-05-25 19:34:57
[Thumbprint]  2FA77EF894D983BA9D37AD699C84AB0F657BE1C8


[Subject]  L=Seattle, CN=Amazon RDS us-east-1 Root CA RSA4096 G1, S=WA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", C=US
[Issuer]  L=Seattle, CN=Amazon RDS us-east-1 Root CA RSA4096 G1, S=WA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", C=US
[Serial Number]  6911DA12AA9A717376D1EF33649B660C
[Not Before]  2021-05-25 18:38:35
[Not After]  2121-05-25 19:38:35
[Thumbprint]  9DA6FA7FD2EC09C569A400D876B01B0C12759A96


[Subject]  L=Seattle, CN=Amazon RDS us-east-1 Root CA ECC384 G1, S=WA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", C=US
[Issuer]  L=Seattle, CN=Amazon RDS us-east-1 Root CA ECC384 G1, S=WA, OU=Amazon RDS, O=""Amazon Web Services, Inc."", C=US
[Serial Number]  00F025124F1524F984CD5451696BD38760
[Not Before]  2021-05-25 18:41:55
[Not After]  2121-05-25 19:41:55
[Thumbprint]  24A97B91CBE86911190576C35C36AAB4FA7B25DE

However, in the custom certificate validation code around https://github.com/mysql-net/MySqlConnector/blob/master/src/MySqlConnector/Core/ServerSession.cs#L1390
, we have variable caCertificateChain built with the 5 root certificates in ExtraStore (see https://github.com/mysql-net/MySqlConnector/blob/master/src/MySqlConnector/Core/ServerSession.cs#L1368). This is okay.
But at line 1390, the callback receives as rcbCertificate only the base certificate (thumbprint 424B43DAA5E6431865CB20664410539CC6A4F852), issued by "Subordinate CA", and the other variable rcbChain contains both the base server certificate and the subordinate certificate.

Later at line 1398 when doing caCertificateChain.Build((X509Certificate2) rcbCertificate) we are attempting to verify the base certificate, but the chain is missing the subordinate, so it will always fail.

I am no expert on the .NET X509Chain API, but I would think that we must start with rcbChain.ChainElements.Last(), verify that one, and then work our way back the input chain until we hit the leaf.

[frivard-coveo]

I just found this helpful link : dotnet/runtime#26449 (comment)

which suggests

The thing to do at that point is read chain.ChainElements[chain.ChainElements.Count - 1].Certificate and determine if it is a certificate that you trust.

[frivard-coveo]

(even more digging...)
I found that the C# Kubernetes client had a similar issue, requested guidance to the .net maintainers, and arrived at the following implementation:
https://github.com/kubernetes-client/csharp/blob/master/src/KubernetesClient/Kubernetes.ConfigInit.cs#L188

[bgrainger]

I would think that we must start with rcbChain.ChainElements.Last(), verify that one

This looks to be the key change that is necessary. Instead of calling .Build(rcbCertificate) (in the RemoteCertifcateValidationCallback), we need to verify the final certificate in the chain that is presented to us, i.e., .Build(rcbChain.ChainElements[^1].Certificate). I'm not an X509Chain API expert either, but my assumption is that this can connect two chains of certificates--one presented by the remote host, and one from the CA bundle--together, allowing us to verify the certificate chain up to a root certificate that we do choose to trust.

It's a bit ugly, but you can use MySqlConnection.RemoteCertificateValidationCallback to work around this bug until it's fixed:

// load the AWS PEM bundle
var caCollection = new X509Certificate2Collection();
caCollection.ImportFromPemFile(@"C:\Path\To\global-bundle.pem");

// construct a chain that allows us to use our own root CA
X509Chain certificateChain = new()
{
	ChainPolicy =
	{
		RevocationMode = X509RevocationMode.NoCheck,
		VerificationFlags = X509VerificationFlags.AllowUnknownCertificateAuthority,
	},
};
certificateChain.ChainPolicy.ExtraStore.AddRange(caCollection);

using (var connection = new MySqlConnection(csb.ConnectionString))
{
	// use a custom RemoteCertificateValidationCallback
	connection.RemoteCertificateValidationCallback = (object sender, X509Certificate? certificate, X509Chain? chain, SslPolicyErrors sslPolicyErrors) =>
	{
		// same as MySqlConnector's implementation but builds using chain.ChainElements[^1]
		if ((sslPolicyErrors & SslPolicyErrors.RemoteCertificateChainErrors) != 0 &&
			certificate is not null &&
			certificateChain is not null &&
			certificateChain.Build(chain.ChainElements[^1].Certificate) &&
			certificateChain.ChainStatus.Length > 0)
		{
			// now also verify that the root is part of the SSL CA bundle
			var rootCertificate = certificateChain.ChainElements[^1].Certificate;
			foreach (var sslCaCertificate in certificateChain.ChainPolicy.ExtraStore)
			{
				if (rootCertificate.RawData.SequenceEqual(sslCaCertificate.RawData))
				{
					var chainStatus = certificateChain.ChainStatus[0].Status & ~X509ChainStatusFlags.UntrustedRoot;
					if (chainStatus == X509ChainStatusFlags.NoError)
						sslPolicyErrors &= ~SslPolicyErrors.RemoteCertificateChainErrors;
					break;
				}
			}
		}
		
		return sslPolicyErrors == SslPolicyErrors.None;
	};

	connection.Open();
}

[frivard-coveo]

Wow, that was very responsive.
Thank you and keep up the good work 👍

[bgrainger]

Thanks for the well-written bug report with helpful diagnosis!

[bgrainger]

Fixed in 2.3.6.