Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to use SslMode=VerifyFull when running on AWS Lambda #498

Closed
paya-cz opened this issue May 18, 2018 · 20 comments
Closed

Unable to use SslMode=VerifyFull when running on AWS Lambda #498

paya-cz opened this issue May 18, 2018 · 20 comments

Comments

@paya-cz
Copy link

paya-cz commented May 18, 2018

Steps to reproduce

  1. Launch AWS RDS Aurora in VPC, setup a MySQL user to use IAM database auth
  2. Create AWS Lambda in .NET Core, use Entity Framework and Pomelo.EntityFrameworkCore.MySql to connect to the Aurora. You have to setup roles and privileges and security groups such that IAM auth works and the lambda can connect to the VPC MySQL. The lambda runs in the same VPC.
  3. Use the following SSL cert (I tried both, both have the same issue):

I suspect messing with VPC and IAM auth is not necessary, as this looks to be a SSL issue. It's just a setup that I was using when I discovered the issue.

For reference, this is the OnConfiguring method of DbContext:

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    // Generate user authentication token using 
    var token = RDSAuthTokenGenerator.GenerateAuthToken(RegionEndpoint.USEast1, Endpoint, Port, UserID);

    // See https://mysql-net.github.io/MySqlConnector/connection-options/ for more info
    var connectionStringBuilder = new MySqlConnectionStringBuilder
    {
        Server = Endpoint,
        Port = Port,
        UserID = UserID,
        Password = token,
        Database = Schema,
        SslMode = MySqlSslMode.VerifyFull,
        CACertificateFile = Path.Combine(Environment.CurrentDirectory, "Certificates", "rds-ca-2015-root.pem")
    };
    
    optionsBuilder.UseMySql(connectionStringBuilder.ConnectionString);
}

Just put your cert in a Certificates directory with the same name as above, and set build option to "Copy to output always". When you use AWS Tools for Visual Studio, the Publish to AWS lambda option will include that file automatically.

The issue

When I run the lambda function and try to connect to Aurora/MySQL RDS instance using SSL, I get an IndexOutOfRangeException. The error is thrown inside of MySqlConnector library, even though I am using Pomelo EntityFramework.

Exception message: Index was outside the bounds of the array.: IndexOutOfRangeException
Stack trace:
at MySqlConnector.Core.ServerSession.<>c__DisplayClass71_0.<InitSslAsync>g__ValidateRemoteCertificate1(Object rcbSender, X509Certificate rcbCertificate, X509Chain rcbChain, SslPolicyErrors rcbPolicyErrors) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 742
at System.Net.Security.SslStream.UserCertValidationCallbackWrapper(String hostName, X509Certificate2 certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertValidationCallback remoteCertValidationCallback, ProtocolToken& alertToken)
at System.Net.Security.SslState.CompleteHandshake(ProtocolToken& alertToken)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
at MySqlConnector.Core.ServerSession.<InitSslAsync>d__71.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 806
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySqlConnector.Core.ServerSession.<ConnectAsync>d__56.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 243
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySql.Data.MySqlClient.MySqlConnection.<CreateSessionAsync>d__77.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 365
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySql.Data.MySqlClient.MySqlConnection.<OpenAsync>d__19.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 152
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySql.Data.MySqlClient.MySqlConnection.Open() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 137
at Microsoft.EntityFrameworkCore.Storage.Internal.MySqlConnectionSettings.<>c__DisplayClass2_0.<GetSettings>b__0(String key)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Lazy`1.CreateValue()
at Microsoft.EntityFrameworkCore.Storage.Internal.MySqlSmartTypeMapper.MaybeConvertMapping(RelationalTypeMapping mapping)
at Microsoft.EntityFrameworkCore.Storage.RelationalTypeMapper.IsTypeMapped(Type clrType)
at System.Linq.Enumerable.WhereArrayIterator`1.MoveNext()
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.PropertyDiscoveryConvention.Apply(InternalEntityTypeBuilder entityTypeBuilder)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ImmediateConventionScope.OnEntityTypeAdded(InternalEntityTypeBuilder entityTypeBuilder)
at Microsoft.EntityFrameworkCore.Metadata.Internal.Model.AddEntityType(EntityType entityType)
at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalModelBuilder.Entity(TypeIdentity type, ConfigurationSource configurationSource)
at Microsoft.EntityFrameworkCore.ModelBuilder.Entity(Type type)
at Microsoft.EntityFrameworkCore.Infrastructure.ModelCustomizer.FindSets(ModelBuilder modelBuilder, DbContext context)
at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.FindSets(ModelBuilder modelBuilder, DbContext context)
at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.Customize(ModelBuilder modelBuilder, DbContext context)
at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.CreateModel(DbContext context, IConventionSetBuilder conventionSetBuilder, IModelValidator validator)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel()
at Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model()
at lambda_method(Closure , ServiceProvider )
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider()
at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
at Microsoft.EntityFrameworkCore.DbContext.EntryWithoutDetectChanges[TEntity](TEntity entity)
at Microsoft.EntityFrameworkCore.DbContext.SetEntityState[TEntity](TEntity entity, EntityState entityState)

Further technical details

MySQL version: Aurora v1.16, MySQL engine 5.6.10a
Operating system: AWS Lambda + AWS RDS Aurora cluster (lambda is Linux, RDS probably also)
MySqlConnector version: 0.40.3
Pomelo.EntityFrameworkCore.MySql version: 2.0.1

@mguinness
Copy link
Contributor

The first step you can take is to create a simple console app like below to repro the issue. Use the latest version from nuget and provide the stacktrace.

using (var db = new MySqlConnection(connStr))
{
    db.Open();

    using (var cmd = new MySqlCommand("SELECT @@version", db))
    {
        cmd.ExecuteNonQuery();
    }
}

Also you could copy the CACertificateFile code and see if you can narrow down what is occurring.

@bgrainger
Copy link
Member

Given the exception message, the bug seems most likely to be on this line:

var chainStatus = caCertificateChain.ChainStatus[0].Status & ~X509ChainStatusFlags.UntrustedRoot;

However, I've not been able to reproduce locally (connecting to Aurora from netcoreapp2.0, not AWS Lambda).

@mguinness
Copy link
Contributor

@bgrainger I guess you could do some bounds checking using caCertificateChain.ChainStatus.Length.

@bgrainger
Copy link
Member

According to Using SSL to Encrypt a Connection to a DB Instance:

You might need to use an intermediate certificate to connect to your region.

It doesn't provide an indication of when this "might" be necessary, but I was unable to connect from .NET 4.7.1 until I had provided an intermediate certificate for my region and the root certificate. If so, that's going to require an API change to allow multiple intermediate and root certificates to be specified.

I guess you could do some bounds checking using caCertificateChain.ChainStatus.Length.

Agreed; but that will only prevent the crash; without ChainStatus we can't validate the remote certificate, no VerifyFull will fail.

@paya-cz Do you need SslMode = MySqlSslMode.VerifyFull? As a workaround, can you just use MySqlSslMode.Required?

@bgrainger
Copy link
Member

If so, that's going to require an API change to allow multiple intermediate and root certificates to be specified.

rds-combined-ca-bundle.pem contains all the certificates we need; it's simply multiple PEM files concatenated together. The problem is that loading a certificate from this file only loads the first one; we would need to loop over the bundled certificates and load them all: #499

I'm not sure if that'll fix the IndexOutOfRangeException, but it's a necessary precondition to connecting to Aurora.

bgrainger added a commit that referenced this issue May 19, 2018
Suppress the exception reported in #498.
@bgrainger
Copy link
Member

@paya-cz Please update to MySqlConnector 0.40.4 and set CACertificateFile=rds-combined-ca-bundle.pem.

This works for me under netcoreapp2.0; I'm interested to hear if it solves your problems on AWS Lambda.

@paya-cz
Copy link
Author

paya-cz commented May 20, 2018

@bgrainger Hi. Thank you for such a quick update. My current setup is using MySqlConnector 0.40.4, CACertificateFile=rds-combined-ca-bundle.pem and MySqlConnectionStringBuilder.SslMode = MySqlSslMode.VerifyCA. This setup works, and I can connect to Aurora via SSL and AWS IAM auth, and data is getting through to the server safe and sound.

If I use MySqlConnectionStringBuilder.SslMode = MySqlSslMode.VerifyFull, I cannot connect due to SSL issue. My Aurora is running in a cluster, and as per documentation I tried connecting against both the cluster endpoint and primary instance endpoint. Both cases fail.

I tried running the lambda with console logger (see below) but this didn't yield any useful info:
MySqlConnectorLogManager.Provider = new ConsoleLoggerProvider(MySqlConnectorLogLevel.Debug);

So, I tested MySQL Workbench on my local machine and tried to connect to the two endpoints (cluster and primary), via SSL. Connecting to primary endpoint works just fine, including full identity verification. Connecting to the cluster endpoint does not work, probably because MySQL Workbench does not support alternate subject names. Furthermore, I have used Wireshark to capture the traffic and inspect the certificate being returned by the Aurora in the MySql connection. The certificate is fine, with the subject name being the primary instance endpoint, and cluster endpoint being listed in the alternate subject names.

Thus, there is a bug somewhere in MySqlConnector such that VerifyFull fails, despite the certificate having the proper domain name in the subject name when connecting to primary instance. Furthermore, MySqlConnector does not seem to support subject alternate names.

When running against primary instance with VerifyFull:

MySql.Data.MySqlClient.MySqlException (0x80004005): SSL Authentication Error ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
at MySqlConnector.Core.ServerSession.<InitSslAsync>d__71.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 959
at MySqlConnector.Core.ServerSession.<InitSslAsync>d__71.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 976
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySqlConnector.Core.ServerSession.<ConnectAsync>d__56.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 300
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySql.Data.MySqlClient.MySqlConnection.<CreateSessionAsync>d__87.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 410
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
at MySql.Data.MySqlClient.MySqlConnection.<OpenAsync>d__25.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 178
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySql.Data.MySqlClient.MySqlConnection.Open() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 156
at Microsoft.EntityFrameworkCore.Storage.Internal.MySqlConnectionSettings.<>c__DisplayClass2_0.<GetSettings>b__0(String key)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at Microsoft.EntityFrameworkCore.Storage.Internal.MySqlSmartTypeMapper.MaybeConvertMapping(RelationalTypeMapping mapping)
at Microsoft.EntityFrameworkCore.Storage.RelationalTypeMapper.IsTypeMapped(Type clrType)
at System.Reflection.PropertyInfoExtensions.FindCandidateNavigationPropertyType(PropertyInfo propertyInfo, Func`2 isPrimitiveProperty)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.NavigationAttributeEntityTypeConvention`1.Apply(InternalEntityTypeBuilder entityTypeBuilder, String ignoredMemberName)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ImmediateConventionScope.OnEntityTypeMemberIgnored(InternalEntityTypeBuilder entityTypeBuilder, String ignoredMemberName)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.RunVisitor.VisitOnEntityTypeMemberIgnored(OnEntityTypeMemberIgnoredNode node)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionVisitor.VisitConventionScope(ConventionScope node)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionBatch.Run()
at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder.Ignore(String name, ConfigurationSource configurationSource)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.NotMappedMemberAttributeConvention.Apply(InternalEntityTypeBuilder entityTypeBuilder)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ImmediateConventionScope.OnEntityTypeAdded(InternalEntityTypeBuilder entityTypeBuilder)
at Microsoft.EntityFrameworkCore.Metadata.Internal.Model.AddEntityType(EntityType entityType)
at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalModelBuilder.Entity(TypeIdentity type, ConfigurationSource configurationSource)
at Microsoft.EntityFrameworkCore.ModelBuilder.Entity(Type type)
at Microsoft.EntityFrameworkCore.Infrastructure.ModelCustomizer.FindSets(ModelBuilder modelBuilder, DbContext context)
at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.FindSets(ModelBuilder modelBuilder, DbContext context)
at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.Customize(ModelBuilder modelBuilder, DbContext context)
at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.CreateModel(DbContext context, IConventionSetBuilder conventionSetBuilder, IModelValidator validator)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel()
at Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model()
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProvider provider)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider()
at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
at Microsoft.EntityFrameworkCore.DbContext.EntryWithoutDetectChanges[TEntity](TEntity entity)
at Microsoft.EntityFrameworkCore.DbContext.SetEntityState[TEntity](TEntity entity, EntityState entityState)

When running against cluster endpoint with VerifyFull:

MySql.Data.MySqlClient.MySqlException (0x80004005): SSL Authentication Error ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
at MySqlConnector.Core.ServerSession.<InitSslAsync>d__71.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 959
at MySqlConnector.Core.ServerSession.<InitSslAsync>d__71.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 976
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySqlConnector.Core.ServerSession.<ConnectAsync>d__56.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 300
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySql.Data.MySqlClient.MySqlConnection.<CreateSessionAsync>d__87.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 410
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
at MySql.Data.MySqlClient.MySqlConnection.<OpenAsync>d__25.MoveNext() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 178
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MySql.Data.MySqlClient.MySqlConnection.Open() in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlConnection.cs:line 156
at Microsoft.EntityFrameworkCore.Storage.Internal.MySqlConnectionSettings.<>c__DisplayClass2_0.<GetSettings>b__0(String key)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at Microsoft.EntityFrameworkCore.Storage.Internal.MySqlSmartTypeMapper.MaybeConvertMapping(RelationalTypeMapping mapping)
at Microsoft.EntityFrameworkCore.Storage.RelationalTypeMapper.IsTypeMapped(Type clrType)
at System.Reflection.PropertyInfoExtensions.FindCandidateNavigationPropertyType(PropertyInfo propertyInfo, Func`2 isPrimitiveProperty)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.NavigationAttributeEntityTypeConvention`1.Apply(InternalEntityTypeBuilder entityTypeBuilder, String ignoredMemberName)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ImmediateConventionScope.OnEntityTypeMemberIgnored(InternalEntityTypeBuilder entityTypeBuilder, String ignoredMemberName)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.RunVisitor.VisitOnEntityTypeMemberIgnored(OnEntityTypeMemberIgnoredNode node)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionVisitor.VisitConventionScope(ConventionScope node)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionBatch.Run()
at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder.Ignore(String name, ConfigurationSource configurationSource)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.NotMappedMemberAttributeConvention.Apply(InternalEntityTypeBuilder entityTypeBuilder)
at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ImmediateConventionScope.OnEntityTypeAdded(InternalEntityTypeBuilder entityTypeBuilder)
at Microsoft.EntityFrameworkCore.Metadata.Internal.Model.AddEntityType(EntityType entityType)
at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalModelBuilder.Entity(TypeIdentity type, ConfigurationSource configurationSource)
at Microsoft.EntityFrameworkCore.ModelBuilder.Entity(Type type)
at Microsoft.EntityFrameworkCore.Infrastructure.ModelCustomizer.FindSets(ModelBuilder modelBuilder, DbContext context)
at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.FindSets(ModelBuilder modelBuilder, DbContext context)
at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.Customize(ModelBuilder modelBuilder, DbContext context)
at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.CreateModel(DbContext context, IConventionSetBuilder conventionSetBuilder, IModelValidator validator)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel()
at Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model()
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProvider provider)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider()
at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
at Microsoft.EntityFrameworkCore.DbContext.EntryWithoutDetectChanges[TEntity](TEntity entity)
at Microsoft.EntityFrameworkCore.DbContext.SetEntityState[TEntity](TEntity entity, EntityState entityState)

@paya-cz
Copy link
Author

paya-cz commented May 20, 2018

Btw AWS Lambda is running the .NET Core app on Linux. Maybe there is some discrepancy in the X509 certificate implementation between these systems? Take a look here:
https://github.com/dotnet/corefx/issues/21622

So you should be able to reproduce the issue on Linux.

@bgrainger
Copy link
Member

I ran my test app (using CACertificateFile = rds-combined-ca-bundle.pem) on Ubuntu 14.04 and had no problems with SslMode = VerifyFull against both an instance and a cluster endpoint.

OTOH, I can't even get SslMode = Required to work on macOS; it throws a CryptographicException before it even tries to validate the remote certificate.

So unfortunately this is looking like it might be very OS-specific, and may even be an issue in .NET Core (or the platform-specific underlying crypto libraries it uses).

The only difference between VerifyCA and VerifyFull is this line:

rcbPolicyErrors &= ~SslPolicyErrors.RemoteCertificateNameMismatch;

It appears that the RemoteCertificateNameMismatch bit is set for you, but I can't reproduce that on Windows or Ubuntu 14.04.

@paya-cz
Copy link
Author

paya-cz commented May 25, 2018

Amazon is using their own Linux distribution to run lambdas, as per https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html

There may be differences with Amazon Linux and Ubuntu. I have created a new issue here aws/aws-lambda-dotnet#282 as this may be an Amazon Linux issue.

@bgrainger
Copy link
Member

While investigating this, I found a macOS-specific bug in loading concatenated certificates (“PEM bundles”): dotnet/corefx#29910

@bgrainger bgrainger changed the title Unable to use SSL when connecting to AWS RDS Aurora Unable to use SslMode=VerifyFull when running on AWS Lambda Jun 5, 2018
@bgrainger
Copy link
Member

@paya-cz I know this is a very old issue. Are you still using MySqlConnector on AWS Lambda? Have you tried running with .NET Core 2.1 or later? Does this help resolve this issue?

@paya-cz
Copy link
Author

paya-cz commented Feb 9, 2020

@bgrainger Yes I am still using MySqlConnector but the project where its being used is already deprecated. The lambda was written in .NET Core 2.1 from the get-go, which is the latest .NET Core version supported by AWS lambda.

@bgrainger
Copy link
Member

Are you still having a problem with VerifyFull or can this issue be closed out?

@paya-cz
Copy link
Author

paya-cz commented Feb 9, 2020

I just used VerifyCA because VerifyFull didn't work at the time. I do not know whether the issue is still present or whether it was silently fixed over time, because the project is out of my hands now and I would have to go and setup the entire test environment again including all the code for it to find out. Given the lack of community interest in this issue both at this repo and at at aws-lambda-dotnet repo, I think you can close the issue as it looks like not many people are using the same tech stack.

@KarlKl
Copy link

KarlKl commented Jun 9, 2020

I have the exact same problem. :(
I have a minimal setup with just the sample code from the website.

  • VerifyCA always works
  • VerifyFull works when using the instance url
  • VerifyFull does not work when using the cluster url

According to the docs this is caused by not "using" SAN while checking the cert.

@bgrainger bgrainger reopened this Jun 9, 2020
@bgrainger
Copy link
Member

The SSL certificate includes the DB instance endpoint as the Common Name (CN) for the SSL certificate to guard against spoofing attacks. As a result, you can only use the DB cluster endpoint to connect to a DB cluster using SSL if your client supports Subject Alternative Names (SAN). Otherwise, you must use the instance endpoint of a writer instance.

It sounds like SslStream should support SANs. According to dotnet/runtime#26583 (comment) there was a problem with mixed-case SAN entries that has been fixed in 2.1 latest and 3.1 latest. @KarlKl which exact version of .NET Core are you using?

@KarlKl
Copy link

KarlKl commented Jun 10, 2020

Thanks for the fast response. 🙂

On the my Linux Docker(official MS image core/aspnet:3.1-bionic) we use .NET Core 3.1.4 and locally on my Windows machine I have 3.1.5.
I have also tried it with the current .NET Core 5-preview.4 build without success.

The MySQL/Aurora version is: 5.7.mysql_aurora.2.07.1 and I have copied the MySQL server url directly from the AWS RDS console, it looks like this:
abc-cluster.cluster-xxxxxxxxx.eu-west-1.rds.amazonaws.com

When I connect to the instance(abc-instance.xxxxxxxxx.eu-west-1.rds.amazonaws.com) directly, SSL connection works as intended.

This is my project that I test with: https://github.com/KarlKl/MysqlSslAurora

@bgrainger
Copy link
Member

I'm not able to reproduce this problem. I created a new "Aurora (MySQL 5.7) 2.07.1" cluster and connected to it with SslMode=VerifyFull using the hostname testdb.cluster-XYZ.us-west-2.rds.amazonaws.com.

I added some debugging to ValidateRemoteCertificate:

bool ValidateRemoteCertificate(object rcbSender, X509Certificate rcbCertificate, X509Chain rcbChain, SslPolicyErrors rcbPolicyErrors)

(I don't know a way to dump a MySQL Server's SSL certificate; this answer suggests using tcpdump: https://serverfault.com/a/904072/85501.)

Inspecting the server's certificate in the debugger showed that it had the following subject: C=US, S=Washington, L=Seattle, O=Amazon.com, OU=RDS, CN=testdb-instance-1.XYZ.us-west-2.rds.amazonaws.com

And the following SANs:

  • testdb.cluster-XYZ.us-west-2.rds.amazonaws.com
  • testdb.cluster-ro-XYZ.us-west-2.rds.amazonaws.com
  • testdb-instance-1.XYZ.us-west-2.rds.amazonaws.com

It seems that in my case, the cluster presents a certificate that has the cluster's name as a SAN in the certificate, and this is accepted by SslStream when performing SSL negotiation.

The next step for you might be trying to dump your Aurora Server's SSL cert (with Wireshark or by building MySqlConnector from source?), and see what SANs it has. If it's missing SANs, you might need to follow up with AWS support. If it has them but SSL negotiation fails, it could be a problem in MySqlConnector, or possibly in .NET Core.

@bgrainger bgrainger added the waiting for answer Needs more information from the bug reporter label Jun 14, 2020
@KarlKl
Copy link

KarlKl commented Jun 15, 2020

Thank you for your investigation and your tip on checking the certificate. This finally led me to my problem.
There was of course absolutely no issue with MySqlConnector, the problem was on our end.
It appears that the name that is displayed in the RDS area does not match that in the certificate.
Name that is displayed in the AWS RDS Console:
abc-cluster.cluster...
Name in Cert:
abc-1-cluster.cluster...

I could think of two possible ways this was caused by:

  • The cluster was created under a different name and then renamed (if that is possible at all)
  • The cluster was created, deleted, and then re-created with the same name (and AWS added -1 to the name)

I need to do a little more research into why the problem occurred (the colleague who created the cluster is on vacation this week).

Edit: However, the connection to "abc-1-cluster" does not work. Thanks anyway for the great tip to check the certificate. Very smart. 🧠 💡 😉

Edit#2: The problem was a rename of the cluster without restarting the instances. Without a restart the DNS name of the cluster changes, but the cert that is used by the instances does not.

@bgrainger bgrainger removed the waiting for answer Needs more information from the bug reporter label Jun 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

4 participants