-
Notifications
You must be signed in to change notification settings - Fork 335
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to use SslMode=VerifyFull when running on AWS Lambda #498
Comments
The first step you can take is to create a simple console app like below to repro the issue. Use the latest version from nuget and provide the stacktrace. using (var db = new MySqlConnection(connStr))
{
db.Open();
using (var cmd = new MySqlCommand("SELECT @@version", db))
{
cmd.ExecuteNonQuery();
}
} Also you could copy the CACertificateFile code and see if you can narrow down what is occurring. |
Given the exception message, the bug seems most likely to be on this line:
However, I've not been able to reproduce locally (connecting to Aurora from |
@bgrainger I guess you could do some bounds checking using |
According to Using SSL to Encrypt a Connection to a DB Instance:
It doesn't provide an indication of when this "might" be necessary, but I was unable to connect from .NET 4.7.1 until I had provided an intermediate certificate for my region and the root certificate. If so, that's going to require an API change to allow multiple intermediate and root certificates to be specified.
Agreed; but that will only prevent the crash; without @paya-cz Do you need |
I'm not sure if that'll fix the IndexOutOfRangeException, but it's a necessary precondition to connecting to Aurora. |
Suppress the exception reported in #498.
@paya-cz Please update to MySqlConnector 0.40.4 and set This works for me under |
@bgrainger Hi. Thank you for such a quick update. My current setup is using MySqlConnector 0.40.4, If I use I tried running the lambda with console logger (see below) but this didn't yield any useful info: So, I tested MySQL Workbench on my local machine and tried to connect to the two endpoints (cluster and primary), via SSL. Connecting to primary endpoint works just fine, including full identity verification. Connecting to the cluster endpoint does not work, probably because MySQL Workbench does not support alternate subject names. Furthermore, I have used Wireshark to capture the traffic and inspect the certificate being returned by the Aurora in the MySql connection. The certificate is fine, with the subject name being the primary instance endpoint, and cluster endpoint being listed in the alternate subject names. Thus, there is a bug somewhere in MySqlConnector such that VerifyFull fails, despite the certificate having the proper domain name in the subject name when connecting to primary instance. Furthermore, MySqlConnector does not seem to support subject alternate names. When running against primary instance with
When running against cluster endpoint with
|
Btw AWS Lambda is running the .NET Core app on Linux. Maybe there is some discrepancy in the X509 certificate implementation between these systems? Take a look here: So you should be able to reproduce the issue on Linux. |
I ran my test app (using OTOH, I can't even get So unfortunately this is looking like it might be very OS-specific, and may even be an issue in .NET Core (or the platform-specific underlying crypto libraries it uses). The only difference between
It appears that the |
Amazon is using their own Linux distribution to run lambdas, as per https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html There may be differences with Amazon Linux and Ubuntu. I have created a new issue here aws/aws-lambda-dotnet#282 as this may be an Amazon Linux issue. |
While investigating this, I found a macOS-specific bug in loading concatenated certificates (“PEM bundles”): dotnet/corefx#29910 |
@paya-cz I know this is a very old issue. Are you still using MySqlConnector on AWS Lambda? Have you tried running with .NET Core 2.1 or later? Does this help resolve this issue? |
@bgrainger Yes I am still using MySqlConnector but the project where its being used is already deprecated. The lambda was written in .NET Core 2.1 from the get-go, which is the latest .NET Core version supported by AWS lambda. |
Are you still having a problem with |
I just used |
I have the exact same problem. :(
According to the docs this is caused by not "using" SAN while checking the cert. |
It sounds like |
Thanks for the fast response. 🙂 On the my Linux Docker(official MS image The MySQL/Aurora version is: When I connect to the instance( This is my project that I test with: https://github.com/KarlKl/MysqlSslAurora |
I'm not able to reproduce this problem. I created a new "Aurora (MySQL 5.7) 2.07.1" cluster and connected to it with I added some debugging to
(I don't know a way to dump a MySQL Server's SSL certificate; this answer suggests using tcpdump: https://serverfault.com/a/904072/85501.) Inspecting the server's certificate in the debugger showed that it had the following subject: And the following SANs:
It seems that in my case, the cluster presents a certificate that has the cluster's name as a SAN in the certificate, and this is accepted by The next step for you might be trying to dump your Aurora Server's SSL cert (with Wireshark or by building MySqlConnector from source?), and see what SANs it has. If it's missing SANs, you might need to follow up with AWS support. If it has them but SSL negotiation fails, it could be a problem in MySqlConnector, or possibly in .NET Core. |
Thank you for your investigation and your tip on checking the certificate. This finally led me to my problem. I could think of two possible ways this was caused by:
I need to do a little more research into why the problem occurred (the colleague who created the cluster is on vacation this week). Edit: However, the connection to "abc-1-cluster" does not work. Thanks anyway for the great tip to check the certificate. Very smart. 🧠 💡 😉 Edit#2: The problem was a rename of the cluster without restarting the instances. Without a restart the DNS name of the cluster changes, but the cert that is used by the instances does not. |
Steps to reproduce
from https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/AuroraMySQL.Security.html)
from https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html
I suspect messing with VPC and IAM auth is not necessary, as this looks to be a SSL issue. It's just a setup that I was using when I discovered the issue.
For reference, this is the OnConfiguring method of DbContext:
Just put your cert in a
Certificates
directory with the same name as above, and set build option to "Copy to output always". When you use AWS Tools for Visual Studio, the Publish to AWS lambda option will include that file automatically.The issue
When I run the lambda function and try to connect to Aurora/MySQL RDS instance using SSL, I get an IndexOutOfRangeException. The error is thrown inside of MySqlConnector library, even though I am using Pomelo EntityFramework.
Further technical details
MySQL version: Aurora v1.16, MySQL engine 5.6.10a
Operating system: AWS Lambda + AWS RDS Aurora cluster (lambda is Linux, RDS probably also)
MySqlConnector version: 0.40.3
Pomelo.EntityFrameworkCore.MySql version: 2.0.1
The text was updated successfully, but these errors were encountered: