IAM Database Authentication results in Authentication fallback / failure

[iamcarbon]

When attempting to authenticate to MySQL using IAM authentication, we seem to be attempting to switch protocols and falling back to "mysql_clear_password".

This may have have something to do with AWS reusing the user & password properties. Will do some more digging -- but posting here for now to track the issue.

DB USER

CREATE USER testing IDENTIFIED WITH AWSAuthenticationPlugin as 'RDS';        

CONNECTION STRING

var sb = new StringBuilder();

var token = rds.GenerateAuthenticationToken(
    new GetAuthenticationTokenRequest(endpoint.Host, endpoint.Port, username)
);

sb.Append($"Server={endpoint.Host};");
sb.Append($"Database={schema};");
sb.Append("SslMode=Required;");
sb.Append($"user={username};");
sb.Append($"password={token};");

return sb.ToString();

ERROR WHEN CONNECTING

System.NotSupportedException : Authentication method 'mysql_clear_password' is not supported.

MySql.Data.Serialization.MySqlSession.<SwitchAuthenticationAsync>d__50.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySql.Data.Serialization.MySqlSession.<ConnectAsync>d__48.MoveNext()

[bgrainger]

According to Client-Side Cleartext Pluggable Authentication:

A client-side authentication plugin is available that sends the password to the server without hashing or encryption. This plugin is built into the MySQL client library.

There is currently no support for mysql_clear_password in this library, but it should be very easy to add.

The Amazon documentation says:

With IAM database authentication, you must use a Secure Sockets Layer (SSL) connection, so all data transmitted from and to your DB instance or DB cluster is encrypted.

Thus it seems that it wouldn't be insecure to send the password in cleartext because it will be protected by a TLS connection.

[bgrainger]

Just curious, have you tried using IAM Authentication with Oracle's Connector/NET (MySql.Data)? I wasn't able to find support for mysql_clear_password in that library (after a quick look) so I'm wondering if it's also a missing feature in that library.

[iamcarbon]

@bgrainger -- support for sending the password in the clear over SSL should enable this use case. I'll test things out using MySqlData sometime tomorrow and let you know. also, thanks for the quick response!!!!

[bgrainger]

We'll add support for mysql_clear_password in the authentication method switch request packet. For safety, the connector will require a secure connection in order to send the password (and throw otherwise).

This doesn't protect against a malicious proxy that attempts to steal clients' credentials; to defend against that we could require explicit opt-in, e.g., through a new connection string setting or by supporting the LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN environment variable used by mysql. However, note that any malicious proxy that can intercept client traffic can already establish a connection to the server and perform any data access operations the client is permitted to execute, so getting the client's password seems like a low-severity problem. To defend against this, use TLS and verify the server's certificate.

[bgrainger]

Implemented in 0.20.0.

[iamcarbon]

@bgrainger I just updated to v0.20 and can confirm that the fix works! Thanks again!

[bgrainger]

@iamcarbon I just noticed that the IAM documentation page says:

An authentication token is a value created by the AWS Signature Version 4 algorithm, which has a 15-minute lifetime. If an expired token is used to connect to the database, the connection request is denied.

This makes me think that connection pools will stop working after about 15 minutes. (Existing connections in the pool might still work, but opening a new connection will just fail with no way to recover.)

I think a workaround could be to set ConnectionIdleTimeout=0 in your connection string so idle connections don't get removed from the pool. When #85 is implemented, you could set MinimumPoolSize to the same value as MaximumPoolSize so that all connections are opened immediately (when the auth token is still valid).

Please let me know what you find.

[iamcarbon]

We're dynamically updating the password every 10 minutes: which should in theory, create a new pool -- with the existing connections getting removed after going idle for some period of time (i.e. 5 minutes) Going to clone the project and dive into the guts to see what's actually happening.

if (Expires <= DateTime.UtcNow)
{
   onExpiration(this); // requests a new token & updates the password

   currentConnectionString = builder.ToString();
}

return currentConnectionString;

[caleblloyd]

This makes me think that connection pools will stop working after about 15 minutes. (Existing connections in the pool might still work, but opening a new connection will just fail with no way to recover.)

I believe the only thing that may not work anymore is closing/reopening an existing connection to a database version that does not support the connection reset payload if connection.Close/CloseAsync followed by connection.Open/OpenAsync is called after the IAM password has timed out. In this case MySqlConnector will attempt to login with the password again after it has expired.

All calls to new MySqlConnection must include the password in the connection string, so if they send an expired password, it is the caller's fault, not the connector's fault. The caller should be responsible for updating the password as it expires.

We're dynamically updating the password every 10 minutes

In theory you should be in good shape then 😄 I believe that any recent version of Amazon RDS should support the connection reset payload, in which case you would never hit the above issue.