Implement CommandTimeout

[caleblloyd]

1. Implement DefaultCommandTimeout in the connection string

Connection String Documentation

2. Implement CommandTimeout in MySqlCommand

MySqlCommand cmd = new MySqlCommand();
cmd.CommandTimeout = 60;

MySqlCommand Documentation:

Implementation Notes:

MySQL Connector/Net 6.2 introduced timeouts that are aligned with how Microsoft handles SqlCommand.CommandTimeout. This property is the cumulative timeout for all network reads and writes during command execution or processing of the results. A timeout can still occur in the MySqlReader.Read method after the first row is returned, and does not include user processing time, only IO operations. The 6.2 implementation uses the underlying stream timeout facility, so is more efficient in that it does not require the additional timer thread as was the case with the previous implementation.

It sounds like Oracle's implementation of CommandTimeout is setting Socket.SendTimeout and Socket.ReceiveTimeout. They start at the timeout value (e.g. 30 seconds) and subtract only time spent in I/O. This becomes the send or receive timeout for the next socket command.

[caleblloyd]

I have a WIP of this here: caleblloyd@d061332

Commands are properly timing out, but the next command that goes over the same connection hits issues with reading packets out of order. Should the default logic be to reset a connection that hits a command timeout?

[bgrainger]

This is potentially going to be complicated.

I would expect a CommandTimeout to behave the same as passing in a CancellationToken that cancels after a timeout. (EDIT: I meant for the internals of the library; externally, it will throw a different exception type.) As per the notes on #3 this needs to involve KILL QUERY to free up server resources (and make the connection usable again). In that situation, the library shouldn't cancel socket operations, but actually read the final response from the server and leave the connection in a good state (so another command can be executed, it can be returned to the pool, etc.).

[caleblloyd]

MariaDB/J opens another connection to run the KILL QUERY, code is here.

You're correct in #3, load balanced connections could come into play here. If it's round-robin DNS balanced connections, we could store the resolved IP Address of the server in the Connection. If it's Layer 4 load balancing, there's no way to guarantee you get the same server (some router with a consistent IP address is maintaining a NAT Table to a cluster of MySQL servers).

If the driver has to open another connection anyways, why not just dispose the timed-out socket first, and replace it with a fresh socket? As long as MySQL internally cleans up the transaction on a disposed socket, this approach should work. This approach could be implemented in MySqlSession while the connection is still "leased out" of the pool, so that the socket that is eventually returned is in a good state.

[bgrainger]

why not just dispose the timed-out socket first, and replace it with a fresh socket?

This would lose all session variables, temporary tables, etc. If the client is able to keep using the same MySqlConnection object, then they would expect it to have the same logical state. (I don't know if MySql.Data implements this; I assume it does.)

I'm thinking that the connection pool should have n dedicated KILL QUERY threads (one per IP address per comma-delimited hostname) even though this runs the risk of exceeding the maximum number of connections to the server if the user configures the maximum pool size to be exactly the same as the maximum number of server connections. (This seems a rare-enough scenario that it could be listed as a known issue.)

[caleblloyd]

MySql.Data does implement KILL QUERY as well, code is here. It looks like they do not attempt to connect to a specific IP.

I'm thinking that the connection pool should have n dedicated KILL QUERY threads (one per IP address per comma-delimited hostname)

I like the idea of saving the IP Address of the server that was connected to in the Session and trying to reconnect to this specific server. I think this should be a one-off connection outside of the pool when KILL QUERY is needed though. It's a lot of extra effort to try to track the connection after the one-off command, MySql.Data and MariaDB/J do not.

If it's Layer 4 load balancing, there's no way to guarantee you get the same server

Does the Server ID get sent from the server when opening a connection? If so we should save this in the Session also.

As a last resort, if we can't open a new connection to the server (server connection limit hit or server went down without a clean shutdown), or server IDs do not match (layer 4 load balancing) should we dispose the socket and switch it out with a new one? Would this require a different exception message?

[bgrainger]

I think this should be a one-off connection outside of the pool when KILL QUERY is needed though. It's a lot of extra effort to try to track the connection after the one-off command, MySql.Data and MariaDB/J do not.

Sounds good.

Does the Server ID get sent from the server when opening a connection?

Not sure what you mean by "Server ID". There is a server version and a (server-local) connection ID in the initial handshake packet. AFAIK there's no value that uniquely identifies a server.

should we dispose the socket and switch it out with a new one?

I think it would be best to throw a MySqlException that explains that the connection is no longer usable and leave the session in a disconnected state. It would have to be up to the user to decide how to recover: is opening a new MySqlConnection sufficient, or is this error condition unrecoverable (from the client's perspective) due to abandoned temporary tables, etc.

[caleblloyd]

Not sure what you mean by "Server ID"

The server_id global variable is used in replicated setups. It is required to be unique for replication to start, but looks like it's not sent in the initial handshake packet.

There is a "Replication" option that can be set in the connection string. If this was set, we could run a SELECT @@server_id query as part of the connection open process.

The worry is that without comparing server_id before killing a query from a separate connection on a replicated setup, we could actually be killing a valid query on a different server that has the same connection_id.

[bgrainger]

Some well-written bugs against Connector/NET that should be tested when this is implemented: 86009, 86010, 85978.