Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection Pool Reaper #218

Merged
merged 3 commits into from Apr 8, 2017
Merged

Connection Pool Reaper #218

merged 3 commits into from Apr 8, 2017

Conversation

caleblloyd
Copy link
Contributor

@caleblloyd caleblloyd commented Mar 31, 2017

  • Opening this as an initial design proposal for the Connection Pool Reaper Connection Pool Reaper Job #217
  • We probably need a configurable time for the reaper via Connection String Option. This would allow us to set a low time for testing.
  • This would need to be adjusted to support Min Pool Size once we support that

Please provide feedback and I can update this PR to something more formal over the next few days. Let me know if there's a "more accepted/efficient" way to create long-running jobs than just spinning up an async task. Also open for ideas on how to better monitor this job to ensure that it doesn't hang.

@@ -138,6 +147,14 @@ public static ConnectionPool GetPool(ConnectionSettings cs)
await pool.ClearAsync(ioBehavior, cancellationToken).ConfigureAwait(false);
}

public static async Task ReapPoolsAsync(IOBehavior ioBehavior, CancellationToken cancellationToken)
{
var pools = new List<ConnectionPool>(s_pools.Values);
Copy link
Member

@bgrainger bgrainger Apr 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that the Values collection is thread-safe and can be enumerated on one thread while the dictionary is being modified on a different thread. (It will represent a snapshot of the dictionary at the time the property was accessed.)

Thus, the local copy of s_pools.Values is unnecessary and this could be inlined. (Same for the existing ClearPoolsAsync.)

@@ -147,6 +164,21 @@ private ConnectionPool(ConnectionSettings cs)
}

static readonly ConcurrentDictionary<string, ConnectionPool> s_pools = new ConcurrentDictionary<string, ConnectionPool>();
static readonly TimeSpan ReapTimeSpan = TimeSpan.FromMinutes(3);
static readonly Task Reaper = Task.Run(async () => {
Copy link
Member

@bgrainger bgrainger Apr 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In times past, I would have created a dedicated background worker thread that used Thread.Sleep. I haven't personally used this particular approach, but I don't see anything wrong with it.

I was wondering if we needed to use a CancellationToken... but there's no mechanism by which it could ever be cancelled; the connection pools live for the lifetime of the process.

(I can imagine an edge case where a consumer finishes using MySQL, calls MySqlConnection.ClearAllPools and would like all background code to stop running, but that seems too unlikely to be worth writing code to support.)

@bgrainger
Copy link
Member

bgrainger commented Apr 1, 2017

I'm on the fence about whether to merge this in without implementing MinimumPoolSize; it could effectively undo the benefit of connection pooling (although I suppose that would only affect processes that aren't executing a query at least once every three minutes).

@caleblloyd
Copy link
Contributor Author

caleblloyd commented Apr 7, 2017

I've updated the reaper to include a connection string option, ConnectionWaitTimeout, which defaults to 180 seconds. It also now respects MinPoolSize. There's still a couple design aspects I'm working out right now:

  • We use a ConcurrentQueue to store unused pooled sessions presently. This is a FIFO Queue, so it will use connections evenly, which is sub-optimal for reaping. If heavy use is encountered, the connection pool will grow to MaxPoolSize. Then once usage backs off, as long as MaxPoolSize connections are used within a ConnectionWaitTimeout period, each connection will get used once, and nothing will ever get reaped.
    • Switching to a ConcurrentStack (LIFO) would solve this issue, except the reaper needs to read from the back of the queue, which is not possible with a ConcurrentStack. Our ideal data type would be a ConcurrentDoubleEndedQueue, but that does not exist.
    • Right now I'm considering using a List with locking, is there a better option?
  • The reaper is hardcoded to run once per minute. How do we write a test for this without waiting a full minute?

</tr>
<tr>
<td>Connection Reset, ConnectionReset </td>
<td>false</td>
<td>If true, the connection state is reset when it is retrieved from the pool. The default value of false avoids making an additional server round trip when obtaining a connection, but the connection state is not reset.</td>
</tr>
<tr>
<td>Connection Wait Timeout, ConnectionWaitTimeout</td>
Copy link
Member

@bgrainger bgrainger Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would Idle be a better name than Wait?

Oh, I guess this matches the server's wait_timeout variable.

Copy link
Contributor Author

@caleblloyd caleblloyd Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did name it after the server's wait_timeout, but ConnectionIdleTimeout is more self-explanitory, I like that better. I'll update it to ConnectionIdleTimeout.

</tr>
<tr>
<td>Connection Reset, ConnectionReset </td>
<td>false</td>
<td>If true, the connection state is reset when it is retrieved from the pool. The default value of false avoids making an additional server round trip when obtaining a connection, but the connection state is not reset.</td>
</tr>
<tr>
<td>Connection Wait Timeout, ConnectionWaitTimeout</td>
<td>180</td>
<td>The amount of time in seconds that a connection can remain unused in the pool. Any connection that is unused for longer is subject to being closed by a background task runs every minute, unless there are only MinimumPoolSize connections left in the pool. A value of zero (0) means pooled connections will never incur a ConnectionWaitTimeout.</td>
Copy link
Member

@bgrainger bgrainger Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

task that runs

@bgrainger
Copy link
Member

bgrainger commented Apr 7, 2017

Right now I'm considering using a List with locking, is there a better option?

I don't have any objection to this. (I've heard the advice in the past--don't remember where I read it--to start with lock because it's the simplest to get right, then move on to more esoteric concurrency mechanisms if you determine that it's a bottleneck in the application.)

It seems like if you set the size of your List (or array) to MaximumPoolSize then you could use a simple ring buffer to store active connections. Inserting/removing would be writing/reading an array entry and updating an integer, which would be a small amount of code to have in the lock. Hopefully this would result in little contention in typical use cases.

The reaping process might hold the lock for a long time, but that would happen infrequently.

@caleblloyd
Copy link
Contributor Author

caleblloyd commented Apr 7, 2017

Trying it with lock and LinkedList, which is O(1) for .First and .Last (the two we need right now)

@caleblloyd
Copy link
Contributor Author

caleblloyd commented Apr 7, 2017

This implementation is working nicely; I run 1000 connections in 1s in the performance test to load the pool up. Then I run a steady 100rps after that and only a few connections get touched:

mysql> SHOW FULL PROCESSLIST;
+------+-----------+------------------+-----------+---------+------+----------+-----------------------+
| Id   | User      | Host             | db        | Command | Time | State    | Info                  |
+------+-----------+------------------+-----------+---------+------+----------+-----------------------+
|   58 | root      | 172.17.0.1:54384 | NULL      | Query   |    0 | starting | SHOW FULL PROCESSLIST |
| 1041 | mysqltest | 172.17.0.1:33282 | mysqltest | Sleep   |    0 |          | NULL                  |
| 1045 | mysqltest | 172.17.0.1:33290 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1048 | mysqltest | 172.17.0.1:33296 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1073 | mysqltest | 172.17.0.1:33346 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1132 | mysqltest | 172.17.0.1:33464 | mysqltest | Sleep   |    2 |          | NULL                  |
| 1137 | mysqltest | 172.17.0.1:33540 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1138 | mysqltest | 172.17.0.1:33542 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1139 | mysqltest | 172.17.0.1:33544 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1140 | mysqltest | 172.17.0.1:33546 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1141 | mysqltest | 172.17.0.1:33548 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1142 | mysqltest | 172.17.0.1:33550 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1143 | mysqltest | 172.17.0.1:33552 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1144 | mysqltest | 172.17.0.1:33554 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1145 | mysqltest | 172.17.0.1:33556 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1146 | mysqltest | 172.17.0.1:33558 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1147 | mysqltest | 172.17.0.1:33560 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1148 | mysqltest | 172.17.0.1:33562 | mysqltest | Sleep   |    0 |          | NULL                  |
| 1149 | mysqltest | 172.17.0.1:33564 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1150 | mysqltest | 172.17.0.1:33566 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1151 | mysqltest | 172.17.0.1:33568 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1152 | mysqltest | 172.17.0.1:33570 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1153 | mysqltest | 172.17.0.1:33572 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1154 | mysqltest | 172.17.0.1:33574 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1155 | mysqltest | 172.17.0.1:33576 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1156 | mysqltest | 172.17.0.1:33578 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1157 | mysqltest | 172.17.0.1:33580 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1158 | mysqltest | 172.17.0.1:33582 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1159 | mysqltest | 172.17.0.1:33586 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1160 | mysqltest | 172.17.0.1:33584 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1161 | mysqltest | 172.17.0.1:33588 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1162 | mysqltest | 172.17.0.1:33590 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1163 | mysqltest | 172.17.0.1:33592 | mysqltest | Sleep   |   24 |          | NULL                  |
| 1164 | mysqltest | 172.17.0.1:33594 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1165 | mysqltest | 172.17.0.1:33596 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1166 | mysqltest | 172.17.0.1:33598 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1167 | mysqltest | 172.17.0.1:33600 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1168 | mysqltest | 172.17.0.1:33602 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1169 | mysqltest | 172.17.0.1:33604 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1170 | mysqltest | 172.17.0.1:33606 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1171 | mysqltest | 172.17.0.1:33608 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1172 | mysqltest | 172.17.0.1:33610 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1173 | mysqltest | 172.17.0.1:33612 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1174 | mysqltest | 172.17.0.1:33614 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1175 | mysqltest | 172.17.0.1:33618 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1176 | mysqltest | 172.17.0.1:33616 | mysqltest | Sleep   |   24 |          | NULL                  |
| 1177 | mysqltest | 172.17.0.1:33620 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1178 | mysqltest | 172.17.0.1:33624 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1179 | mysqltest | 172.17.0.1:33622 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1180 | mysqltest | 172.17.0.1:33626 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1181 | mysqltest | 172.17.0.1:33628 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1182 | mysqltest | 172.17.0.1:33630 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1183 | mysqltest | 172.17.0.1:33632 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1184 | mysqltest | 172.17.0.1:33634 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1185 | mysqltest | 172.17.0.1:33636 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1186 | mysqltest | 172.17.0.1:33638 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1187 | mysqltest | 172.17.0.1:33640 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1188 | mysqltest | 172.17.0.1:33642 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1189 | mysqltest | 172.17.0.1:33644 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1190 | mysqltest | 172.17.0.1:33646 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1191 | mysqltest | 172.17.0.1:33648 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1192 | mysqltest | 172.17.0.1:33650 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1193 | mysqltest | 172.17.0.1:33652 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1194 | mysqltest | 172.17.0.1:33654 | mysqltest | Sleep   |    1 |          | NULL                  |
| 1195 | mysqltest | 172.17.0.1:33656 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1196 | mysqltest | 172.17.0.1:33658 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1197 | mysqltest | 172.17.0.1:33660 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1198 | mysqltest | 172.17.0.1:33662 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1199 | mysqltest | 172.17.0.1:33664 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1200 | mysqltest | 172.17.0.1:33666 | mysqltest | Sleep   |   24 |          | NULL                  |
| 1201 | mysqltest | 172.17.0.1:33668 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1202 | mysqltest | 172.17.0.1:33670 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1203 | mysqltest | 172.17.0.1:33672 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1204 | mysqltest | 172.17.0.1:33674 | mysqltest | Sleep   |   24 |          | NULL                  |
| 1205 | mysqltest | 172.17.0.1:33676 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1206 | mysqltest | 172.17.0.1:33678 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1207 | mysqltest | 172.17.0.1:33680 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1208 | mysqltest | 172.17.0.1:33682 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1209 | mysqltest | 172.17.0.1:33684 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1210 | mysqltest | 172.17.0.1:33690 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1211 | mysqltest | 172.17.0.1:33686 | mysqltest | Sleep   |   24 |          | NULL                  |
| 1212 | mysqltest | 172.17.0.1:33688 | mysqltest | Sleep   |   13 |          | NULL                  |
| 1213 | mysqltest | 172.17.0.1:33694 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1214 | mysqltest | 172.17.0.1:33692 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1215 | mysqltest | 172.17.0.1:33696 | mysqltest | Sleep   |    0 |          | NULL                  |
| 1216 | mysqltest | 172.17.0.1:33698 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1217 | mysqltest | 172.17.0.1:33700 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1218 | mysqltest | 172.17.0.1:33702 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1219 | mysqltest | 172.17.0.1:33704 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1220 | mysqltest | 172.17.0.1:33708 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1221 | mysqltest | 172.17.0.1:33711 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1222 | mysqltest | 172.17.0.1:33714 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1223 | mysqltest | 172.17.0.1:33712 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1224 | mysqltest | 172.17.0.1:33716 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1225 | mysqltest | 172.17.0.1:33720 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1226 | mysqltest | 172.17.0.1:33706 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1227 | mysqltest | 172.17.0.1:33724 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1228 | mysqltest | 172.17.0.1:33718 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1229 | mysqltest | 172.17.0.1:33722 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1230 | mysqltest | 172.17.0.1:33728 | mysqltest | Sleep   |   25 |          | NULL                  |
| 1231 | mysqltest | 172.17.0.1:33726 | mysqltest | Sleep   |   25 |          | NULL                  |
+------+-----------+------------------+-----------+---------+------+----------+-----------------------+

Eventually, everything that's unused gets reaped:

mysql> SHOW FULL PROCESSLIST;
+------+-----------+------------------+-----------+---------+------+----------+-----------------------+
| Id   | User      | Host             | db        | Command | Time | State    | Info                  |
+------+-----------+------------------+-----------+---------+------+----------+-----------------------+
|   58 | root      | 172.17.0.1:54384 | NULL      | Query   |    0 | starting | SHOW FULL PROCESSLIST |
| 1041 | mysqltest | 172.17.0.1:33282 | mysqltest | Sleep   |    0 |          | NULL                  |
| 1132 | mysqltest | 172.17.0.1:33464 | mysqltest | Sleep   |    0 |          | NULL                  |
| 1194 | mysqltest | 172.17.0.1:33654 | mysqltest | Sleep   |    2 |          | NULL                  |
| 1212 | mysqltest | 172.17.0.1:33688 | mysqltest | Sleep   |    0 |          | NULL                  |
| 1215 | mysqltest | 172.17.0.1:33696 | mysqltest | Sleep   |    0 |          | NULL                  |
+------+-----------+------------------+-----------+---------+------+----------+-----------------------+

Now to figure out how to write a unit test for this. The reaper is hardcoded to run once per minute, any suggestions on how to write a test for this without waiting a full minute?

@bgrainger
Copy link
Member

bgrainger commented Apr 7, 2017

The logic in the Reaper task itself is fairly straightforward, so it could be sufficient to just test the ReapAsync method directly. (Of course this doesn't catch regressions that might get introduced into the Reaper task.)

}

static readonly ConcurrentDictionary<string, ConnectionPool> s_pools = new ConcurrentDictionary<string, ConnectionPool>();
#if DEBUG
static readonly TimeSpan ReaperInterval = TimeSpan.FromSeconds(1);
Copy link
Contributor Author

@caleblloyd caleblloyd Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a conditional compilation flag for DEBUG to adjust timing


namespace SideBySide
{
public class DebugOnlyTests : IClassFixture<DatabaseFixture>
Copy link
Contributor Author

@caleblloyd caleblloyd Apr 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SideBySide.DebugOnlyTests class is only compiled when running with "Debug" configuration

@caleblloyd
Copy link
Contributor Author

caleblloyd commented Apr 7, 2017

  • Changed ConnectionWaitTimeout to ConnectionIdleTimeout
  • Added SideBySide.DebugOnlyTests class that only run when "Debug" configuration is used
    • Added Debug tests to Travis and AppVeyor

@bgrainger this is ready for a final review when you get a chance. Thanks!

@bgrainger bgrainger merged commit 6e3affa into mysql-net:master Apr 8, 2017
2 checks passed
@bgrainger
Copy link
Member

bgrainger commented Apr 12, 2017

Shipped in 0.17.0.

@caleblloyd caleblloyd deleted the f_reaper branch Aug 12, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants