Limit maximum size of connection pool

[jsha]

Right now, connections stay in the pool indefinitely. If someone makes requests to a wide variety of hosts, that can quickly fill up the pool. Each entry in the pool uses an FD, and eventually a program will hit ulimit_nofile and stop working.

We should set a max size for the pool, and when a new connection needs to be added but the pool is full, remove one of the existing connections. A couple of options for how to remove:

• We could scan the pool for connections where the server has already closed them. Such connections are taking up resources needlessly. But if the connection pool is full in steady state, this could be a lot of scanning.
• We could use an LRU cache.
• We could remove an entry at random.

[algesten]

What is a reasonable pool size? I think the size to some extent could guide the strategy.

[jsha]

On Linux systems, 1024 is a common default for ulimit_nofile, so we want to be below that, with some headroom for other tasks the program needs to do that use up fds. Go defaults to a MaxIdleConns of 100, which seems to work fine; let's do 100.