Hyper not cleaning up idle connections in some cases in a long running application and running into the file descriptor limit

[glyphpoch]

Hopefully this is the right place to open this issue in, it was spotted while using reqwest but seems to be occuring due to how hyper cleans up idle connections.

Main dependencies

[[package]]
name = "tokio"
version = "1.1.1"

[[package]]
name = "reqwest"
version = "0.11.0"

[[package]]
name = "hyper"
version = "0.14.2"

System

Linux 4.14.86-gentoo Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz

Hyper idle connection cleanup

We're using reqwest at work in an application that was designed to just run forever. We also keep the same reqwest::Client around the entire time the application is running.

While doing some loads testing, all HTTP requests started failing with the following error (don't worry about the random host name, we're not actually doing anything crazy, it's from the reproduction app):

error sending request for url (http://ypni4syq24hfujpserga:8080/): error trying to connect: dns error: Too many open files (os error 24)

It seems like the following code is responsible for cleaning up idle connections - when there are too many in the pool or when they've expired due to configured timeouts:

hyper/src/client/pool.rs

Line 591 in 4d2125c

fn checkout(&mut self, cx: &mut task::Context<'_>) -> Option<Pooled<T>> {

However, IdlePopper::pop, which does the actual cleanup, is never called if we never send a request to the same domain again, leaving the connection open forever, or at least long enough that the process hits Linux's file descriptor limit which is 1024 by default.

Sample reproduction application

Sorry for the messy approach but using HOSTALIASES was the simplest/fastest way to get this working.

This application won't hit the file descriptor limit, it will just make 500 GET requests to random domains and then sleep for 2 hours, to make it easier to observe how long the connections are left open for.

https://gist.github.com/glyphpoch/3f92005ae8b7d5ea0e3722ab10414f14

Running the app:

# Start an HTTP server
python -m http.server 12030

# Run the app
export HOSTALIASES=${PATH_TO_TEST_HOSTS}/testhosts
cargo run --release

# Observe process's file descriptors
pgrep reqtest
lsof -a -p ${PID_FROM_PGREP}

Workarounds

• While not ideal, recreating the reqwest::Client often, solves the issue..
• Disabling connection pooling works as well, e.g.:

let client= reqwest::Client::builder()
    .pool_max_idle_per_host(0)
    .build();

[seanmonstar]

Hm, there should also be an interval task spawned for the pool, which checks for expired connections every pool_idle_timeout. Is there a case where it doesn't do that? I wonder why...

[glyphpoch]

If it's supposed to be spawned by PoolInner::spawn_idle_interval -

hyper/src/client/pool.rs

Line 402 in 4d2125c

fn spawn_idle_interval(&mut self, pool_ref: &Arc<Mutex<PoolInner<T>>>) {

- then it looks like that's gated behind the runtime feature which is not enabled by reqwest?

[seanmonstar]

Wow, that's fragile. So yea, reqwest should probably enable that feature...

[glyphpoch]

Alright, thanks. I'll close this then since it's in the wrong project, sorry about that, and I'll open a new issue for reqwest.

[sfackler]

The client itself could perform cleanup periodically during calls if spawning isn't available potentially as a fallback.