Proposal: hyper-util composable pool services

[seanmonstar]

Problem

hyper's "legacy" pool did many things at once, and needed to support the ability to turn off various parts of it.

Users often ask to be able to customize parts of the connection pool. Either with smaller options, such as adding a limit for maximum number of connections, or modifying behavior, such as manually evicting specific connections.

Solution

We plan to create several new utility types, all that are kinds of pools, to separate out the concerns. This will allow each type to worry about a single use case. (See #3849).

Developer Experience (DX)

Before getting to the implementation, this considers the developer experience of using the pool types and their composition to create their own pooling needs.

HTTP/1 pool to a single host

let svc = ServiceBuilder::new()
    .layer(pool::cache())
    .layer(conn::http1())
    .layer(connect:tcp())
    .service(dns::hickory());

HTTP/2 pool to a single host

let svc = ServiceBuilder::new()
    .layer(pool::singleton())
    .layer(conn::http2())
    .layer(connect:tcp())
    .service(dns::hickory());

Adding expiration

let svc = ServiceBuilder::new()
    .layer(pool::expire(|svc, now| {
        // enforce a max lifetime of 300s
        svc.retain(|inner| {
            now - inner.meta().connected_at < Duration::from_secs(300)
        });
        // check again in
        Duration::from_secs(30)
    }))
    .layer(pool::singleton())
    // record some metadata when created
    .layer(svc::meta(|_s| MyMeta {
        connected_at: Instant::now(),
    }))
    .layer(conn::http2())
    .layer(connect:tcp())
    .service(dns::hickory());

General purpose pool

let connect = svc::builder()
    .layer(tls::rustls())
    .layer(tls::alpn(["h2", "http/1.1"]))
    .layer(connect::tcp())
    .service(dns::hickory());

// layer will be applied after negotiation
let http1 = svc::builder()
    .layer(pool::cache())
    .layer(conn::http1());
let http2 = svc::builder()
    .layer(pool::singleton())
    .layer(conn::http2());

let svc = pool::map::builder()
    .key(pool::map::scheme_and_auth())
    .value(move |req| {
        pool::negotiate()
            .connect(connect.clone())
            .inspect(|c| {
                c.negotiated_protocols() == b"h2"
            })
            .left(http1.clone())
            .right(http2.clone())
    });

Implementation

The DX examples are purposefully more concise, while still being possible. The implementation will include types and builders that allow more customization, as desired. But they will also start conservatively, trying to not expose implementation internals while these types are explored more in the real world.

This may mean expose publicly Builder types, but returning currently-unnameable Service or Layer types.

The module structure will be in hyper_util::client::pool, with the following submodules:

• cache
• expire
• map
• negotiate
• singleton

They each work with different layers, so they are laid out below grouped by role, instead of alphabetically.

Cache

The cache is a single list of cached services, bundled with a MakeService. Calling the cache returns either an existing service, or makes a new one. The returned impl Service can be used to send requests, and when dropped, it will try to be returned back to the cache.

Singleton

The singleton pool combines a MakeService that should only produce a single active connection. It can bundle all concurrent calls to it, so that only one connection is made. All calls to the singleton will return a clone of the inner service once established. This fits the HTTP/2 case well.

Negotiate

The negotiate pool allows for a service that can decide between two service types based on an intermediate return value. It differs from typical routing since it doesn't depend on the request, but the response. The main use case is support ALPN upgrades to HTTP/2, with a fallback to HTTP/1.

It's possible we could make this type even more general purpose, but that's a potential feature for the future.

Expire

Most pools need a way to expire services/connections within them. Instead of making each type know how to spawn a watching task and evict connections, the expire pool can wrap any other pool type with a single watcher concept. This has the additional benefit of allowing more customization on when and why to expire a connection.

The expire pool wraps another, along with a timer and way to inspect the wrapped pool. The other pool types all will have a retain method on them, which will allow passing a closure to determine which cached services should be retained or expired. Thus, an inner service could check both it's idle time and its lifetime, for instance.

Map

The map isn't a typical Service, but rather stand-alone type that can map requests to a key and service factory. This is because the service is more of a router, and cannot determine which inner service to check for backpressure since it's not know until the request is made.

The map implementation allows customization of extracting a key, and how to construct a MakeService for that key.

FAQ

Can I limit the number of connections being made at once?

let svc = ServiceBuilder::new()
    .layer(pool::cache())
+   .concurrency_limit(5)    
    .layer(conn::http1())
    .layer(connect:tcp())
    .service(dns::hickory());

Can I limit the number of max connections?

A layer which shares a count and only drops once the connection is fully dropped could be implemented and added:

let svc = ServiceBuilder::new()
    .layer(pool::cache())
+   .layer(conn_semaphore(5))
    .layer(conn::http1())
    .layer(connect:tcp())
    .service(dns::hickory());

Can I evict a connection whenever I want?

pool.retain(|c| {
    if should_evict(c) {
        // bye
        false
    } else {
        // keep it
        true
    }
})

[seanmonstar]

cc @cratelyn @howardjohn @devsnek @jlizen; You've mentioned interest in the idea, or know others in your org who may be interested.

Also, I have the implementation for each of these 90% of the way done, to ensure the design actually worked. I'll make a PR for each one separately.

[jlizen]

Thanks for the ping! I'm about to be on some medical leave, but I'm looping in other interested folks from my org

[aajtodd]

One of the use cases we are looking at is better multi-NIC support (e.g. round robin and spread load across multiple NIC interfaces). Currently the HttpConnector has the ability to set a single interface to bind to. Would this new modularization enable this possibility?

It would be interesting to see what building a client/connector looks like with this new modularized pool.

The other thing that comes to mind is how telemetry for pools/connections may eventually look (maybe not a concern in this first past but worth considering how it will be done that doesn't require everyone reinventing the wheel).

[devsnek]

@seanmonstar we basically have two use cases:

• "normal" http client which pools connections by hostname. we currently use the hyper-util legacy client for this.
• http client which pools connections by a custom value that is not derived from the hostname, with a custom tcp connector that writes a special stream header before passing the stream up to the http client. this is what the PR i asked you about was meant to enable. we currently do not pool these connections at all and feel sad about the connection overhead, so this is what i'm most interested in solving.

it seems like, depending on what the Request type of these various services is (hopefully it is like &http::Request or &Parts, instead of Uri like the current legacy pool?) it could be general enough for our purposes.

[seanmonstar]

@aajtodd

One of the use cases we are looking at is better multi-NIC support (e.g. round robin and spread load across multiple NIC interfaces). Currently the HttpConnector has the ability to set a single interface to bind to. Would this new modularization enable this possibility?

You already could take something like tower::balance and use that with multiple HttpConnectors configured to each interface. These pieces should make it easier to compose how you want, though. For instance, it'd be easier to put the balance layer in-between the map, and still have a cache of connections on each interface. Something like:

let svc = pool::map::builder()
    .key(pool::map::scheme_and_auth())
    .value(move |req| {
        ServiceBuilder::new()
            .layer(balance::robin(list_of_nics))
            .layer(pool::cache())
            .service(connect::http())
    });

---

@devsnek

it seems like, depending on what the Request type of these various services is (hopefully it is like &http::Request or &Parts, instead of Uri like the current legacy pool?) it could be general enough for our purposes.

Yes! I imagine a full client stack internally looks like this:

// shorter example of Service::call
async fn call(&mut self, req: Request<B>) -> Result<Resp, Error> {
    let conn_svc = self.pool.call(&req).await?;
    conn_svc.call(req).await
}

So, the pool::Map could grab whatever it wants from its generic "request" argument, both for making the key, but also perhaps for checking extensions or something when constructing the service for that key (maybe one could customize ALPN that way, etc).