Pooling

[slinkydeveloper]

The goal of this issue is to implement connection pooling within the invoker. Probably depends on #96

[slinkydeveloper]

Requirements for the pool (ordered by priority):

• Reuse TCP/TLS connections across invocations
• Be able to always provide quota for a new invocation, eventually opening a new tcp connection when the http2 max concurrent streams is exhausted. This is required to avoid logical deadlocks!
• For use cases where no load balancing exists at L7, provide a tunable load balancing policy to decide whether to reuse an existing tcp connection or open a new one
• When possible, hit the endpoint that previously served service id to allow state caching

Non-requirements:

• Support HTTP/1

[slinkydeveloper]

Different network deployments to support:

• Protocol is in request/response mode and there is an API gateway/L7 load balancer between restate and the service. This is in particular the case of AWS Lambda
• Protocol is in bidi-stream mode and there is an API gateway/L7 load balancer between restate and the service. Example load balancers are Envoy/Linkerd
• Vanilla Kubernetes direct pod to pod communication with a vanila Service deployment. This can be either L4 load balancing with vIPs or DNS.

[slinkydeveloper]

Now that #237 is in place, we could implement a simple pooling strategy by sharing the hyper::Client among invocation tasks. This is still problematic with vanilla kubernetes pods (only L4 load balancing), and can still block when the max-streams is reached (as no new connections are opened when quota is reached), but the suspension timeout guarantees that after a while stream quotas are released for invocations waiting for very long time on completions.

[slinkydeveloper]

I've opened #293 with the simple pooling strategy that shares the hyper::Client. Let's keep this issue open though, as I think #293 is only a temporary solution but not the long term strategy.