Limit the number of concurrent outbound HTTP requests

[rylev]

I'm trying to limit the number of outbound requests guests can make to make sure no single guest can exhaust resources on the host system. Ideally, I'd be able to limit the number of client ports used by a guest application because I believe this (besides open file descriptors) is the resource I'm really worried about exhausting. However, I believe that this number is buried deep in the internals of hyper, and we don't really have access to it from inside Spin.

So the next best idea is to just limit the number of outbound requests which should be a fairly close approximation of what we want. I have a branch with a POC implementation that just adds an AtomicUsize to the instance state and increment/decrements this number before and after each request. Additionally it checks this number against some configured max and rejects the request if we're currently doing too many.

Does this sound like a good approach? If so, I will clean up the branch and open a PR. Otherwise, I'd love to brainstorm a better approach.

cc @dicej who recently added connection pooling and @lann who I know will be interested.

[lann]

I was thinking this would limit concurrency by blocking requests rather than killing them, which might point at our friend Mr Semaphore.

[lann]

Would it make sense for this to be part of runtime config?

[radu-matei]

Being able to limit the number of outbound connections is definitely a feature that would be helpful to have, and it being part of runtime config SGTM.

@lann would you mind expanding a bit how you're thinking about blocking vs. killing them re: host resource exhaustion?

[lann]

would you mind expanding a bit how you're thinking about blocking vs. killing them re: host resource exhaustion?

Part of this is about portability; if I have an app that makes 20 concurrent requests, test it in an environment with a concurrency limit of 100, and then deploy to an environment with a limit of 10 I would prefer that all my requests go through with lower concurrency than have them start failing at some arbitrary point.

[radu-matei]

That's a great point, thanks for expanding on it.

The approach LGTM!