New sub-crate: tower-steer

[akshayknarayan]

In some scenarios, you have a list of services and want to send requests to one of them, and get the corresponding response. In the example that motivated me to write this, I had a sharded key-value store, and needed to route requests to the service corresponding to the right shard.

To make this easier, tower-router provides functionality which keeps track of which services have pending requests, waits for those services to be ready, issues calls against them, and returns the response.

This can be thought of as an unbounded (to avoid head-of-line blocking) tower-buffer with extra details to allow routing requests to specific services.

@jonhoo

[jonhoo]

I'll avoid line-by-line reviewing for the time being, and focus on the high-level points:

First, I think this is a cool service, and we should almost definitely have something like this in tower (cc @LucioFranco).

Second, as we discussed, I think ServiceRouter::poll_ready should poll all the services. If the user wants unlimited buffering, they should wrap the Service in an unbounded tower-buffer (which doesn't exist yet 😅), but we should not force this on the user given that it is not a requirement of router that it buffers. This would also, I believe, remove the need for the whole oneshot-arc-mutex-spawn song and dance the code currently has to do.

Also some slightly smaller points:

• Router should probably also be given a reference to the slice of services it is choosing among.
• Should there be a way to add or remove services from a Router?
• I wonder if we could use something like tower-ready-cache to let us more efficiently poll_ready all the services.

And for the interested reader: the reason we need to poll all services is that we do not know at the time of the poll_ready which service the request subsequently sent to call will be for. We have two options. Either, we unconditionally accept the next element, and only then poll_ready on the relevant service, but then we are forced to buffer potentially an unlimited number of elements in case they are all for an unready service. Or, we get head-of-line blocking because every request requires all services to be ready. I suppose in theory there is a third option of trying to "guess" which service the next request will be for, but that still deteriorates to the "buffer indefinitely" in the worst case.

[akshayknarayan]

I've renamed the implementation to tower-steer and deleted most of the code (which was there mostly for the oneshot-mutex stuff).

[hawkw]

I suspect @olix0r will have some insights into this, though he's currently out of the office...

[LucioFranco]

Looks good, I like how simple this is! Left a few comments, otherwise LGTM.

[jonhoo]

Haha, okay, so, given that #432 just landed, @akshayknarayan can you re-organize to match the new layout? Sorry for the trouble! Should be a matter of:

• Add steer feature to Cargo.toml
• git mv tower-steer/src tower/src/steer
• git mv tower/src/steer/lib.rs tower/src/steer/mod.rs
• git rm tower-steer/Cargo.toml tower-steer/README.md
• Add

  #[cfg(feature = "steer")]
  pub mod steer;

  to tower/src/lib.rs

[jonhoo]

The docs should also be updated to not talk about "crate" any more. And the various #! items should probably be removed/lifted to tower/src/lib.rs.

Also, I just noticed. There aren't any tests 🤔

[jonhoo]

Beautiful, thank you!