[serve] Add exponential backoff for queue_len_response_deadline_s

[edoakes]

Why are these changes needed?

We currently have a flat deadline of 0.1s (by default). Under heavy load or high network latency conditions, this deadline might be consistently missed and cause requests to pile up because they're unable to be scheduled.

#42001 made this deadline configurable, but setting it high by default defeats its purpose (to reduce tail latency when a single replica is overloaded/blocked/unresponsive).

This change backs off the deadline exponentially so the initial deadline can still be low while avoiding "halting" under degraded conditions.

The max is set to 1s by default but can be configured using RAY_SERVE_MAX_QUEUE_LENGTH_RESPONSE_DEADLINE_S.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[edoakes]

Example of artificially injecting a 0.8s sleep into the response:

(ProxyActor pid=33502) WARNING 2023-12-21 11:06:20,159 proxy 127.0.0.1 router.py:740 - Failed to get queue length from replica default#A#u3m0im2u within 0.1s. If this happens repeatedly it's likely caused by high network latency in the cluster. You can configure the deadline using the `RAY_SERVE_QUEUE_LENGTH_RESPONSE_DEADLINE_S` environment variable.
(ProxyActor pid=33502) WARNING 2023-12-21 11:06:20,361 proxy 127.0.0.1 router.py:740 - Failed to get queue length from replica default#A#u3m0im2u within 0.2s. If this happens repeatedly it's likely caused by high network latency in the cluster. You can configure the deadline using the `RAY_SERVE_QUEUE_LENGTH_RESPONSE_DEADLINE_S` environment variable.
(ProxyActor pid=33502) WARNING 2023-12-21 11:06:20,815 proxy 127.0.0.1 router.py:740 - Failed to get queue length from replica default#A#u3m0im2u within 0.4s. If this happens repeatedly it's likely caused by high network latency in the cluster. You can configure the deadline using the `RAY_SERVE_QUEUE_LENGTH_RESPONSE_DEADLINE_S` environment variable.
(ProxyActor pid=33502) WARNING 2023-12-21 11:06:21,718 proxy 127.0.0.1 router.py:740 - Failed to get queue length from replica default#A#u3m0im2u within 0.8s. If this happens repeatedly it's likely caused by high network latency in the cluster. You can configure the deadline using the `RAY_SERVE_QUEUE_LENGTH_RESPONSE_DEADLINE_S` environment variable.
(ServeReplica:default:A pid=33504) INFO 2023-12-21 11:06:22,688 default_A u3m0im2u c39176f8-7c1b-4966-bd5b-c45135dc3e46 / replica.py:745 - __CALL__ OK 0.1ms

[GeneDer]

LGTM!

[GeneDer]

Can probably follow up by a doc change 🙃