[serve] Cherry-pick fix for router queue length metric #38020
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…isconnects (ray-project#37965) The `ray_serve_deployment_queued_queries` metric tracks the number of queries that have yet to be assigned a replica. If a client disconnects before its query has been assigned a replica– but after the metric has counted their query– the query terminates, but the metric doesn't decrease. This change decrements `ray_serve_deployment_queued_queries` when a queued request is disconnected. Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
|
(I am not qualified to review this.) |
|
@edoakes This is a P1 fix. Is it OK to put it in 2.7? |
|
@zhe-thoughts we discovered today that the bug doesn't only affect observability but also can affect our autoscaling behavior, preventing downscaling in some circumstances. This bumps it to more like a P0.5 -- would like to get it in if possible but if it delays the release significantly then I think we can stomach 2.7. |
zhe-thoughts
approved these changes
Aug 7, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Cherry-pick #37965
Related issue number
#37943
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.