Make PrefixCacheAwareRouter imbalance threshold less surprising#59390
Make PrefixCacheAwareRouter imbalance threshold less surprising#59390kouroshHakha merged 2 commits intoray-project:masterfrom
Conversation
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the imbalanced_threshold in PrefixCacheAffinityRouter to a very large value, making prefix-aware routing the default behavior, which is a sensible change. My review includes suggestions to update the corresponding documentation and test fixtures to align with this new default, ensuring consistency and correctness.
python/ray/llm/_internal/serve/routing_policies/prefix_aware/prefix_aware_router.py
Outdated
Show resolved
Hide resolved
python/ray/llm/_internal/serve/routing_policies/prefix_aware/prefix_aware_router.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
github-actions
bot
added
the
stale
…prising (ray-project#59390) Signed-off-by: Seiji Eicher <seiji@anyscale.com> Signed-off-by: jasonwrwang <jasonwrwang@tencent.com>
…prising (ray-project#59390) Signed-off-by: Seiji Eicher <seiji@anyscale.com>
…prising (ray-project#59390) Signed-off-by: Seiji Eicher <seiji@anyscale.com>
3 participants
Uh oh!
There was an error while loading. Please reload this page.