[serve] Optionally namespace serve clusters #8447
Conversation
|
Can one of the admins verify this patch? |
|
Test FAILed. |
| @@ -81,6 +82,9 @@ def init(blocking=False, | |||
| requirement. | |||
|
|
|||
| Args: | |||
| cluster_name (str): A unique name for this serve cluster. This allows | |||
There was a problem hiding this comment.
- (str, None).
- Discuss the default option?
| ).remote(queueing_policy.value, policy_kwargs, start_server, http_host, | ||
| http_port, metric_exporter) | ||
| ).remote(cluster_name, queueing_policy.value, policy_kwargs, start_server, | ||
| http_host, http_port, metric_exporter) | ||
|
|
There was a problem hiding this comment.
can we check the port is available somewhere here? With multiple serve cluster, it will be quite easy to forget to assign new port number and have a port collision. Currently, we just silently failed to start HTTP actor and timeout in trying to reach HTTP, which is not a pleasant user experience
|
Test FAILed. |
|
Test FAILed. |
|
Test FAILed. |
|
@simon-mo The primary use case is that this provides a hacky way to get horizontal scalability if need be - the user can start 3 clusters each with N/3 replicas instead of one with N. |
|
Test FAILed. |
|
Test PASSed. |
Why are these changes needed?
Enables users to launch multiple serve clusters on one ray cluster.
Checks
scripts/format.shto lint the changes in this PR.