[serve] Make HTTP proxy (almost) a regular Serve deployment #15820

edoakes · 2021-05-14T18:14:41Z

Why are these changes needed?

Entirely removes http_state.py and instead expresses the HTTP proxy as a regular deployment. This reduces the number of core concepts we need to maintain. It also offers a clear avenue to make the HTTP settings declarative and updateable in the future .

We do need some special casing here to handle the "EveryNode" deployment strategy. We should be able to remove this in the future if we can lean on placement group policies.

Related issue number

Closes #15561

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

…-proxy-as-deployment

edoakes · 2021-05-19T15:59:40Z

Blocking this on #15909 so we can wait for config changes to propagate without needing a hacky solution

…-proxy-as-deployment

simon-mo · 2021-07-02T23:08:58Z

python/ray/serve/config.py

@@ -31,7 +31,7 @@ class BackendConfig(BaseModel):
            for shutdown. Defaults to 20s.
    """

-    num_replicas: PositiveInt = 1
+    num_replicas: Any = 1


Union[str, PositiveInt]

jiaodong · 2021-07-07T18:19:08Z

python/ray/serve/api.py

-            raise TimeoutError(
-                "HTTP proxies not available after {HTTP_PROXY_TIMEOUT}s.")
+    client = Client(controller, controller_name, detached=detached)
+    _set_global_client(client)


are we calling these lines twice ? Line 730

jiaodong · 2021-07-07T18:23:32Z

python/ray/serve/backend_state.py

@@ -81,6 +88,8 @@ def actor_handle(self) -> ActorHandle:

    def start_or_update(self, backend_info: BackendInfo):
        self._actor_resources = backend_info.replica_config.resource_dict
+        if self._node_resource is not None:
+            self._actor_resources[self._node_resource] = 0.001


what does 0.001 mean here ?

jiaodong · 2021-07-07T18:29:55Z

python/ray/serve/backend_state.py

@@ -837,8 +864,14 @@ def _scale_backend_replicas(
        """
        assert (backend_tag in self._backend_metadata
                ), "Backend {} is not registered.".format(backend_tag)
-        assert target_replicas >= 0, ("Number of replicas must be"
-                                      " greater than or equal to 0.")
+        if isinstance(target_replicas, str):


This changes target_replicas' type from int (which also implies it from its name) defined in line 853, maybe we can assign a special value for it like -1, or introduce a new flag to control its behavior ?

jiaodong · 2021-07-07T18:31:36Z

python/ray/serve/backend_state.py

-        assert target_replicas >= 0, ("Number of replicas must be"
-                                      " greater than or equal to 0.")
+        if isinstance(target_replicas, str):
+            assert target_replicas == "EveryNode"


sorry i don't have full context on this concept yet .. does "EveryNode" means the target replica num == all nodes we have in the ray cluster ? If so it might be simpler to handle it outside of this function call and simply pass in a number.

…-proxy-as-deployment

jiaodong · 2021-08-10T20:45:17Z

is this WIP or blocked on review ?

edoakes · 2021-08-10T22:49:11Z

@jiaodong feel free to review, the only thing it's really blocked on is better handling of constructor failure so we can raise an error if the HTTP proxy fails to start up (e.g., bad HTTP host or port)

jiaodong · 2021-08-11T00:38:33Z

i see .. let me ensure to actually put sometime on the constructor failure issue to unblock this.

…-proxy-as-deployment

jiaodong · 2021-09-16T17:12:52Z

Is this still WIP ?

edoakes · 2021-09-20T17:10:32Z

@jiaodong this is unblocked now that you made the constructor failure PR, but there are a ton of merge conflicts from the refactors. I will try to clean it up in the background otherwise we can prioritize sometime in the coming weeks.

bveeramani · 2022-01-30T05:51:14Z

‼️ ACTION REQUIRED ‼️

We've switched our code formatter from YAPF to Black (see #21311).

To prevent issues with merging your code, here's what you'll need to do:

Install Black

pip install -I black==21.12b0

Format changed files with Black

curl -o format-changed.sh https://gist.githubusercontent.com/bveeramani/42ef0e9e387b755a8a735b084af976f2/raw/7631276790765d555c423b8db2b679fd957b984a/format-changed.sh
chmod +x ./format-changed.sh
./format-changed.sh
rm format-changed.sh

Commit your changes.

git add --all
git commit -m "Format Python code with Black"

Merge master into your branch.

git pull upstream master

Resolve merge conflicts (if necessary).

After running these steps, you'll have the updated format.sh.

simon-mo · 2022-02-01T18:25:54Z

@edoakes should we close this as a prototype and I can open a new PR to tackle this?

edoakes · 2022-02-01T18:54:39Z

@simon-mo sounds good to me

edoakes added 3 commits May 14, 2021 11:58

WIP

1a7259c

working!

98e410b

Merge branch 'master' of https://github.com/ray-project/ray into http…

4a91de7

…-proxy-as-deployment

edoakes force-pushed the http-proxy-as-deployment branch from e75ce44 to 4a91de7 Compare May 14, 2021 18:23

edoakes added 2 commits May 14, 2021 14:20

fix some tests

838f730

Fix some tests

30396f5

edoakes force-pushed the http-proxy-as-deployment branch from 87360c9 to 30396f5 Compare May 14, 2021 20:35

edoakes added 2 commits May 18, 2021 10:24

Merge branch 'master' of https://github.com/ray-project/ray into http…

aa876c5

…-proxy-as-deployment

Merge branch 'master' of https://github.com/ray-project/ray into http…

1277bdc

…-proxy-as-deployment

edoakes mentioned this pull request May 19, 2021

[serve] Rolling updates for user_config changes #15909

Merged

6 tasks

edoakes changed the title ~~[serve] Make HTTP proxy (almost) a regular Serve deployment~~ [WIP][serve] Make HTTP proxy (almost) a regular Serve deployment May 19, 2021

edoakes added 3 commits June 30, 2021 16:38

Merge branch 'master' of https://github.com/ray-project/ray into http…

92f0b1a

…-proxy-as-deployment

Merge branch 'master' of https://github.com/ray-project/ray into http…

4a9e061

…-proxy-as-deployment

fixes

fc0ec55

edoakes assigned jiaodong and simon-mo Jul 1, 2021

edoakes changed the title ~~[WIP][serve] Make HTTP proxy (almost) a regular Serve deployment~~ [serve] Make HTTP proxy (almost) a regular Serve deployment Jul 1, 2021

simon-mo reviewed Jul 6, 2021

View reviewed changes

jiaodong reviewed Jul 7, 2021

View reviewed changes

edoakes added 10 commits July 7, 2021 13:59

Merge branch 'master' of https://github.com/ray-project/ray into http…

5f43a49

…-proxy-as-deployment

fix some tests

78bd39e

Merge branch 'master' of https://github.com/ray-project/ray into http…

8e505a2

…-proxy-as-deployment

Merge branch 'master' of https://github.com/ray-project/ray into http…

e0fbb35

…-proxy-as-deployment

small fix

405a625

wait for create_endpoint

59530ed

wait properly for both goals

cdcd2a7

Merge branch 'master' of https://github.com/ray-project/ray into http…

0408504

…-proxy-as-deployment

Merge branch 'master' of https://github.com/ray-project/ray into http…

35cce4b

…-proxy-as-deployment

Merge branch 'master' of https://github.com/ray-project/ray into http…

910c023

…-proxy-as-deployment

Merge branch 'master' of https://github.com/ray-project/ray into http…

d5f254d

…-proxy-as-deployment

simon-mo mentioned this pull request Aug 4, 2021

[Serve] http_options only works in the first serve.start call #17577

Closed

Merge branch 'master' of https://github.com/ray-project/ray into http…

69da6d5

…-proxy-as-deployment

edoakes added 2 commits August 19, 2021 13:15

Merge branch 'master' of https://github.com/ray-project/ray into http…

e42161f

…-proxy-as-deployment

fixes

09945b3

jiaodong mentioned this pull request Aug 19, 2021

Better constructor failure handling in ray serve #16922

Merged

6 tasks

edoakes mentioned this pull request Oct 6, 2021

Properly health check HTTP proxy & report its status #19151

Closed

edoakes closed this Feb 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[serve] Make HTTP proxy (almost) a regular Serve deployment #15820

[serve] Make HTTP proxy (almost) a regular Serve deployment #15820

edoakes commented May 14, 2021 •

edited

Loading

edoakes commented May 19, 2021

simon-mo Jul 2, 2021

jiaodong Jul 7, 2021

jiaodong Jul 7, 2021

jiaodong Jul 7, 2021

jiaodong Jul 7, 2021

jiaodong commented Aug 10, 2021

edoakes commented Aug 10, 2021

jiaodong commented Aug 11, 2021

jiaodong commented Sep 16, 2021

edoakes commented Sep 20, 2021

bveeramani commented Jan 30, 2022

simon-mo commented Feb 1, 2022

edoakes commented Feb 1, 2022

[serve] Make HTTP proxy (almost) a regular Serve deployment #15820

[serve] Make HTTP proxy (almost) a regular Serve deployment #15820

Conversation

edoakes commented May 14, 2021 • edited Loading

Why are these changes needed?

Related issue number

Checks

edoakes commented May 19, 2021

simon-mo Jul 2, 2021

Choose a reason for hiding this comment

jiaodong Jul 7, 2021

Choose a reason for hiding this comment

jiaodong Jul 7, 2021

Choose a reason for hiding this comment

jiaodong Jul 7, 2021

Choose a reason for hiding this comment

jiaodong Jul 7, 2021

Choose a reason for hiding this comment

jiaodong commented Aug 10, 2021

edoakes commented Aug 10, 2021

jiaodong commented Aug 11, 2021

jiaodong commented Sep 16, 2021

edoakes commented Sep 20, 2021

bveeramani commented Jan 30, 2022

‼️ ACTION REQUIRED ‼️

simon-mo commented Feb 1, 2022

edoakes commented Feb 1, 2022

edoakes commented May 14, 2021 •

edited

Loading