[SkyServe] Update Autoscaler Decision, Use Target QPS, Deprecate Auto Restart

[MaoZiming]

Changes:

• Update the AutoscalerDecision;
• Change the type of return value for evaluate_scaling (i.e., List[AutoscalerDecision]);
• Using the same interface (target_qps instead of {upper,lower}_threshold) as spot policy;
• Migrate the _get_desired_num_replicas function.
• Deprecate auto_restart

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Manual Test: sky serve up tests/test_yamls/test_serve_autoscaler.yaml followed by python3 tests/test_serve_autoscaler.py
• pytest tests/test_smoke.py:test_skyserve_gcp_http
• pytest tests/test_smoke.py::test_skyserve_auto_restart
• All smoke tests: pytest tests/test_smoke.py

[cblmemo]

Thanks for migrating this!! Left several comments 🫡

[MaoZiming]

@cblmemo Let's use the resource_override_dict as in the spot policy. Updated

[MaoZiming]

        if not auto_restart:
            with ux_utils.print_exception_no_traceback():
                raise ValueError('auto_restart=False is deprecated.')

Deprecate auto_restart in service_spec.py

[cblmemo]

Thanks for the prompt fix!!! 🚀 Left some comments.

[MaoZiming]

@cblmemo Thanks for the quick reviews!

[cblmemo]

Thanks for the prompt fix!! Sorry for the delay, left some comments (mostly nits). After these the PR should be ready to go!

[cblmemo]

Thanks for the fix!! More comments, mostly nits ;)

[MaoZiming]

@cblmemo Thanks for the comments! fixed

[Michaelvll]

Thanks for submitting the PR @MaoZiming! This will be a great improvement to the UX of the serving autoscaling policy. Left several comments.

[MaoZiming]

@Michaelvll Thanks! PTAL

[Michaelvll]

Thanks for this important change @MaoZiming! LGTM. Left several nits. : )

[cblmemo]

Thanks for the awesome work!! 🚀 It looks great. Left some nits 🫡 Please make sure all smoke tests passed before merging it 👀

[cblmemo]

Could we still have a round-robin test for 3 replicas?

[MaoZiming]

I think we might not need the smoke test for round-robin, as now the default behavior for failed replicas is auto-restart.
We already have the test_skyserve_auto_restart

[MaoZiming]

@cblmemo Thanks for the comments! All smoke tests passed.