Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[autoscaler] Better validation for min_workers and max_workers #13779

Merged
merged 118 commits into from
Jan 29, 2021
Merged

[autoscaler] Better validation for min_workers and max_workers #13779

merged 118 commits into from
Jan 29, 2021

Conversation

AmeerHajAli
Copy link
Contributor

closes #13775 reported by Ed.

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Ameer Haj Ali and others added 30 commits September 24, 2020 00:26
prepare for head node
7248cf9
move command runner interface outside _private
bc43e46
Merge github.com:ray-project/ray
14be0a1
remove space
ab660a8
Eric
1ea0c1f
flake
dad31ae
Merge github.com:ray-project/ray
49bcf56
min_workers in multi node type
16d736d
Merge github.com:ray-project/ray
06911df
fixing edge cases
0d8dddb
eric not idle
fe69ce3
fix target_workers to consider min_workers of node types
35832ed
idle timeout
ca0be53
minor
c9518bd
minor fix
5452d39
test
9e904cd
lint
f9edcbe
eric v2
cb02267
eric 3
5e5d403
min_workers constraint before bin packing
4d44cd8
Merge github.com:ray-project/ray
614abbf
Update resource_demand_scheduler.py
818a63a
@wuisawesome
Revert "Update resource_demand_scheduler.py"
539b29c 
This reverts commit 818a63a.
reducing diff
9a63866
Merge branch 'master' of github.com:AmeerHajAli/ray
7501623
make get_nodes_to_launch return a dict
b4edd21
Merge github.com:ray-project/ray
fc48725
merge
0aef789
weird merge fix
39245a8
auto fill instance types for AWS
c7eb4ad
@AmeerHajAli AmeerHajAli added this to the Serverless Autoscaling milestone Jan 29, 2021
@AmeerHajAli
Copy link
Contributor Author

@ericl, can you please merge?

@AmeerHajAli AmeerHajAli added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Jan 29, 2021
@ericl ericl merged commit 4d6817c into ray-project:master Jan 29, 2021
simon-mo added a commit that referenced this pull request Jan 29, 2021
@simon-mo
Revert "[autoscaler] Better validation for min_workers and max_workers (
384fc1f 
#13779)"

This reverts commit 4d6817c.
@simon-mo simon-mo mentioned this pull request Jan 29, 2021
Merged
@simon-mo
Copy link
Contributor

Reverting this PR because it consistently fail on Windows, please re-revert once you fix windows build.


================================== FAILURES ===================================
________ AutoscalingConfigTest.testValidateDefaultConfigMinMaxWorkers _________

self = <com_github_ray_project_ray.python.ray.tests.test_autoscaler_yaml.AutoscalingConfigTest testMethod=testValidateDefaultConfigMinMaxWorkers>

    def testValidateDefaultConfigMinMaxWorkers(self):
        aws_config_path = os.path.join(
            RAY_PATH, "autoscaler/aws/example-multi-node-type.yaml")
>       with open(aws_config_path) as f:
E       OSError: [Errno 22] Invalid argument: '\\\\?\\C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\Bazel.runfiles_6y4y9g7o\\runfiles\\com_github_ray_project_ray\\python\\ray\\autoscaler/aws/example-multi-node-type.yaml'

\\?\C:\Users\RUNNER~1\AppData\Local\Temp\Bazel.runfiles_6y4y9g7o\runfiles\com_github_ray_project_ray\python\ray\tests\test_autoscaler_yaml.py:51: OSError

simon-mo added a commit that referenced this pull request Jan 29, 2021
@simon-mo
Revert "[autoscaler] Better validation for min_workers and max_workers (
5080802 
#13779)" (#13807)

This reverts commit 4d6817c.
fishbone pushed a commit to fishbone/ray that referenced this pull request Feb 16, 2021
@wuisawesome @ericl
@DmitriGekhtman @fishbone
[autoscaler] Better validation for min_workers and max_workers (ray-p…
a51dfbc 
…roject#13779)

* prepare for head node

* move command runner interface outside _private

* remove space

* Eric

* flake

* min_workers in multi node type

* fixing edge cases

* eric not idle

* fix target_workers to consider min_workers of node types

* idle timeout

* minor

* minor fix

* test

* lint

* eric v2

* eric 3

* min_workers constraint before bin packing

* Update resource_demand_scheduler.py

* Revert "Update resource_demand_scheduler.py"

This reverts commit 818a63a.

* reducing diff

* make get_nodes_to_launch return a dict

* merge

* weird merge fix

* auto fill instance types for AWS

* Alex/Eric

* Update doc/source/cluster/autoscaling.rst

* merge autofill and input from user

* logger.exception

* make the yaml use the default autofill

* docs Eric

* remove test_autoscaler_yaml from windows tests

* lets try changing the test a bit

* return test

* lets see

* edward

* Limit max launch concurrency

* commenting frac TODO

* move to resource demand scheduler

* use STATUS UP TO DATE

* Eric

* make logger of gc freed refs debug instead of info

* add cluster name to docker mount prefix directory

* grrR

* fix tests

* moving docker directory to sdk

* move the import to prevent circular dependency

* smallf fix

* ian

* fix max launch concurrency bug to assume failing nodes as pending and consider only load_metric's connected nodes as running

* small fix

* deflake test_joblib

* lint

* placement groups bypass

* remove space

* Eric

* first ocmmit

* lint

* exmaple

* documentation

* hmm

* file path fix

* fix test

* some format issue in docs

* modified docs

* joblib strikes again on windows

* add ability to not start autoscaler/monitor

* a

* remove worker_default

* Remove default pod type from operator

* Remove worker_default_node_type from rewrite_legacy_yaml_to_availble_node_types

* deprecate useless fields

* fix error msg

* validate sum min_workers < max_workers

* 1 more edge case test

* lint

Co-authored-by: Ameer Haj Ali <ameerhajali@ameers-mbp.lan>
Co-authored-by: Alex Wu <alex@anyscale.io>
Co-authored-by: Alex Wu <itswu.alex@gmail.com>
Co-authored-by: Eric Liang <ekhliang@gmail.com>
Co-authored-by: Ameer Haj Ali <ameerhajali@Ameers-MacBook-Pro.local>
Co-authored-by: root <root@ip-172-31-56-188.us-west-2.compute.internal>
Co-authored-by: Dmitri Gekhtman <dmitri.m.gekhtman@gmail.com>
fishbone pushed a commit to fishbone/ray that referenced this pull request Feb 16, 2021
@simon-mo @fishbone
Revert "[autoscaler] Better validation for min_workers and max_workers (
2c4c83b 
ray-project#13779)" (ray-project#13807)

This reverts commit 4d6817c.
fishbone added a commit to fishbone/ray that referenced this pull request Feb 16, 2021
@fishbone
Revert "Revert "[autoscaler] Better validation for min_workers and ma…
8ce069e 
…x_workers (ray-project#13779)" (ray-project#13807)"

This reverts commit 2c4c83b.
fishbone added a commit to fishbone/ray that referenced this pull request Feb 16, 2021
@fishbone
Revert "[autoscaler] Better validation for min_workers and max_workers (
5ffdb5f 
ray-project#13779)"

This reverts commit a51dfbc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wuisawesome wuisawesome wuisawesome approved these changes

@ericl ericl ericl approved these changes

@edoakes edoakes Awaiting requested review from edoakes

Assignees

@ericl ericl

@wuisawesome wuisawesome

@edoakes edoakes

Labels
tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Milestone
Infra Backlog
Development

Successfully merging this pull request may close these issues.

[autoscaler] Better validation for min_workers and max_workers
6 participants
@AmeerHajAli @simon-mo @ericl @wuisawesome @edoakes @DmitriGekhtman