[autoscaler] Flag flip for resource_demand_scheduler should take into account queue

[AmeerHajAli]

I set the max back log size from 1 to 1M and it did not seem to affect much the performance, I set it now to 10k, we can change that in the future if necessary.

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[AmeerHajAli]

Here are a few data points (I updated the PR accordingly):

backlog_size = 10k
number of nodes = 2k (1k cpu nodes and 1k gpu nodes)
demands = [{"GPU": 1}, {"CPU":1}] * (backlog_size=10k)

• The time get_nodes_to_launch takes is 4.6 seconds.
• Reducing backlog_size to 1k makes it take 0.32 seconds.
• Reducing the number of nodes to 200 (100 cpu nodes and 100 gpu nodes) makes it take 1.02 second.
• Reducing backlog_size to 1k and the number of nodes to 200 (100 cpu nodes and 100 gpu nodes) make it take 0.06 seconds.

Bottom line, If we want to be on the safe side (always less than 1 second) with 100-1000 nodes, we can set the backlog size to 1k. I also prefer this option since it is also less aggressive.

[AmeerHajAli]

@ericl @wuisawesome , I added a couple of more tests to see different resource demand vector sizes and different available nodes. Note that I changed the monitor function not to just trim the backlog bundles but also the waiting_bundles + infeasible_bundles So that we really bound the execution time of get_nodes_to_launch.

[wuisawesome]

Hey we need to block on merging this until we make sure we aren't scaling to aggressively. I will describe this in more detail in my simulator PR.

[ericl]

Hey we need to block on merging this until we make sure we aren't scaling to aggressively. I will describe this in more detail in my simulator PR.

Do you mean beyond the upscaling throttling? (e.g., launch at most 5 instances starting off, then at most 20% pending launches)

[wuisawesome]

This may actually need some investigating (there's a chance this is a bug in the simulator still). I will get back to y'all on this by EOD.

[AmeerHajAli]

Isn't that handled in the max launch concurrency?

[ericl]

Looks good, but let's raise the timeouts to prevent possible test flakiness.