[autoscaler] run setup commands with restart_only=True

[ijrsvt]

Why are these changes needed?

The Docker Command Runner may restart containers, requiring setup_commands to be re-run.

NOTE: This does not apply to workers at the moment, as per #13861

Related issue number

Closes #13587

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ijrsvt]

Also do this for init_commands in autoscaler.py (WORKERS)

[ijrsvt]

TODO(ilr): Make sure to validate with workers!

[ijrsvt]

@AmeerHajAli I'm not sure what I should do with respect to worker nodes, I just realized that we don't have a field in ray-schema.json for "restart_only". We try to get that value here in autoscaler.py, but we never set it in commands.py. Should I:

1. Leave this PR as only for head node and open an issue for adding it to worker nodes
2. Open a separate PR & fix it for worker nodes (then merge this PR on top of that)

[AmeerHajAli]

@AmeerHajAli I'm not sure what I should do with respect to worker nodes, I just realized that we don't have a field in ray-schema.json for "restart_only". We try to get that value here in autoscaler.py, but we never set it in commands.py. Should I:

1. Leave this PR as only for head node and open an issue for adding it to worker nodes
2. Open a separate PR & fix it for worker nodes (then merge this PR on top of that)

This is a good catch @ijrsvt. Lets go with option 1 for now.
Thanks!

[AmeerHajAli]

LGTM.

[richardliaw]

haven't tried this out but looks fine - thanks @ijrsvt !

[AmeerHajAli]

@richardliaw , can you please merge?