set STATUS_UNINITIALIZED TAG launching head node

[AmeerHajAli]

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[AmeerHajAli]

@ericl FYI. We need this because non_terminated_nodes is getting KeyError for the head node is when we check the status. This might not happen when we start autoscaler after the head node already started (because the node updater sets some tags in the process and hence the STATUS tag will be in the tags).

[wuisawesome]

We should include this in at least one of the testGetOrCreateHeadNode type tests. Other than that lgtm

[AmeerHajAli]

We should include this in at least one of the testGetOrCreateHeadNode type tests. Other than that lgtm

are you ok with a patch to the node provider create_node that calls the node provider create node and asserts that the tag was set? Something like:

        def _create_node(....):
            self.provider.create_node()
            assert RAY_STATUS in tags
        with patch self.provider.create_node = _create_node:
            commands.get_or_create_head_node(
                SMALL_CLUSTER,
                printable_config_file=config_path,
                no_restart=False,
                restart_only=False,
                yes=True,
                override_cluster_name=None,
                _provider=self.provider,
                _runner=runner)

[wuisawesome]

Yeah you'll have to add a special case check for unmanaged nodes, but that actually sounds better since we're directly asserting that all nodes started by the autoscaler have a status tag set.

[AmeerHajAli]

@wuisawesome , does it look good to you?