[Core] Make multi-node job fail fast when one fails, and output segment fault

[Michaelvll]

Fixes #3080
ray driver somehow failed to print the output segmentation fault when the job fails due to that error. We try to make the UX better for this case, by checking the returncode with 139 and printing out the possible reason.

This PR also fixes #3116 the multi-node case where one of the worker fails, we should fail the entire job quickly, instead of waiting for the other workers.

num_nodes: 4

run: |
  if [ "$SKYPILOT_NODE_RANK" == "2" ]; then
      exit 1
  fi
  sleep 10000

This PR also makes the job submission a bit more efficient, by releasing the resources a earlier.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • Reproducible code in [Core] Fail to show the task error of segmentation fault #3080
  • Reproducible code in [Core] Multi-node job does not fail fast when one of the worker fails #3116
  • sky launch -c test --num-nodes 100 --cpus 2 --cloud aws echo hi
  • sky launch -c test --num-nodes 100 --cpus 2 --cloud aws tests/test_yaml/failed_worker_run.yaml
  • sky launch -c test --num-nodes 100 --cpus 2 --cloud aws --detach-setup tests/test_yaml/failed_worker_setup.yaml
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
  • pytest tests/test_smoke.py::test_multi_node_failure
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[concretevitamin]

Thanks @Michaelvll. Could we also add a basic smoke tests to cover the multinode job failure case?

[concretevitamin]

Thanks @Michaelvll, LGTM.

[Michaelvll]

@concretevitamin In order to support the case of having different setup on different nodes, I added a SKYPILOT_SETUP_NODE_IPS and SKYPILOT_SETUP_NODE_RANK env var for the setup phase (we don't use the same name as the env var in run section to avoid the confusion for why the same env vars are different in setup and run section). Fixes #2546 Wdyt?

[concretevitamin]

Mostly LGTM with some nits, thanks @Michaelvll.

[concretevitamin]

Add something like:

Since setup commands always run on all nodes of a cluster, SkyPilot ensures both of these environment variables (the ranks and the IP list) never change across multiple setups on the same cluster.

What about after restarting a cluster?

[Michaelvll]

Good point! Added the sentence.

For restarted clusters, we only guarantee the NODE_RANK==0 should always be the head node, but the order of the other instances depend on the cloud's implementation for assigning the external IPs for the restarted nodes. We sort the nodes by the external IP, so if the external IPs change the node rank may be reordered for the workers:

skypilot/sky/backends/cloud_vm_ray_backend.py

Lines 2429 to 2430 in 30b39a6

	stable_internal_external_ips = [internal_external_ips[0]] + sorted(
	internal_external_ips[1:], key=lambda x: x[1])

I guess the order for the worker nodes should be fine, as long as we keep the head node to be the first one, as most heterogenity after restarting the cluster comes from the head node vs the workers, e.g., we only store the job state and skypilot logs on the head node.