[Feature] Support head nodes with SSH fleets

[peterschmidt85]

Currently, if you set up an SSH fleet, dstack requires that the dstack server can access each host via the provided SSH credentials. In many cases, compute hosts aren't accessible directly. Instead, they are accessible through so called "head" nodes.

Setup example:

1. A head-host that can access SSH into compute-host-1 and compute-host-2.
2. On a server node, run dstack server
3. Create a fleet config:

type: fleet
name: private-fleet

placement: cluster

ssh_config:
  hosts:
    - compute-host-1
    - compute-host-2
   proxy_jump: # also can be defined per host; (or it can be a hostname instead of an object)
     hostname: host-node
     identity_file: ~/.ssh/private_key

4. The dstack apply and dstack attach will try to reuse the jump host SSH config for the specified host (login-node) automatically.

It means the user must have head-node SSH config defined.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.

[timsolovev]

I might be reading the issue wrong, but it might potentially unlock the use of mpirun/mpiexec, which in some cases are a must-have for infrastructure benchmarking (e.g. nccl-tests). A critical requirement for mpirun is a passwordless SSH access from the head node to workers, so it seems to be adjacent to this opened issue.

Some additional docs on OpenMPI and remote shells: OpenMPI v4.* and OpenMPI v5.