[AIR][Train] Add default network setting for pytorch multi-host training

[jiaodong]

What happened + What you expected to happen

Currently user using AIR to do pytorch distributed training across multiple hosts will run into ganging issue if ens was not set or not set correctly. This hurts dev UX.

In https://github.com/inkawhich/pt-distributed-tutorial/blob/master/pytorch-aws-distributed-tutorial.py it mentioned about using ipconfig to find and set right interface with env var NCCL_SOCKET_IFNAME

ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.31.65.108  netmask 255.255.224.0  broadcast 172.31.95.255
        inet6 fe80::865:1bff:fe4c:2e03  prefixlen 64  scopeid 0x20<link>
        ether 0a:65:1b:4c:2e:03  txqueuelen 1000  (Ethernet)
        RX packets 12552852  bytes 18852483513 (18.8 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4753080  bytes 6866629048 (6.8 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

    runtime_env = {"env_vars": {"NCCL_SOCKET_IFNAME": "ens3"}}
    ray.init(runtime_env=runtime_env)

NOTE: Different hardware might require different values. g3 uses ens3 while g4dn uses ens5.

Versions / Dependencies

master

Reproduction script

https://sourcegraph.com/github.com/ray-project/ray/-/blob/release/air_tests/air_benchmarks/workloads/pytorch_training_e2e.py

Running pytorch_training_e2e.py with works across multiple hosts

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[matthewdeng]

It would be cool to find a way to exhaustively codify this 🤔

We could also add this to the FAQ as a stopgap, but I think the hard part for the user is identifying this issue in the first place...

cc @krfricke @amogkam

[jiaodong]

Yes this is very confusing dev UX that both Richard and I got trapped into while doing CUJs .. i can't imagine how our users gonna figure this out easily. @amogkam do we have any current mechanism in Ray Train to set this automatically -- how users resolve this in the past ?

[amogkam]

In general, I believe NCCL should automatically be doing this. But agreed we should do this programmatically for safety. This has only surfaced as a problem in Anyscale though, specifically with anyscale_default_cloud, and thus so far has only affected release tests and not any users.

[matthewdeng]

@amogkam can you just add this as a Train FAQ?

[xwjiang2010]

cc @ilee300a FYI