[Train] Change default `NCCL_SOCKET_IFNAME` to blacklist `veth` #31824

amogkam · 2023-01-21T00:22:32Z

Signed-off-by: amogkam amogkamsetty@yahoo.com

Previously, we would set a default NCCL interface whitelist in Ray Train to prioritize ethernet. This is to avoid this issue: https://github.com/anyscale/product/issues/8310.

However, this default whitelist is not fully exhaustive, and prevents users from doing distributed GPU training over wireless: #30333.

Instead, we change to a blacklist so that NCCL does not use veth interface which resolves both issues (thanks @cadedaniel for identifying this!)

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

amogkam · 2023-01-21T01:56:26Z

Passing distributed GPU training test on anyscale_default_cloud: https://buildkite.com/ray-project/release-tests-pr/builds/26067

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

cadedaniel · 2023-01-21T02:18:35Z

nice!

From the logs, looks like NCCL is still seeing veth in some of the nodes. Any idea why?

I see 12 worker logs with veth, but all 16 have ens3

$ ag --case-sensitive --literal 'Using [0]ens' | wc -l
      16

$ ag --case-sensitive 'veth' | wc -l
      12

:job_id:03000000
:actor_name:RayTrainWorker
ip-172-31-201-206:7411:7871 [0] NCCL INFO cudaDriverVersion 11060
ip-172-31-201-206:7411:7871 [0] NCCL INFO Bootstrap : Using ens3:172.31.201.206<0>
ip-172-31-201-206:7411:7871 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ip-172-31-201-206:7411:7885 [0] NCCL INFO Failed to open libibverbs.so[.1]
ip-172-31-201-206:7411:7885 [0] NCCL INFO NET/Socket : Using [0]ens3:172.31.201.206<0> [1]veth01fa248:fe80::343a:e3ff:fe09:aeff%veth01fa248<0>
ip-172-31-201-206:7411:7885 [0] NCCL INFO Using network Socket
ip-172-31-201-206:7411:7885 [0] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1
ip-172-31-201-206:7411:7885 [0] NCCL INFO Channel 00 : 2[1d0] -> 3[1e0] via SHM/direct/direct
ip-172-31-201-206:7411:7885 [0] NCCL INFO Channel 01 : 2[1d0] -> 3[1e0] via SHM/direct/direct
ip-172-31-201-206:7411:7885 [0] NCCL INFO Connected all rings
ip-172-31-201-206:7411:7885 [0] NCCL INFO Channel 00 : 2[1d0] -> 1[1c0] via SHM/direct/direct
ip-172-31-201-206:7411:7885 [0] NCCL INFO Channel 01 : 2[1d0] -> 1[1c0] via SHM/direct/direct
ip-172-31-201-206:7411:7885 [0] NCCL INFO Connected all trees
ip-172-31-201-206:7411:7885 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
ip-172-31-201-206:7411:7885 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
ip-172-31-201-206:7411:7885 [0] NCCL INFO comm 0x7f4cd4008ec0 rank 2 nranks 16 cudaDev 0 busId 1d0 - Init COMPLETE
ip-172-31-201-206:7411:7887 [0] NCCL INFO [Service thread] Connection closed by localRank 2
ip-172-31-201-206:7411:7411 [0] NCCL INFO comm 0x7f4cd4008ec0 rank 2 nranks 16 cudaDev 0 busId 1d0 - Abort COMPLETE

cadedaniel

discussed offline. we're not sure if that line is printed before or after blacklist. we should catch it in nightly if the order is nondeterministic.

can we file an issue against product to remove the virtual ethernet interface? then Ray Train can get out of the way of users completely here (default to NCCL behavior).

cadedaniel · 2023-01-21T02:28:16Z

python/ray/train/constants.py

-# "en".
-DEFAULT_NCCL_SOCKET_IFNAME = "en,eth,bond"
+# Blacklist virtualized networking.
+DEFAULT_NCCL_SOCKET_IFNAME = "^lo,docker,vethc"


oh actually the reason is that it should be veth not vethc. c happened to be the first hex character of the ID I sent you.

that also explains why it was present in only 12 or the 16 -- the other 4 must have had c as the first character of the id.

Ah good catch...then now I'm wondering why this passed. Maybe this is not needed at all 🤔. Let me run the test with this default removed.

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

amogkam · 2023-01-23T20:45:55Z

fails without it: https://buildkite.com/ray-project/release-tests-pr/builds/26088#0185d2ae-4fae-47ee-b8f6-671cdc492ece

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

amogkam · 2023-01-23T22:32:47Z

test is passing here: https://buildkite.com/ray-project/release-tests-pr/builds/26227#0185e07f-407c-4067-bc3f-40da248cd547

cadedaniel

Can we create an issue against product to remove the virtual network interface?

krfricke

Quick question, the title says we blacklist vethc - instead we ~~whitelist~~ veth. Is that what we want? Edit: Sorry misread the line

amogkam · 2023-01-23T23:27:56Z

Updated the title-- it should be veth

krfricke

Thanks!

@cadedaniel

…project#31824) Signed-off-by: amogkam amogkamsetty@yahoo.com Closes ray-project#30333. Previously, we would set a default NCCL interface whitelist in Ray Train to prioritize ethernet. This is to avoid this issue: anyscale/product#8310. However, this default whitelist is not fully exhaustive, and prevents users from doing distributed GPU training over wireless: ray-project#30333. Instead, we change to a blacklist so that NCCL does not use veth interface which resolves both issues (thanks @cadedaniel for identifying this!) Signed-off-by: amogkam <amogkamsetty@yahoo.com>

@cadedaniel

…project#31824) Signed-off-by: amogkam amogkamsetty@yahoo.com Closes ray-project#30333. Previously, we would set a default NCCL interface whitelist in Ray Train to prioritize ethernet. This is to avoid this issue: anyscale/product#8310. However, this default whitelist is not fully exhaustive, and prevents users from doing distributed GPU training over wireless: ray-project#30333. Instead, we change to a blacklist so that NCCL does not use veth interface which resolves both issues (thanks @cadedaniel for identifying this!) Signed-off-by: amogkam <amogkamsetty@yahoo.com>

When running distributed Pytorch without GPUs, Pytorch selects a localhost interface for gloo (i.e. 127.0.0.1:XXX), breaking distributed training. This method in Pytorch can yield the incorrect interface when a) the the hostname resolves locally to the loopback address or b) when hostname lookups fail. This is scoped to DBR specifically because eth0 is guaranteed to exist there. Pytorch+Gloo does not support deny-listing like NCCL (as we do in #31824) because Pytorch directly uses the environment variable GLOO_SOCKET_IFNAME as the interface to use https://github.com/pytorch/pytorch/blob/7956ca16e649d86cbf11b6e122090fa05678fac3/torch/csrc/distributed/c10d/init.cpp#L2243. Signed-off-by: Ian Rodney <ian.rodney@gmail.com>

…2202) When running distributed Pytorch without GPUs, Pytorch selects a localhost interface for gloo (i.e. 127.0.0.1:XXX), breaking distributed training. This method in Pytorch can yield the incorrect interface when a) the the hostname resolves locally to the loopback address or b) when hostname lookups fail. This is scoped to DBR specifically because eth0 is guaranteed to exist there. Pytorch+Gloo does not support deny-listing like NCCL (as we do in ray-project#31824) because Pytorch directly uses the environment variable GLOO_SOCKET_IFNAME as the interface to use https://github.com/pytorch/pytorch/blob/7956ca16e649d86cbf11b6e122090fa05678fac3/torch/csrc/distributed/c10d/init.cpp#L2243. Signed-off-by: Ian Rodney <ian.rodney@gmail.com>

amogkam added 4 commits January 20, 2023 16:22

change

88d02c8

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

add

49b4093

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

Merge branch 'master' of github.com:ray-project/ray into default-nccl

5f0fdc5

remove

ba276ee

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

amogkam marked this pull request as ready for review January 21, 2023 01:56

amogkam assigned cadedaniel, matthewdeng and krfricke Jan 21, 2023

comment

25210ff

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

cadedaniel approved these changes Jan 21, 2023

View reviewed changes

cadedaniel requested changes Jan 21, 2023

View reviewed changes

remove vethc

2c7f721

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

amogkam added 2 commits January 23, 2023 12:47

no veth

3b23dcb

Signed-off-by: amogkam <amogkamsetty@yahoo.com>

Merge branch 'master' of github.com:ray-project/ray into default-nccl

79c7b31

amogkam requested a review from cadedaniel January 23, 2023 22:32

cadedaniel approved these changes Jan 23, 2023

View reviewed changes

krfricke reviewed Jan 23, 2023

View reviewed changes

amogkam changed the title ~~[Train] Change default NCCL_SOCKET_IFNAME to blacklist vethc~~ [Train] Change default NCCL_SOCKET_IFNAME to blacklist veth Jan 23, 2023

krfricke approved these changes Jan 23, 2023

View reviewed changes

amogkam merged commit 9bebf57 into ray-project:master Jan 24, 2023

amogkam deleted the default-nccl branch January 24, 2023 01:04

ijrsvt mentioned this pull request Jan 5, 2024

[spark] Fix Gloo detecting incorrect Interfaces on DBR #42202

Merged

8 tasks

matthewdeng mentioned this pull request Jan 31, 2024

[train] remove DEFAULT_NCCL_SOCKET_IFNAME #42808

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Train] Change default `NCCL_SOCKET_IFNAME` to blacklist `veth` #31824

[Train] Change default `NCCL_SOCKET_IFNAME` to blacklist `veth` #31824

amogkam commented Jan 21, 2023 •

edited

Loading

amogkam commented Jan 21, 2023 •

edited

Loading

cadedaniel commented Jan 21, 2023

cadedaniel left a comment

cadedaniel Jan 21, 2023

amogkam Jan 21, 2023

amogkam commented Jan 23, 2023

amogkam commented Jan 23, 2023

cadedaniel left a comment

krfricke left a comment •

edited

Loading

amogkam commented Jan 23, 2023

krfricke left a comment

[Train] Change default NCCL_SOCKET_IFNAME to blacklist veth #31824

[Train] Change default NCCL_SOCKET_IFNAME to blacklist veth #31824

Conversation

amogkam commented Jan 21, 2023 • edited Loading

Why are these changes needed?

Related issue number

Checks

amogkam commented Jan 21, 2023 • edited Loading

cadedaniel commented Jan 21, 2023

cadedaniel left a comment

Choose a reason for hiding this comment

cadedaniel Jan 21, 2023

Choose a reason for hiding this comment

amogkam Jan 21, 2023

Choose a reason for hiding this comment

amogkam commented Jan 23, 2023

amogkam commented Jan 23, 2023

cadedaniel left a comment

Choose a reason for hiding this comment

krfricke left a comment • edited Loading

Choose a reason for hiding this comment

amogkam commented Jan 23, 2023

krfricke left a comment

Choose a reason for hiding this comment

[Train] Change default `NCCL_SOCKET_IFNAME` to blacklist `veth` #31824

[Train] Change default `NCCL_SOCKET_IFNAME` to blacklist `veth` #31824

amogkam commented Jan 21, 2023 •

edited

Loading

amogkam commented Jan 21, 2023 •

edited

Loading

krfricke left a comment •

edited

Loading