Check PyTorch supports TCP_TLS gloo transport

[pbelevich]

Depends on pytorch/pytorch#58996

[zhouzhuojie]

LGTM, btw, is pytorch/pytorch#58996 going into 1.9?

[pbelevich]

LGTM, btw, is pytorch/pytorch#58996 going into 1.9?

It seems that it's already included

[seemethere]

Does the wheel building script also have the USE_GLOO_WITH_OPENSLL flag as well?

[seemethere]

I think to be safe with this it'd be good to have a test PR upstream with pytorch/pytorch to verify this does not introduce a regression in the nightlies

[seemethere]

I think to be safe with this it'd be good to have a test PR upstream with pytorch/pytorch to verify this does not introduce a regression in the nightlies

Testing PR can be found here: pytorch/pytorch#59306

Looks like in spite of the environment variable being present for the conda build the test still fails:

• Build Logs: https://app.circleci.com/pipelines/github/pytorch/pytorch/329915/workflows/120b1885-5424-4adf-866f-60e83d1b794d/jobs/13842464

+ [[ Linux == \L\i\n\u\x ]]
+ export USE_GLOO_WITH_OPENSSL=1
+ USE_GLOO_WITH_OPENSSL=1

• Test Logs: https://app.circleci.com/pipelines/github/pytorch/pytorch/329915/workflows/120b1885-5424-4adf-866f-60e83d1b794d/jobs/13843014

Jun 02 16:25:42 + grep 'unsupported gloo device'
Jun 02 16:25:42 + GLOO_DEVICE_TRANSPORT=TCP_TLS
Jun 02 16:25:42 + MASTER_ADDR=localhost
Jun 02 16:25:42 + MASTER_PORT=63945
Jun 02 16:25:42 + python -c 'import torch; import torch.distributed as dist; print(torch.__version__); dist.init_process_group('\''gloo'\'', rank=0, world_size=1)'
Jun 02 16:25:43 Traceback (most recent call last):
Jun 02 16:25:43   File "<string>", line 1, in <module>
Jun 02 16:25:43   File "/opt/conda/envs/testenv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 563, in init_process_group
Jun 02 16:25:43     default_pg = _new_process_group_helper(
Jun 02 16:25:43   File "/opt/conda/envs/testenv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 664, in _new_process_group_helper
Jun 02 16:25:43     pg = ProcessGroupGloo(
Jun 02 16:25:43 RuntimeError: makeDeviceForHostname(): unsupported gloo device

Doesn't appear as though USE_GLOO_WITH_OPENSSL was included in the Summary for the cmake as well which makes it hard to know if it was actually enabled or not, Created a PR to do this: pytorch/pytorch#59321

[seemethere]

Suggested change

	###############################################################################
	# Check PyTorch supports TCP_TLS gloo transport
	###############################################################################
	if [[ "$(uname)" == 'Linux' ]]; then
	GLOO_DEVICE_TRANSPORT=TCP_TLS MASTER_ADDR=localhost MASTER_PORT=63945 python -c "import torch; import torch.distributed as dist; print(torch.__version__); dist.init_process_group('gloo', rank=0, world_size=1)" | grep "unsupported gloo device" &> /dev/null
	RESULT=$?
	if [ $RESULT -eq 0 ]; then
	echo "PyTorch doesn't support TLS_TCP transport, please set USE_GLOO_WITH_OPENSSL=1"
	exit 1
	fi
	fi
	###############################################################################
	# Check PyTorch supports TCP_TLS gloo transport
	###############################################################################
	function built_without_gloo_tls() {
	GLOO_DEVICE_TRANSPORT=TCP_TLS MASTER_ADDR=localhost MASTER_PORT=63945 python -c "import torch; import torch.distributed as dist; print(torch.__version__); dist.init_process_group('gloo', rank=0, world_size=1)" | grep "unsupported gloo device" &> /dev/null
	}
	if [[ "$(uname)" == 'Linux' ]]; then
	if built_without_gloo_tls; then
	echo "PyTorch doesn't support TLS_TCP transport, please set USE_GLOO_WITH_OPENSSL=1"
	exit 1
	fi
	fi

[seemethere]

I think this should work, but I also wrote this suggestion in the github UI so it'd be good to double check that it checks out locally as well

[seemethere]

Looks like tests are passing in the tester PR for pytorch/pytorch, going to go ahead and merge!