Compile using C++17 when using PyTorch 2.1

[thomas-bouvier]

Checklist before submitting

• Did you read the contributor guide?
• Did you update the docs?
• Did you write any tests to validate this change?
• Did you update the CHANGELOG, if this change affects users?

Description

The minimum required C++ version for compiling PyTorch 2.1 is C++17 (pytorch/pytorch#100557). This commit updates this requirement so that Horovod will also require C++17 if PyTorch >= 2.1.0.

Fixes #3996.

If you validate this change, I'll update the CHANGELOG accordingly.

Review process to land

1. All tests and other checks must succeed.
2. At least one member of the technical steering committee must review and approve.
3. If any member of the technical steering committee requests changes, they must be addressed.

[github-actions]

Unit Test Results

   724 files  ±0     724 suites  ±0   7h 58m 38s ⏱️ - 7m 29s
   887 tests ±0     768 ✅ ±0    119 💤 ±0  0 ❌ ±0 
16 209 runs  ±0  11 353 ✅ ±0  4 856 💤 ±0  0 ❌ ±0 

Results for commit 215b1f7. ± Comparison against base commit 9f88e1d.

♻️ This comment has been updated with latest results.

[github-actions]

Unit Test Results (with flaky tests)

   838 files   -    50     838 suites   - 50   8h 32m 33s ⏱️ - 26m 8s
   887 tests ±    0     768 ✅ ±  0    119 💤 ±  0  0 ❌ ±0 
19 101 runs   - 1 138  13 045 ✅  - 744  6 056 💤  - 394  0 ❌ ±0 

Results for commit 215b1f7. ± Comparison against base commit 9f88e1d.

♻️ This comment has been updated with latest results.

[thomas-bouvier]

Pinging @maxhgerlach to get some feedback, as I think this PR is quite important.

[jiwq]

+1
And should we had plan to use C++17 instead of C++14 as the minimum required C++ standard?

[wenfeiy-db]

Hi @thomas-bouvier @jiwq, can we prioritize merging this to fix #3996?

[yiliu30]

Hi @maxhgerlach @EnricoMi @jiwq , #3996 seems to block all users with PT >= 2.1. Some CIs failed due to No space left on device, could you please take a look? Thanks

[EnricoMi]

Has someone tested this? The installation still fails, though on a different issue:

/usr/bin/ld: ../../third_party/compatible_gloo/gloo/libcompatible_gloo.a(store.cc.o):(.rodata+0x0): multiple definition of `gloo::rendezvous::Store::kDefaultTimeout'; CMakeFiles/pytorch.dir/__/common/gloo/http_store.cc.o:(.rodata._ZN4gloo10rendezvous5Store15kDefaultTimeoutE[_ZN4gloo10rendezvous5Store15kDefaultTimeoutE]+0x0): first defined here

Tested with

 # Use the official Python image as the base image
 FROM python:3.10-slim-buster
 
 # Update the system and install necessary libraries
 RUN apt-get update && apt-get install -y \
     build-essential \
     cmake \
     git \
     curl \
     ca-certificates \
     libjpeg-dev \
     libpng-dev && \
     rm -rf /var/lib/apt/lists/*
 
 # Install PyTorch
 RUN pip install torch==2.1.0
 
 # Install Horovod
-RUN HOROVOD_WITH_PYTORCH=1 pip install horovod==0.28.1
+RUN HOROVOD_WITH_PYTORCH=1 pip install git+https://github.com/thomas-bouvier/horovod.git@compile-cpp17

[EnricoMi]

Looks like we have to build Gloo with C++17 as well: pytorch/pytorch#90448 (comment)

[EnricoMi]

Looks like there is some related code in CMakeLists.txt:

horovod/CMakeLists.txt

Lines 377 to 393 in 7e4d993

	# Gloo for c++17 TF
	if(HAVE_GLOO AND (DEFINED Tensorflow_CXX17 AND Tensorflow_CXX17))
	file(COPY ${PROJECT_SOURCE_DIR}/third_party/gloo/ DESTINATION ${PROJECT_SOURCE_DIR}/third_party/compatible17_gloo)
	file(READ ${PROJECT_SOURCE_DIR}/third_party/compatible17_gloo/gloo/CMakeLists.txt GLOO_CMAKE)
	string(REPLACE "gloo " "compatible17_gloo " GLOO_CMAKE "${GLOO_CMAKE}")
	file(WRITE ${PROJECT_SOURCE_DIR}/third_party/compatible17_gloo/gloo/CMakeLists.txt "${GLOO_CMAKE}")
	file(READ ${PROJECT_SOURCE_DIR}/third_party/compatible17_gloo/CMakeLists.txt GLOO_CMAKE)
	string(REPLACE "-std=c++11" "-std=c++17" GLOO_CMAKE "${GLOO_CMAKE}")
	string(PREPEND GLOO_CMAKE "set(CMAKE_CXX_STANDARD 17)\n")
	file(WRITE ${PROJECT_SOURCE_DIR}/third_party/compatible17_gloo/CMakeLists.txt "${GLOO_CMAKE}")
	add_subdirectory(third_party/compatible17_gloo)
	if (Tensorflow_CXX11)
	target_compile_definitions(compatible17_gloo PRIVATE _GLIBCXX_USE_CXX11_ABI=1)
	else()
	target_compile_definitions(compatible17_gloo PRIVATE _GLIBCXX_USE_CXX11_ABI=0)
	endif()
	endif()

@maxhgerlach any suggestions?

[EnricoMi]

There is some similar logic for tensorflow, which I have mimicked for Pytorch here: 9f7722d

[jiwq]

I like the latest patch. Thanks @EnricoMi catch the important issue.

[EnricoMi]

Any suggestions how to debug why horovod.torch.init() throws an std::bad_alloc when compiled with C++-17 and PyTorch 2.1.x? #3979 (comment)

This can be tested by rebasing with master.

[thomas-bouvier]

I guess #3979 needs to be merged to test against PyTorch 2.1. I will have a look at the bad alloc.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[heliar-k]

Any progress of this PR?

[thomas-bouvier]

Unfortunately, no progress so far. The blocker is a bad allocation std::bad_allocerror mentioned by @EnricoMi above. Feel free to investigate #3979 (comment) if you have some time.