Skip to content

Conversation

kunalb
Copy link

@kunalb kunalb commented Feb 10, 2024

I was trying to build PyTorch with USE_GLOG=ON (so we could get better timestamps around the nccl logging) and ran into this error

[1/7] Linking CXX executable bin/verify_api_visibility
FAILED: bin/verify_api_visibility
: && /opt/rh/gcc-toolset-11/root/usr/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O2 -g -DNDEBUG -rdynamic     -Wl,--no-as-needed caffe2/CMakeFiles/verify_api_visibility.dir/__/aten/src/ATen/test/verify_api_visibility.cpp.o -o bin/verify_api_visibility -L/lib/intel64   -L/lib/intel64_win   -L/lib/win-x64 -Wl,-rpath,/lib/intel64:/lib/intel64_win:/lib/win-x64:/usr/local/cuda/lib64:/root/conda/lib:/mnt/code/pytorch/build/lib:  lib/libgtest_main.a  -Wl,--no-as-needed,"/mnt/code/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/mnt/code/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /root/conda/lib/libmkl_intel_lp64.so  /root/conda/lib/libmkl_gnu_thread.so  /root/conda/lib/libmkl_core.so  -fopenmp  /usr/lib64/libpthread.so  -lm  /usr/lib64/libdl.so  -Wl,--no-as-needed,"/mnt/code/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /root/conda/lib/libglog.so.0.4.0  /root/conda/lib/libgflags.so.2.2.2  -lpthread  /usr/local/cuda/lib64/libcudart.so  /usr/local/cuda/lib64/libnvToolsExt.so  lib/libgtest.a  -pthread && /root/conda/bin/cmake -E __run_co_compile --lwyu="ldd;-u;-r" --source=bin/verify_api_visibility && :
/opt/rh/gcc-toolset-11/root/usr/bin/ld: /mnt/code/pytorch/build/lib/libtorch.so: undefined reference to symbol '_ZTVN10__cxxabiv117__class_type_infoE@@CXXABI_1.3'
/opt/rh/gcc-toolset-11/root/usr/bin/ld: /usr/lib64/libstdc++.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

Adding stdc++ explicitly to the list of libraries to link seems to fix the build, and I was able to get a working build of PyTorch.

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Feb 10, 2024
Copy link

pytorch-bot bot commented Feb 10, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/119643

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit ee14203 with merge base c4d24b5 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@kunalb kunalb requested a review from malfet February 10, 2024 20:16
@kunalb kunalb requested review from wconstab and zdevito February 20, 2024 11:50
@kunalb
Copy link
Author

kunalb commented Feb 24, 2024

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Feb 24, 2024
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

Failing merge rule: Core Maintainers

@kunalb
Copy link
Author

kunalb commented Feb 24, 2024

@pytorchmergebot merge --rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased kunalb-patch-1 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout kunalb-patch-1 && git pull --rebase)

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 2 jobs have failed, first few of them are: trunk / win-vs2019-cuda11.8-py3 / build, trunk / win-vs2019-cpu-py3 / build

Details for Dev Infra team Raised by workflow job

@malfet
Copy link
Contributor

malfet commented Feb 25, 2024

Added a guard against not-windows (stdc++ is a very linux library), but this change feels very suspicious to me as dependency to C++ runtime should be added by automatically for C++ projects, and libtorch is one of them

@cyyever
Copy link
Collaborator

cyyever commented Feb 25, 2024

I suspect that gcc-11 was used in an old centOS system so it has the path '/opt/rh/gcc-toolset-11/', and the system's default libstdc++ may have older ABI. May be the proper libstdc++ with /opt/rh/gcc-toolset-11/ should be installed and CMake should be able to detect it. (By using some kind of environment variable?)

PS: A quick Google search gave pkg names like "gcc-toolset-11-libstdc++-devel-11.2.1-9.1.el8.i686.rpm", I think this is an issue of incomplete toolchain setting, use

sudo yum install gcc-toolset-11-libstdc++-devel -y 

and re-run CMake may solve the problem.

@kunalb
Copy link
Author

kunalb commented Feb 25, 2024

Hm, I was using a full toolchain installed into centos 8 (gcc-toolset-11-11.1-1.el8.x86_64), then run with scl enable gcc-toolset-11 to have everything pointed to the right location.

I'll double check

@malfet
Copy link
Contributor

malfet commented Feb 25, 2024

@kunalb I don't think there is a big harm in adding libstdc++ as a dependency, so feel free to land it (windows builds are passing now and runtime failure is unrelated), but again this dependency feels a bit superfluous though not wrong

@kunalb
Copy link
Author

kunalb commented Feb 26, 2024

Thanks, I'll go ahead and land it in that case and then keep looking on the side.

@pytorchmergebot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

Failing merge rule: Core Maintainers

@kunalb
Copy link
Author

kunalb commented Mar 18, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions bot deleted the kunalb-patch-1 branch April 18, 2024 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged topic: not user facing topic category

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants