Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build issue with GPU on Ubuntu 16.04 #458

Closed
txjmb opened this issue Apr 25, 2017 · 13 comments
Closed

Build issue with GPU on Ubuntu 16.04 #458

txjmb opened this issue Apr 25, 2017 · 13 comments

Comments

@txjmb
Copy link

txjmb commented Apr 25, 2017

I tried rebuilding today to get the latest updates with GPU support following the GPU tutorial, and I get this error when running make -j$(nproc):

[ 94%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 98%] Linking CXX executable ../lightgbm
[ 98%] Built target lightgbm
[100%] Linking CXX shared library ../lib_lightgbm.so
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libboost_filesystem.a(operations.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libboost_filesystem.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
CMakeFiles/_lightgbm.dir/build.make:695: recipe for target '../lib_lightgbm.so' failed
make[2]: *** [../lib_lightgbm.so] Error 1
CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/_lightgbm.dir/all' failed
make[1]: *** [CMakeFiles/_lightgbm.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

I have tried a couple of recommended fixes for similar errors, but have been unable to resolve this yet. I am on Ubuntu 16.04.

Please let me know if you need any further information.

@Laurae2
Copy link
Contributor

Laurae2 commented Apr 25, 2017

The solution to your problem is in the error message:

  • You have the boost filesystem library compiled with absolute addresses, so it is suitable for programs, not libraries
  • It tells you to recompile the boost filesystem library with -fPIC, because the compiler can't learn by itself how to translate absolute addresses into relative addresses (it is suitable for programs, not libraries)

When compiling boost with b2 or bjam, use cxxflags=-fPIC

@txjmb
Copy link
Author

txjmb commented Apr 25, 2017

Thank you. The boost library I am using is the default that installs with the Ubuntu package manager. I will custom compile it per your suggestion. I do think that there is value to the project in understanding what issues users might have compiling on different platforms with the default packages, however. At least we can update the documentation for particular default platform issues.

@huanzhang12
Copy link
Contributor

This problem is introduced by commit fb96b71.
@txjmb No, you don't need to recompile boost. The library installed by Ubuntu works. Just comment out the line set(Boost_USE_STATIC_LIBS ON) in CMakeList.txt, rm -rf build and rebuild

@huanzhang12
Copy link
Contributor

@guolinke I think we don't want static libs on Linux, which can cause compilation troubles like this one. Maybe we just add set(Boost_USE_STATIC_LIBS ON) for Windows?

@guolinke
Copy link
Collaborator

@huanzhang12 okay.

@guolinke
Copy link
Collaborator

@txjmb can you try the latest code ?

@txjmb
Copy link
Author

txjmb commented Apr 26, 2017

That fixed it. Thank you! Outstanding library with outstanding support! We are seeing very good results.

Michael

@txjmb txjmb closed this as completed Apr 26, 2017
@Ubikas
Copy link

Ubikas commented Jun 14, 2017

Hi @huanzhang12,

It looks like lightGBM doesn't want to let me go easily :) I run into a problem trying to run
sudo make -j

I successfully pass cmake with options
sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/include/ ..
what returns
-- The C compiler identification is GNU 5.4.0 -- The CXX compiler identification is GNU 5.4.0 -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info - done -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features - done -- Try OpenMP C flag = [-fopenmp] -- Performing Test OpenMP_FLAG_DETECTED - Success -- Try OpenMP CXX flag = [-fopenmp] -- Performing Test OpenMP_FLAG_DETECTED - Success -- Found OpenMP: -fopenmp -- Looking for CL_VERSION_2_0 - found -- Found OpenCL: /usr/local/cuda-8.0/lib64/libOpenCL.so (found version "2.0") -- OpenCL include directory:/usr/include/ -- Boost version: 1.58.0 -- Found the following Boost libraries: -- filesystem -- system -- Configuring done -- Generating done -- Build files have been written to: /usr/local/LightGBM/build
When I run 'sudo make -j' everything goes fine till 98% and then I receive few errors

[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o [ 98%] Linking CXX executable ../lightgbm CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o: In function LightGBM::GPUTreeLearner::InitGPU(int, int)':
gpu_tree_learner.cpp:(.text+0x834e): undefined reference to clCreateCommandQueueWithProperties' CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o: In function boost::compute::detail::meta_kernel::compile(boost::compute::context const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)':
gpu_tree_learner.cpp:(.text._ZN5boost7compute6detail11meta_kernel7compileERKNS0_7contextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN5boost7compute6detail11meta_kernel7compileERKNS0_7contextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x3b7): undefined reference to clSetKernelArgSVMPointer' collect2: error: ld returned 1 exit status CMakeFiles/lightgbm.dir/build.make:747: recipe for target '../lightgbm' failed make[2]: *** [../lightgbm] Error 1 CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/lightgbm.dir/all' failed make[1]: *** [CMakeFiles/lightgbm.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... [100%] Linking CXX shared library ../lib_lightgbm.so [100%] Built target _lightgbm Makefile:127: recipe for target 'all' failed make: *** [all] Error 2
What's more interesting, it creates package!
What should I do to make it work? Any Ideas?
Thanks in advance!
EDIT
If You have got CUDA stuff, you have to compile with cuda. At the moment I'm writing Nvidia supports OpenGL 1.2 so if cmake log shows 2.0 it means something wrong. This is what finally worked for me

sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..

@zachmayer
Copy link

zachmayer commented Oct 4, 2017

sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ .. fixes the error for me on ubuntu 14 with CUDA

@qifengzhou
Copy link

I got the same problem on installing the GPU version on Ubuntu.

'''
qifeng@DLS:~/LightGBM/build$ sudo make -j$(nproc)
Scanning dependencies of target lightgbm
Scanning dependencies of target _lightgbm
[ 1%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt.cpp.o
[ 3%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 4%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/boosting.cpp.o
[ 6%] Building CXX object CMakeFiles/lightgbm.dir/src/io/json11.cpp.o
[ 8%] Building CXX object CMakeFiles/lightgbm.dir/src/application/application.cpp.o
[ 9%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 11%] Building CXX object CMakeFiles/lightgbm.dir/src/io/parser.cpp.o
[ 13%] Building CXX object CMakeFiles/lightgbm.dir/src/io/bin.cpp.o
[ 14%] Building CXX object CMakeFiles/lightgbm.dir/src/main.cpp.o
[ 16%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 18%] Building CXX object CMakeFiles/lightgbm.dir/src/io/tree.cpp.o
[ 19%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
[ 21%] Building CXX object CMakeFiles/lightgbm.dir/src/io/dataset.cpp.o
[ 22%] Building CXX object CMakeFiles/_lightgbm.dir/src/lightgbm_R.cpp.o
[ 24%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.o
[ 26%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 27%] Building CXX object CMakeFiles/lightgbm.dir/src/io/file_io.cpp.o
[ 31%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[ 31%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 32%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 34%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[ 36%] Building CXX object CMakeFiles/lightgbm.dir/src/io/dataset_loader.cpp.o
[ 37%] Building CXX object CMakeFiles/lightgbm.dir/src/io/metadata.cpp.o
[ 39%] Building CXX object CMakeFiles/lightgbm.dir/src/io/config.cpp.o
[ 40%] Building CXX object CMakeFiles/lightgbm.dir/src/metric/metric.cpp.o
[ 44%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
[ 44%] Building CXX object CMakeFiles/lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 45%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
[ 47%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 49%] Building CXX object CMakeFiles/lightgbm.dir/src/objective/objective_function.cpp.o
[ 50%] Building CXX object CMakeFiles/lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 52%] Building CXX object CMakeFiles/lightgbm.dir/src/network/linkers_socket.cpp.o
[ 54%] Building CXX object CMakeFiles/lightgbm.dir/src/network/network.cpp.o
[ 55%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
[ 57%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
[ 59%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
[ 60%] Building CXX object CMakeFiles/lightgbm.dir/src/network/linker_topo.cpp.o
[ 62%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 63%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
[ 65%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
[ 67%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
[ 68%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 70%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 72%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 73%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
[ 75%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 77%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
[ 78%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 80%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
[ 81%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
[ 83%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 85%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
[ 86%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 88%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 90%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 91%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
make[2]: *** No rule to make target '/usr/local/cuda-8.0/lib64/libOpenCL.so', needed by '../lightgbm'. Stop.
make[2]: *** Waiting for unfinished jobs....
[ 93%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 95%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
make[2]: *** No rule to make target '/usr/local/cuda-8.0/lib64/libOpenCL.so', needed by '../lib_lightgbm.so'. Stop.
make[2]: *** Waiting for unfinished jobs....
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/lightgbm.dir/all' failed
make[1]: *** [CMakeFiles/lightgbm.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/_lightgbm.dir/all' failed
make[1]: *** [CMakeFiles/_lightgbm.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2
'''
I tried 'sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -', it returns this error:
'CMake Error: The source directory "/home/qifeng/LightGBM/build/-" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.'

I checked the folder, it does exist, what could be the problem?

@Ubikas
Copy link

Ubikas commented May 19, 2018

Hi @qifengzhou ,
I had to spend some time until I manged to collect all info to make it right. It's already been some time since my last lightgbm update run, hope script still valid :)

sudo pip uninstall lightgbm
sudo rm -r LightGBM
sudo rm -r lightgbm
sudo git clone --recursive https://github.com/Microsoft/LightGBM
cd LightGBM
sudo mkdir build ; cd build
sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..
sudo make -j$(nproc)
cd python-package
sudo python setup.py install --precompile

Mind catalogs according to your setup and cuda.

@StrikerRUS
Copy link
Collaborator

@Ubikas Thank you for sharing your experience!

I just want to note, that Python installation can take args with paths now. So, the script can be simplified:

sudo pip uninstall lightgbm
sudo pip install lightgbm --install-option=--gpu --install-option="--opencl-include-dir=/usr/local/cuda-8.0/include/" --install-option="--opencl-library=/usr/local/cuda-8.0/lib64/libOpenCL.so"

@qifengzhou
Copy link

qifengzhou commented May 21, 2018

Hi @Ubikas @StrikerRUS,

Thank you for the instruction!
I followed it and compiled it successfully. However, its still not working, I got an error when running LGB in jupyter notebook:
"[LightGBM] [Warning] Unknown parameter: tree_method
[LightGBM] [Warning] Find whitespaces in feature_names, replace with underlines
[LightGBM] [Warning] Unknown parameter: tree_method
[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build. Recompile with CMake option -DUSE_GPU=1"

@lock lock bot locked as resolved and limited conversation to collaborators Mar 12, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants