Run xgboost using GPU Nvidia jetson nano error.

[lk1983823]

I install xgboost and want to use its GPU acceleration with nano. But I got error:

XGBoostError: [12:54:26] /home/lly2014/xgboost/src/learner.cc:180: XGBoost version not compiled with GPU support.

XGBoostError Traceback (most recent call last)
in

~/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/sklearn.py in fit(self, X, y, sample_weight, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, callbacks)
394 evals_result=evals_result, obj=obj, feval=feval,
395 verbose_eval=verbose, xgb_model=xgb_model,
--> 396 callbacks=callbacks)
397
398 if evals_result:

~/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/training.py in train(params, dtrain, num_boost_round, evals, obj, feval, maximize, early_stopping_rounds, evals_result, verbose_eval, xgb_model, callbacks, learning_rates)
214 evals=evals,
215 obj=obj, feval=feval,
--> 216 xgb_model=xgb_model, callbacks=callbacks)
217
218

~/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/training.py in _train_internal(params, dtrain, num_boost_round, evals, obj, feval, xgb_model, callbacks)
72 # Skip the first update if it is a recovery step.
73 if version % 2 == 0:
---> 74 bst.update(dtrain, i, obj)
75 bst.save_rabit_checkpoint()
76 version += 1

~/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/core.py in update(self, dtrain, iteration, fobj)
1113 if fobj is None:
1114 _check_call(_LIB.XGBoosterUpdateOneIter(self.handle, ctypes.c_int(iteration),
-> 1115 dtrain.handle))
1116 else:
1117 pred = self.predict(dtrain)

~/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/core.py in _check_call(ret)
176 """
177 if ret != 0:
--> 178 raise XGBoostError(py_str(_LIB.XGBGetLastError()))
179
180

XGBoostError: [12:54:26] /home/lly2014/xgboost/src/learner.cc:180: XGBoost version not compiled with GPU support.

Stack trace returned 10 entries:
[bt] (0) /home/lly2014/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/./lib/libxgboost.so(dmlc::StackTrace[abi:cxx11](unsigned long)+0x84) [0x7f2dcf4884]
[bt] (1) /home/lly2014/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/./lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x48) [0x7f2dcf5060]
[bt] (2) /home/lly2014/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/./lib/libxgboost.so(xgboost::LearnerImpl::AssertGPUSupport()+0x54) [0x7f2dd7486c]
[bt] (3) /home/lly2014/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/./lib/libxgboost.so(xgboost::LearnerImpl::ConfigureUpdaters()+0x2d0) [0x7f2dd794e8]
[bt] (4) /home/lly2014/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/./lib/libxgboost.so(xgboost::LearnerImpl::Configure(std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&)+0x4b0) [0x7f2dd7b600]
[bt] (5) /home/lly2014/.local/lib/python3.6/site-packages/xgboost-0.83.dev0-py3.6.egg/xgboost/./lib/libxgboost.so(XGBoosterUpdateOneIter+0xc8) [0x7f2dcedda8]
[bt] (6) /usr/lib/aarch64-linux-gnu/libffi.so.6(ffi_call_SYSV+0x64) [0x7f94776d28]
[bt] (7) /usr/lib/aarch64-linux-gnu/libffi.so.6(ffi_call+0xc8) [0x7f94777698]
[bt] (8) /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-aarch64-linux-gnu.so(_ctypes_callproc+0x41c) [0x7f9479be04]
[bt] (9) /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-aarch64-linux-gnu.so(+0x9178) [0x7f94793178]

Anyone can help me to fixted ? Thanks. Here is a brief introduction of nano:
https://devblogs.nvidia.com/jetson-nano-ai-computing/

[trivialfis]

Would you please share the used cmake compilation flags?

[lk1983823]

I use this command : cmake .. -DPLUGIN_UPDATER_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-10.0 -lpthreads=/usr/include ..

-- Found OpenMP_C: -fopenmp
-- Found OpenMP_CXX: -fopenmp
-- Found OpenMP: TRUE
-- /home/lly2014/xgboost/dmlc-core/cmake/build_config.h.in -> /home/lly2014/xgboost/dmlc-core/include/dmlc/build_config.h
-- Found OpenMP_C: -fopenmp
-- Found OpenMP_CXX: -fopenmp
-- /home/lly2014/xgboost/cmake/build_config.h.in -> /home/lly2014/xgboost/include/xgboost/build_config.h
-- Configuring done
-- Generating done
-- Build files have been written to: /home/lly2014/xgboost/build

[lk1983823]

I rebuild it again and found a warning:

-- The C compiler identification is GNU 7.3.0
-- The CXX compiler identification is GNU 7.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Setting build type to 'Release' as none was specified.
-- Performing Test XGBOOST_MM_PREFETCH_PRESENT
-- Performing Test XGBOOST_MM_PREFETCH_PRESENT - Failed
-- Performing Test XGBOOST_BUILTIN_PREFETCH_PRESENT
-- Performing Test XGBOOST_BUILTIN_PREFETCH_PRESENT - Success
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Looking for fopen64
-- Looking for fopen64 - not found
-- Looking for C++ include cxxabi.h
-- Looking for C++ include cxxabi.h - found
-- Looking for execinfo.h
-- Looking for execinfo.h - found
-- Looking for nanosleep
-- Looking for nanosleep - found
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- /home/lly2014/xgboost/dmlc-core/cmake/build_config.h.in -> /home/lly2014/xgboost/dmlc-core/include/dmlc/build_config.h
-- Performing Test SUPPORT_CXX11
-- Performing Test SUPPORT_CXX11 - Success
-- Performing Test SUPPORT_CXX0X
-- Performing Test SUPPORT_CXX0X - Success
-- Performing Test SUPPORT_MSSE2
-- Performing Test SUPPORT_MSSE2 - Failed
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- /home/lly2014/xgboost/cmake/build_config.h.in -> /home/lly2014/xgboost/include/xgboost/build_config.h
-- Configuring done
-- Generating done
CMake Warning:
Manually-specified variables were not used by the project:

CUDA_TOOLKIT_ROOT_DIR
PLUGIN_UPDATER_GPU

-- Build files have been written to: /home/lly2014/xgboost/build

[hcho3]

Try setting USE_CUDA=ON

[lk1983823]

@hcho3 thanks it works. But I will still check if my code can work as well.:)
.-- Found OpenMP_C: -fopenmp
-- Found OpenMP_CXX: -fopenmp
-- /home/lly2014/xgboost/dmlc-core/cmake/build_config.h.in -> /home/lly2014/xgboost/dmlc-core/include/dmlc/build_config.h
-- Found OpenMP_C: -fopenmp
-- Found OpenMP_CXX: -fopenmp
-- /home/lly2014/xgboost/cmake/build_config.h.in -> /home/lly2014/xgboost/include/xgboost/build_config.h
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda-10.0 (found suitable version "10.0", minimum required is "8.0")
cuda architecture flags: -gencode arch=compute_35,code=sm_35;-gencode arch=compute_50,code=sm_50;-gencode arch=compute_52,code=sm_52;-gencode arch=compute_60,code=sm_60;-gencode arch=compute_61,code=sm_61;-gencode arch=compute_70,code=sm_70;-gencode arch=compute_75,code=sm_75;-gencode arch=compute_75,code=compute_75;
-- Configuring done
-- Generating done
-- Build files have been written to: /home/lly2014/xgboost/build

[lk1983823]

@hcho3 Unfortunately , when I run my code, after running model.fit(), it shows :
terminate called after throwing an instance of 'thrust::system::system_error'
what(): parallel_for failed: no kernel image is available for execution on the device
Aborted (core dumped)

[trivialfis]

@lk1983823 A quick search didn't get me information about compute capability of that GPU. The error message from you is most likely due to compiling cuda code for wrong architecture.

[lk1983823]

@trivialfis Yes, the architecture of nano is Maxwell. So does xgboost supoort this architecture?

[trivialfis]

XGBoost supports CUDA, as long as it's recent CUDA version we support it. But NVidia cards have many different compute capabilities, see :https://developer.nvidia.com/cuda-gpus . We need to pass the correct flags to CUDA compiler nvcc.

[lk1983823]

@trivialfis My card is CUDA-Enabled TEGRA /Jetson Products, so what should I do t pass the correct flags?

[trivialfis]

@lk1983823 I couldn't find the one for nano. It's specific to each card, for example, the above link says that Jetson TX2 is of compute 6.2, then you need to use the following flags for Jetson TX2:

cmake -DUSE_CUDA=ON -DUSE_OPENMP=ON -DGPU_COMPUTE_VER=62 /path/to/xgboost
make

Replace -DGPU_COMPUTE_VER=62 with the correct one for nano, and if you found its info, please share it with us. We have a list for commonly used architectures and build for them by default, but Jetson series are not in the list.

[lk1983823]

Compute Capability for Nvidia Jetson Nano is 5.3.
cmake -DUSE_CUDA=ON -DUSE_OPENMP=ON -DGPU_COMPUTE_VER=53 /path/to/xgboost
it works! Thank you trivialfis!

This Nvidia Jetson Nano is bought for my daughter 路来仪 to let her know AI and this fantastic Algorithm. Wish her happy every day. I love you !

[RAMitchell]

Another tip: train with the parameter 'single_precision_histogram': True and see if it affects performance. I believe maxwell architecture has terrible performance with double precision atomics. Let us known how it goes!

[lk1983823]

Hi, I did a test as to the performance of Nano with double precision atomices vs. single_precision_histogram.
I tested two sets of training data with different rows and same columns, both of which are tested for 3 times.
Here are the results:
(1) x_train = np.random.rand(10000,3),y_train = np.random.rand(10000,1)
time cost: 'single_precision_histogram': True = 2.48/2.71/2.62s; 'single_precision_histogram': False = 3.76/2.93/2.73 seconds
(2) x_train = np.random.rand(100000,3),y_train = np.random.rand(100000,1)
time cost: 'single_precision_histogram': True = 3.15/3.13/3.24 seconds; 'single_precision_histogram': False = 3.52/3.39/3.56 seconds.
When I expand my trainning set to 1,000,000 rows, my Nano automaticly shutdown. I think this maybe due to the limited memory. I tried 3 times with the same results.
Hope this can be helpful.

This Nvidia Jetson Nano is bought for my daughter 路来仪 to let her know AI and this fantastic Algorithm. Wish her happy every day. I love you !

[trivialfis]

@lk1983823 Nice! Thanks for sharing. I'm closing this for now. Feel free to open a new issue for further finding/bugs ...