[BUG] CUDA and OpenCl do not coexist nicely

[eddelbuettel]

Package Version

0.1.11.1.9000

R Version

4.5.2

Operating System

Linux

Bug Description

I can convince configure to find both OpenCLand CUDA on my machine. There was one slight initial nag (mentioned in the matrix app / chat) that I need this diff

modified   configure.ac
@@ -295,7 +295,7 @@ if test "x${NVCC}" != xno; then
     LDFLAGS="${LDFLAGS} -L${CUDA_HOME}/lib64 -L${CUDA_HOME}/lib"
 
     AC_CHECK_LIB([cudart], [cudaMalloc],
-                 [AC_CHECK_LIB([cublas], [cublasCreate],
+                 [AC_CHECK_LIB([cublas], [cublasCreate_v2],
                   [AC_CHECK_LIB([cusolver], [cusolverDnCreate],
                    [AC_CHECK_LIB([curand], [curandCreateGenerator],
                     [AC_CHECK_LIB([nvrtc], [nvrtcCreateProgram],

but while 'all systems detected' happens I then get a silly linker error where seemingly one of the OpenCL symbols is longer found, for example undefined symbol: clBuildProgram (I had others too, and tried other things).

This is apparently not an issue with bandicoot itself as its cmake output is happy:

-- *** Result of configuration:                                                                                                                                                                                    
-- *** COOT_USE_WRAPPER       = true                                                                                                                                                                               
-- *** COOT_USE_OPENCL        = true                                                                     
-- *** COOT_USE_CLBLAST       = true                                                                     
-- *** COOT_USE_CLBLAS        = true                                                                                                                                                                               
-- *** COOT_USE_CUDA          = TRUE                                                                                                                                                                               
-- *** COOT_CUDA_INCLUDE_PATH = /usr/include
-- *** COOT_DEFAULT_BACKEND   = CUDA_BACKEND

I have not really had time to dig deeper and just thought I check in with you. Do you have CUDA and/or CUDA+OpenCL working?

Steps to Reproduce

See above

Expected Behavior

See above

Session Info

> sessionInfo()
R version 4.5.2 (2025-10-31)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 25.04

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.12.1 
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.1;  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8      
 [8] LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: America/Chicago
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.5.2 parallel_4.5.2 colorout_1.3-3 fortunes_1.5-4
> 

[coatless]

Do you have CUDA and/or CUDA+OpenCL working?

Just OpenCL is working at the moment for both Linux + macOS. (Sent an invite over for repo access so you can see the GitHub Actions output).

I need a CUDA equipped machine to go further. Unless can you share the linker errors? Maybe some psychic debugging google and stackoverflow-fu is called for.

I'll patch the change.

[eddelbuettel]

Here it's plain vanilla Ubuntu, but recent so now 25.04 and soon 25.10, and with a decent-enough recent NVidia card. I can do some tests.

[eddelbuettel]

So to keep that ball rolling, on the current main branch the end of configure is

checking for OpenCL... checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for CL/opencl.h... yes
yes (-lOpenCL)
checking for CLBlast... yes
checking for clBLAS... yes
checking for CUDA... checking for nvcc... /usr/bin/nvcc
checking for cuda_runtime.h... yes
checking for cudaMalloc in -lcudart... yes
checking for cublasCreate_v2 in -lcublas... yes
checking for cusolverDnCreate in -lcusolver... yes
checking for curandCreateGenerator in -lcurand... yes
checking for nvrtcCreateProgram in -lnvrtc... yes
yes
configure: 
configure: GPU Backend Configuration Summary:
configure:   Platform:     Linux
configure:   C++ Standard: C++14
configure:   OpenCL:       1
configure:   CLBlast:      1
configure:   clBLAS:       1
configure:   CUDA:         1
configure:   Default:      CL_BACKEND
configure: 
checking for OpenMP... found and suitable
configure: creating ./config.status
config.status: creating src/Makevars
config.status: creating R/flags.R
edd@paul:~/git/rcppbandicoot(main)$ 

and I get (after adding -DCL_TARGET_OPENCL_VERSION=300 to silence a nag from OpenCL; error is the same regardless)

[....]
configure: 
configure: GPU Backend Configuration Summary:
configure:   Platform:     Linux
configure:   C++ Standard: C++14
configure:   OpenCL:       1
configure:   CLBlast:      1
configure:   clBLAS:       1
configure:   CUDA:         1
configure:   Default:      CL_BACKEND
configure: 
checking for OpenMP... found and suitable
configure: creating ./config.status
config.status: creating src/Makevars
config.status: creating R/flags.R
** libs
using C++ compiler: ‘g++-15 (Ubuntu 15-20250404-0ubuntu1) 15.0.1 20250404 (experimental) [master r15-9193-g08e803aa9be]’
using C++14
ccache g++-15  -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -DCL_TARGET_OPENCL_VERSION=300 -I'/usr/local/lib/R/site-library/Rcpp/include'    -DCOOT_USE_OPENCL -DCOOT_USE_CLBLAST -DCOOT_USE_CLBLAS -DCOOT_USE_CUDA -DCOOT_CUDA_INCLUDE_PATH=/usr/include/ -DCOOT_DEFAULT_BACKEND=CL_BACKEND -fopenmp -fopenmp -fpic  -O3 -Wall -pipe -pedantic -Wno-parentheses -Wno-ignored-attributes -Wno-unused-function    -c RcppExports.cpp -o RcppExports.o
ccache g++-15  -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -DCL_TARGET_OPENCL_VERSION=300 -I'/usr/local/lib/R/site-library/Rcpp/include'    -DCOOT_USE_OPENCL -DCOOT_USE_CLBLAST -DCOOT_USE_CLBLAS -DCOOT_USE_CUDA -DCOOT_CUDA_INCLUDE_PATH=/usr/include/ -DCOOT_DEFAULT_BACKEND=CL_BACKEND -fopenmp -fopenmp -fpic  -O3 -Wall -pipe -pedantic -Wno-parentheses -Wno-ignored-attributes -Wno-unused-function    -c coot.cpp -o coot.o
ccache g++-15  -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -DCL_TARGET_OPENCL_VERSION=300 -I'/usr/local/lib/R/site-library/Rcpp/include'    -DCOOT_USE_OPENCL -DCOOT_USE_CLBLAST -DCOOT_USE_CLBLAS -DCOOT_USE_CUDA -DCOOT_CUDA_INCLUDE_PATH=/usr/include/ -DCOOT_DEFAULT_BACKEND=CL_BACKEND -fopenmp -fopenmp -fpic  -O3 -Wall -pipe -pedantic -Wno-parentheses -Wno-ignored-attributes -Wno-unused-function    -c example.cpp -o example.o
ccache g++-15  -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -DCL_TARGET_OPENCL_VERSION=300 -I'/usr/local/lib/R/site-library/Rcpp/include'    -DCOOT_USE_OPENCL -DCOOT_USE_CLBLAST -DCOOT_USE_CLBLAS -DCOOT_USE_CUDA -DCOOT_CUDA_INCLUDE_PATH=/usr/include/ -DCOOT_DEFAULT_BACKEND=CL_BACKEND -fopenmp -fopenmp -fpic  -O3 -Wall -pipe -pedantic -Wno-parentheses -Wno-ignored-attributes -Wno-unused-function    -c gpu-ops.cpp -o gpu-ops.o
ccache g++-15 -std=gnu++14 -Wl,-S -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro -o RcppBandicoot.so RcppExports.o coot.o example.o gpu-ops.o -fopenmp -lclBLAS -lclblast -lOpenCL -L/usr/lib64 -L/usr/lib -lcudart -lcublas -lcusolver -lcurand -lnvrtc -llapack -lblas -lgfortran -lm -lquadmath -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-rcppbandicoot/00new/RcppBandicoot/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘RcppBandicoot’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/usr/local/lib/R/site-library/00LOCK-rcppbandicoot/00new/RcppBandicoot/libs/RcppBandicoot.so':
  /usr/local/lib/R/site-library/00LOCK-rcppbandicoot/00new/RcppBandicoot/libs/RcppBandicoot.so: undefined symbol: cuLaunchKernel
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/usr/local/lib/R/site-library/RcppBandicoot’
* restoring previous ‘/usr/local/lib/R/site-library/RcppBandicoot’
Warning message:
In install.packages(pkgs = f, lib = lib, repos = if (isMatchingFile(f)) NULL else repos) :
  installation of package ‘.’ had non-zero exit status
edd@paul:~/git/rcppbandicoot(main)$ 

And this has stumped for a few days now. I tried reordering the linker instructions, no beans.

[rcurtin]

Missing -lcuda? Check out the example compilation command for when the Bandicoot wrapper library isn't in use:

# for systems with both OpenCL and CUDA backends
g++ prog.cpp -o prog -std=c++11 -O2 -I /home/xyz/bandicoot-1.10.0/include -DCOOT_DONT_USE_WRAPPER -lopenblas -lcuda -lcudart -lcublas -lcurand -lcusolver -lnvrtc -lOpenCL -lCLBlast 

(it is a mouthful... also I see it says -lCLBlast instead of -lclblast; I will fix that)

[eddelbuettel]

No, I can compile CUDA examples and Bandicoot examples just fine.

I can also compile (and use) RcppBandicoot just fine provided we do not turn CUDA on.

[rcurtin]

I mean that I don't see -lcuda in the linker command line in the output above:

ccache g++-15 -std=gnu++14 -Wl,-S -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro -o RcppBandicoot.so RcppExports.o coot.o example.o gpu-ops.o -fopenmp -lclBLAS -lclblast -lOpenCL -L/usr/lib64 -L/usr/lib -lcudart -lcublas -lcusolver -lcurand -lnvrtc -llapack -lblas -lgfortran -lm -lquadmath -L/usr/lib/R/lib -lR

[eddelbuettel]

Damn. You were right, and @coatless and I take a bow. Thanks for spotting that!!

After monkey patching src/Makevars.in (as opposed to configure.ac, that comes next):

[...]
** libs
using C++ compiler: ‘g++-15 (Ubuntu 15-20250404-0ubuntu1) 15.0.1 20250404 (experimental) [master r15-9193-g08e803aa9be]’
using C++14
ccache g++-15  -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -DCL_TARGET_OPENCL_VERSION=300 -I'/usr/local/lib/R/site-library/Rcpp/include'    -DCOOT_USE_OPENCL -DCOOT_USE_CLBLAST -DCOOT_USE_CLBLAS -DCOOT_USE_CUDA -DCOOT_CUDA_INCLUDE_PATH=/usr/include/ -DCOOT_DEFAULT_BACKEND=CL_BACKEND -fopenmp -fopenmp -fpic  -O3 -Wall -pipe -pedantic -Wno-parentheses -Wno-ignored-attributes -Wno-unused-function    -c RcppExports.cpp -o RcppExports.o
ccache g++-15  -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -DCL_TARGET_OPENCL_VERSION=300 -I'/usr/local/lib/R/site-library/Rcpp/include'    -DCOOT_USE_OPENCL -DCOOT_USE_CLBLAST -DCOOT_USE_CLBLAS -DCOOT_USE_CUDA -DCOOT_CUDA_INCLUDE_PATH=/usr/include/ -DCOOT_DEFAULT_BACKEND=CL_BACKEND -fopenmp -fopenmp -fpic  -O3 -Wall -pipe -pedantic -Wno-parentheses -Wno-ignored-attributes -Wno-unused-function    -c coot.cpp -o coot.o
ccache g++-15  -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -DCL_TARGET_OPENCL_VERSION=300 -I'/usr/local/lib/R/site-library/Rcpp/include'    -DCOOT_USE_OPENCL -DCOOT_USE_CLBLAST -DCOOT_USE_CLBLAS -DCOOT_USE_CUDA -DCOOT_CUDA_INCLUDE_PATH=/usr/include/ -DCOOT_DEFAULT_BACKEND=CL_BACKEND -fopenmp -fopenmp -fpic  -O3 -Wall -pipe -pedantic -Wno-parentheses -Wno-ignored-attributes -Wno-unused-function    -c example.cpp -o example.o
ccache g++-15  -std=gnu++14 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -DCL_TARGET_OPENCL_VERSION=300 -I'/usr/local/lib/R/site-library/Rcpp/include'    -DCOOT_USE_OPENCL -DCOOT_USE_CLBLAST -DCOOT_USE_CLBLAS -DCOOT_USE_CUDA -DCOOT_CUDA_INCLUDE_PATH=/usr/include/ -DCOOT_DEFAULT_BACKEND=CL_BACKEND -fopenmp -fopenmp -fpic  -O3 -Wall -pipe -pedantic -Wno-parentheses -Wno-ignored-attributes -Wno-unused-function    -c gpu-ops.cpp -o gpu-ops.o
ccache g++-15 -std=gnu++14 -Wl,-S -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro -o RcppBandicoot.so RcppExports.o coot.o example.o gpu-ops.o -fopenmp -lcuda -lclBLAS -lclblast -lOpenCL -L/usr/lib64 -L/usr/lib -lcudart -lcublas -lcusolver -lcurand -lnvrtc -llapack -lblas -lgfortran -lm -lquadmath -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-rcppbandicoot/00new/RcppBandicoot/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (RcppBandicoot)
edd@paul:~/git/rcppbandicoot(main)$ 

[eddelbuettel]

As I am preparing a PR, what shall we do about the

-PKG_CPPFLAGS = -I../inst/include
+PKG_CPPFLAGS = -I../inst/include -DCL_TARGET_OPENCL_VERSION=300

which I need to not have OpenCL nag all the time. Any other systems / OSs have that?

[rcurtin]

See also https://gitlab.com/bandicoot-lib/bandicoot-code/-/issues/1 , it has been an outstanding issue for a very long time and I don't have a great solution for it. The situation is this:

1. OpenCL warns if CL_TARGET_OPENCL_VERSION is not set when it is included;
2. OpenCL will also warn if a CL_TARGET_OPENCL_VERSION is set that is newer than whatever the OpenCL distribution on the system is;
3. We aren't guaranteed really anything about what OpenCL version a user is using;
4. Bandicoot does its best to support whatever OpenCL version is available, so it does not set CL_TARGET_OPENCL_VERSION (which will restrict functionality to whatever that version is);

and OpenCL appears to provide no way to disable the warning or set some macro such that whatever the OpenCL header version is is what is used.

If from the R side you are able to say "we will always have OpenCL 3.0+" then you can just define it as you suggested above and be good with it, but I can't really do that upstream.