Easier CUDA and OpenCL switch

[eddelbuettel]

It is currently impossible to get CUDA if OpenCL is present:

rcppbandicoot/configure.ac

Lines 362 to 368 in 7e1d378

	if test "x${HAVE_OPENCL}" = x1; then
	DEFAULT_BACKEND="CL_BACKEND"
	BANDICOOT_CXXFLAGS="${BANDICOOT_CXXFLAGS} -DCOOT_DEFAULT_BACKEND=CL_BACKEND"
	elif test "x${HAVE_CUDA}" = x1; then
	DEFAULT_BACKEND="CUDA_BACKEND"
	BANDICOOT_CXXFLAGS="${BANDICOOT_CXXFLAGS} -DCOOT_DEFAULT_BACKEND=CUDA_BACKEND"
	else

I am currently in the 3.0.1 branch from the pending PR, recovered the coot_rt_meat.hpp, and standardized tentatively on using COOT_KERNEL_SOURCE_DIR to allow for OpenCL and CUDA. Still, no chance for me to select CUDA (apart from reordering that same code segment). Which is not great.

It's a tricky problem but can we maybe configure both, and then allow a run-time (i.e. package load) switch? Via options() or an environment variable?

Alternatively, the package could also be greatly simplified by just throwing CUDA out. Right now there is a lot of complexity without delivering functionality. CUDA is hard and annoying to setup so I maybe ... having two packages is easier? Or two backend packages? I also do not quite understand how upstream worked and when I asked it was suggested 'it should switch at run-time' yet the documentation plainly disagrees:

(I had started locally adding a call to coot_init() thinking I might switch. Apparently one cannot.)

I currently have a working OpenCL and CUDA setup and am happy and able to test and help make this better. Let know what you are planning, I may be able to help.

[eddelbuettel]

This all seems to be murkier. I can't bandicoot 3.0.1 to build with CUDA either. It may link but I get stuff like

edd@paul:~/git/bandicoot-code/examples(unstable)$ ./example1                                                                                                                                                       
Bandicoot version: 3.0.1 (Extreme Cable Organization)                    
terminate called after throwing an instance of 'std::runtime_error'                                                                                                                                                
  what():  coot::cuda::compile_kernel(): cuModuleLoadDataEx() failed while compiling f_fill: CUDA_ERROR_NO_BINARY_FOR_GPU
Aborted (core dumped)                                                                                                                                                                                              
edd@paul:~/git/bandicoot-code/examples(unstable)$

OpenCL works for bandicoot. CUDA works on my box for, say, the NVidia examples.

[coatless]

Flying right now. I'll have it working whenever I have CUDA access.

Happy to split to OpenCL or CUDA, but I think it would be cool if we can do both together.

[coatless]

It's a tricky problem but can we maybe configure both, and then allow a run-time (i.e. package load) switch? Via options() or an environment variable?

This is my plan.

Alternatively, the package could also be greatly simplified by just throwing CUDA out... having two packages is easier? Or two backend packages?

For CRAN compliance, I feel like we're going to be stuck splitting the packages. Again, I'm tempted to push with OpenCL; but, I realize that if CUDA Isn't present, we're going to be outta luck. I don't want to canonabilize your time on this right now with the holidays this week.

Realistically, I'm booked till mid-December on projects. So, if I can't play with CUDA around Thanksgiving, we're looking at a later December release that may run into CRAN "winter break" shutdown.

[eddelbuettel]

Let me know how I help.

Edit: Apparently I cannot. While nvidia-smi is happy I cannot get (stock) bandicoot to run CUDA. Sad.

Edit 2: @rcurtin and I (as junior helper without a lab coat) are on it. My card may be too new 🙀

[rcurtin]

I don't have a lab coat but I think we can still get to the bottom of it :) upstream changes may be necessary (probably?) but that's easy enough once we find what the problem is.

[eddelbuettel]

@coatless to keep you in the loop, @rcurtin put a fix into his branch of bandicoot to deal with my new-ish card driven by the cuda kit I have under 25.04 (and that may well be worse under 24.04 but that is for another day).

[coatless]

@rcurtin do you have a CUDA box I could borrow? Otherwise, I'll try going through Google Colab's R interface (very nice remote backend nowadays; but, a jupyter notebook form.)

https://developers.googleblog.com/en/google-colab-is-coming-to-vs-code/

[eddelbuettel]

@coatless I have a CUDA card and the 12.8 software stack from Ubuntu which generally works. And with a pending merge request by @rcurtin it even works with bandicoot.

So as a quick and dirty check I snapshotted my various branches between bandicoot and RcppBandicoot and

• starting from the '3.1.0,rc' by Ryan update inst/include/bandicoot* ie the top level header and the bits
• as before reverted the order of CUDA and OpenCL preference

No configure shows

configure:                                                                                               
configure: GPU Backend Configuration Summary:
configure:   Platform:      Linux                                                                        
configure:   C++ Standard:  C++14
configure:   OpenCL:        1                                                                            
configure:   CLBlast:       1                 
configure:   clBLAS:        1                                                                            
configure:   OpenCL Target: 300
configure:   CUDA:          1                                                                            
configure:   Default:       CUDA_BACKEND                                                                 
configure:                                          
checking for OpenMP... found and suitable

(where the print of the OpenCL target is another local mod I had done here). I have to find where my exposition of coot_init() disappeared to but to a first approximation I can build the library (in its forthcoming 3.1.0 state) and the package.

I still don't know what I should suggest we do for the CRAN package. So much behavior is baked into compilation and I do not think a single CRAN machine has a GPU, and only some may have CUDA. Other packages like torch seem to ship a cpu-only viability version augmented with downloads of (yuge !!) additional blobs. I am not sure we want the complexity.

Maybe recommending local rebuilds? It's not a great solution by any stretch but what else is there?

[eddelbuettel]

I can't run coot::coot_init(type, verbosity) for either "opencl" or "cuda" though so something is possibly amiss with the kernel files environment variable and locations, respectively.

Edit Ok remembered the bit about the new COOT_KERNEL_SOURCE_DIR env var but I still cannot make it dance. All attempts fail at trying to load a kernel from ../inst/include/bandicoot_bits/.... and of course starting with ../inst is generally NOT the right choice at run-time. And I do not see where it creeps in.

[eddelbuettel]

More success thanks to another assist from @rcurtin double-checking what I did play a, for now, brute-force setting of COOT_KERNEL_SOURCE_DIR where it would be used in cuda/kernel_src.hpp. So when installing as

configure:                                                                                
configure: GPU Backend Configuration Summary:         
configure:   Platform:      Linux                                            
configure:   C++ Standard:  C++14                                      
configure:   OpenCL:        1                                                   
configure:   CLBlast:       1                                                     
configure:   clBLAS:        1       
configure:   OpenCL Target: 300 
configure:   CUDA:          1             
configure:   Default:       CUDA_BACKEND    
configure: 

with CUDA as the default and when using the Bandicoot source from the pending MR enabling the hardware / software combo I have here we get

$ R -q -e 'library(RcppBandicoot); gpu_element_square(matrix(1:4,2,2))'
> library(RcppBandicoot); gpu_element_square(matrix(1:4,2,2))
     [,1] [,2]
[1,]    1    9
[2,]    4   16
> 
$ 

[coatless]

Huzzah! It compiles and the data transfers from R into CUDA AND back into R. 😄

@eddelbuettel do you have the changes in a branch or should I prepare it?

@zoq just hooked me up with a CUDA test machine; so, I can start driving tweaks for the CUDA side.

[eddelbuettel]

@coatless Please liaise with @rcurtin (maybe via DM over on the other side). It all works for me too but it needed changes in what will be 3.1.0. So don't get carried.

(And if you have sparce cycles there are dangling Rcpp tasks waiting for you and Kevin and as Kevin seems to AWOL / snowed under may as well proceed ...)

As for changes, I essentially made none on the package side (I can double check) as everything for my particular situation was actually hinging on a newer Bandicoot setup. As for the R package, I guess you and I can experiment with "can we actually compile OpenCL and CUDA in and then change one at package load" which would be a minimum. Switching after package load may be harder. All that really keeps me scratching my chin about what the best design may be (plus, as mentioned, the absense of CUDA as best as I can tell at both CRAN and R-universe) so maybe "we check CUDA but default to OpenCL" ? I really don't know what's best.

[eddelbuettel]

My main local change (other than newer upstream branches / now main) was trying to add something like this

edd@paul:~/git/rcppbandicoot(local/bandicoot-3.1.0.rc)$ cat src/coot.cpp 
#include <RcppBandicoot.h>

// [[Rcpp::export]]
void gpu_initialize(std::string type = "opencl", bool print_info = true) {
  coot::coot_init(type.c_str(), print_info);
}
edd@paul:~/git/rcppbandicoot(local/bandicoot-3.1.0.rc)$ 

but I can't actually get it invoked from .onLoad() for either value so kinda pointless and commented out.

[eddelbuettel]

Ok, after syncing from bandicoot-code ealier today (meaning one merge request still to do but three incorporated) and updating my branch with it ... I get a fresh fail as the R package does not do the cmake step and hence no 'learns' what OpenCL version we have. So the default is the very conservative '120' and with that the build breaks.

Short term fix for me is

PKG_CPPFLAGS = -I../inst/include -DCOOT_TARGET_OPENCL_VERSION=300

and a medium-term fix for you 😁 is to run a little test compilation (or grep headers or ...) to learn the version present.

[eddelbuettel]

The following snippet, based on a bit of googling with further documenation hunting, gets us 'OpenCL 3.0 CUDA' which we can use to infer target version 300:

#include <CL/cl.h>
#include <stdio.h>

int main() {
    cl_platform_id platform;
    cl_device_id device;
    char driver_version[128];
    char opencl_c_version[128];

    // Get platform and device (simplified for demonstration)
    clGetPlatformIDs(1, &platform, NULL);
    clGetDeviceIDs(platform, CL_DEVICE_TYPE_GPU, 1, &device, NULL);

    // Get driver version
    clGetDeviceInfo(device, CL_DRIVER_VERSION, sizeof(driver_version), &driver_version, NULL);
    printf("OpenCL Driver Version: %s\n", driver_version);

    // Get OpenCL C version -- useless 1.2 version returned
    clGetDeviceInfo(device, CL_DEVICE_OPENCL_C_VERSION, sizeof(opencl_c_version), &opencl_c_version, NULL);
    printf("OpenCL C Version: %s\n", opencl_c_version);

    // This gets us OpenCL 3.0 CUDA here
    clGetDeviceInfo(device, CL_DEVICE_VERSION, sizeof(opencl_c_version), &opencl_c_version, NULL);
    printf("OpenCL Device Version: %s\n", opencl_c_version);
    return 0;
}

Demo:

edd@paul:~$ gcc -o /tmp/r/openclversion /tmp/r/openclversion.c -lOpenCL && /tmp/r/openclversion 
In file included from /usr/include/CL/cl.h:20,
                 from /tmp/r/openclversion.c:2:
/usr/include/CL/cl_version.h:22:9: note: ‘#pragma message: cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 300 (OpenCL 3.0)’
   22 | #pragma message("cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 300 (OpenCL 3.0)")
      |         ^~~~~~~
OpenCL Driver Version: 570.195.03
OpenCL C Version: OpenCL C 1.2 
OpenCL Device Version: OpenCL 3.0 CUDA
edd@paul:~$ 

There may be other / better ways but this covers at least parts of it.