Spark RAPIDS takes a long time to initialise GPU memory

[melvin-koh]

Hi all, when testing Spark RAPIDS, I encountered a problem where the executors always take exactly 20mins to initialise the GPU memory. Here is what I see from the executors' log:
RapidExecutorPlugin: Initializing memory from Executor Plugin
and they will stuck at this point for 20mins before starting processing. Does anyone know how can I speed up this initialization?

Details of the hardware I'm testing now:
GPU: H100 (as vGPU to my VM Spark hosts)
RAPIDS version: 22.02.01.20
cuDF version: 22.02.0
Nvidia driver: 550.54.15
Cuda version: 12.4
Spark version: 3.2.3

[jlowe]

Version 22.02 does not officially support H100 GPUs. H100 is not officially supported until version 23.06. The long delay is caused by the driver JIT-compiling all the cudf GPU kernels from the PTX code for H100 so they can run on that GPU. There are a ton of kernels, so this takes a very long time. I strongly suggest updating to a more recent version, e.g.: 24.06.1, and that should fix the long startup delay. Download details for 24.06.1 can be found here: https://github.com/NVIDIA/spark-rapids/blob/branch-24.06/docs/download.md

[melvin-koh]

I have updated the rapids version to 24.06.1 and now the 20mins delay is gone. Thanks @jlowe for the help.