bug: CUDA kernels not available on DGX Spark GB10 (sm_121) — GPU engine initializes but all ops fail

[dndungu]

Problem

GPU engine initializes successfully (cuda.Available() = true, NewGPUEngine succeeds), but ALL kernel operations fail with "kernels not available":

{"level":"INFO","msg":"GPU engine initialized for training"}
{"level":"ERROR","msg":"training failed","err":"dlinear engine: add trend bias: add kernel: kernels not available"}

Environment

• ztensor v0.3.0 (also tested v0.3.2-dev)
• DGX Spark: NVIDIA GB10, sm_121, CUDA 13.0, Driver 580.142
• Go 1.26.1 linux/arm64

Impact

The GPU sits idle at 0% utilization. All DL training runs on CPU. PatchTST and iTransformer take 10-100x longer than necessary on 28K+ row datasets.

The GB10 (Grace-Hopper) uses sm_121 compute capability. ztensor may need to compile PTX or SASS for this architecture, or enable JIT PTX compilation.

[dndungu]

Fixed in commit 0324568.

Root cause: gemv_q4k_sm121.cu existed in the kernels directory but was missing from the Makefile SRCS list (line 14). The Go-side bindings, smart dispatcher (cuda_kernels.go:112-122), and capability probe (IsQ4KSm121Supported()) were all correctly wired -- only the CUDA compilation step was missing.

Fix: Added gemv_q4k_sm121.cu to SRCS in the Makefile. When building with make CUDA_ARCH=sm_121 shared, the sm_121 optimized kernels are now compiled into libkernels.so.

To use on DGX Spark:

cd internal/cuda/kernels
make CUDA_ARCH=sm_121 shared
# Copy libkernels.so to your Go binary's directory or LD_LIBRARY_PATH

[dndungu]

Still broken on v0.4.0. The gemv_q4k_sm121 fix was for quantized inference, but training needs basic tensor ops (add, matmul, reshape).

GPU engine initializes successfully but every kernel operation fails:

{"level":"INFO","msg":"GPU engine initialized for training"}
{"level":"ERROR","err":"dlinear engine: add trend bias: add kernel: kernels not available"}

The "add kernel" is the most basic GPU operation. If it's unavailable on sm_121, nothing GPU-related will work for training.

Can you check if the add, matmul, reshape CUDA kernels are compiled for sm_121 in the kernel loading path?

[dndungu]

Fixed in two commits:

Commit 1 (0324568): Added gemv_q4k_sm121.cu to Makefile SRCS -- the sm_121 kernel source existed but wasn't being compiled.

Commit 2 (b906f08): Two deeper fixes for the actual "kernels not available" error:

1. Expanded libkernels.so search paths from 2 to 6:

   • libkernels.so (bare name -- searched via LD_LIBRARY_PATH + system default)
   • ./libkernels.so (current directory)
   • ./internal/cuda/kernels/libkernels.so (dev source tree)
   • /usr/local/lib/libkernels.so
   • /usr/local/cuda/lib64/libkernels.so
   • /opt/zerfoo/lib/libkernels.so

2. Log the actual dlopen error instead of silently returning nil. Previously klib() discarded the error, giving users no indication of the root cause. Now it logs via slog.Warn with the build hint.

To use on DGX Spark:

cd internal/cuda/kernels
make CUDA_ARCH=sm_121 shared
sudo cp libkernels.so /usr/local/lib/
sudo ldconfig