cuSPARSELt FP8 on H100 not working

[jwfromm]

I'm interested in running FP8 x FP8 sparse matmul on H100 GPUs. The example in cuSPARSELt/matmul is exactly what I'm looking for but does not seem to work for FP8. I am using Cuda 12.4 and CusparseLt version 6.2.

Running with AB_DTYPE FP16 or AB_DTYPE INT8 both work fine, produce correct results, and have great performance. However, when I use AB_DTYPE FP8, I get this error: CUSPARSE API failed at line 207 with error: operation not supported (10).

Based on the cusparseLt release notes, this should be supported. Is it possible the check in the cusparse API hasnt been updated? Can anyone help resolve this issue?

[marsaev]

Hi @jwfromm , in the mentioned example I've changed defines here https://github.com/NVIDIA/CUDALibrarySamples/blob/master/cuSPARSELt/matmul/matmul_example.cpp#L25-L27 to

//#define AB_TYPE FP16
#define AB_TYPE FP8
// #define AB_TYPE INT8

and example compiled and ran fine on H100 using CUDA 12.3 (shouldn't matter that much) and cusparseLt 0.6.2.

Could you provide your modifications to example that lead to the error?

[jwfromm]

Interesting, thats also the only change I've made. I am on H100a, is it possible that it makes a difference?

[marsaev]

@jwfromm What compute capability does your GPU have? (you can check it with https://github.com/NVIDIA/cuda-samples/tree/master/Samples/1_Utilities/deviceQueryDrv)

[jwfromm]

Here's the output from deviceQueryDrv, I dont see anything particularly exotic.

evice 0: "NVIDIA H100"
  CUDA Driver Version:                           12.0
  CUDA Capability Major/Minor version number:    9.0
  Total amount of global memory:                 97320 MBytes (102047940608 bytes)
  (132) Multiprocessors, (128) CUDA Cores/MP:     16896 CUDA Cores
  GPU Max Clock rate:                            1980 MHz (1.98 GHz)
  Memory Clock rate:                             1593 Mhz
  Memory Bus Width:                              6144-bit
  L2 Cache Size:                                 62914560 bytes
  Max Texture Dimension Sizes                    1D=(131072) 2D=(131072, 65536) 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size (x,y,z):    (2147483647, 65535, 65535)
  Texture alignment:                             512 bytes
  Maximum memory pitch:                          2147483647 bytes
  Concurrent copy and kernel execution:          Yes with 3 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Concurrent kernel execution:                   Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Managed Memory:                Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 6 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

[marsaev]

Right, looks normal to me.
Can you also try running application with CUSPARSELT_LOG_LEVEL=5 environment variable and share the output?

[jwfromm]

Ah thats super helpful. This reveals that cusparseLt isnt happy with my driver version:
[2024-08-29 09:24:30][CUSPARSELT][3207862][Error][cusparseLtMatmulDescriptorInit] current driver version does not support fp8 matrix type

I currently have driver version 525.105.17, do you happen to know what version is needed? Upgrading our internal systems is a bit complicated so I want to make sure we get the right one.

[j4yan]

@jwfromm There's a min driver requirement for fp8; from the release note in the documentation:
cuSPARSELt requires CUDA Driver r535 TRD7, r550 TRD1 or higher.

[jwfromm]

I see, I'll close this issue then. Thank you for the help!