cublaslt_gemm fp8 does not work with RTX 40

[edisonchan]

What's the issue, what's expected?:

cuBLAS call cublasLtMatmulAlgoGetHeuristic(handle_.get(), op_desc_.get(), a_desc_.get(), b_desc_.get(), c_desc_.get(), d_desc_.get(), preference_.get(), max_algorithm_count, results.data(), &found_algorithm_count) failed at /opt/superbench/superbench/benchmarks/micro_benchmarks/cublaslt_gemm/cublaslt_utils.cc:98 'the requested functionality is not supported'

How to reproduce it?:

docker run -it --rm --gpus all -e NVIDIA_VISIBLE_DEVICES=0 --shm-size=1g --ulimit memlock=-1  superbench/superbench:v0.8.0-cuda12.1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/superbench/lib
ldconfig
cd bin
./cublaslt_gemm -b 64 -m 2560 -n 2560 -k 16384 -i 1000 -t fp8e4m3

Additional information:
OS: ubuntu 22.04 64-bit
GPU: GeForce RTX 4090 with driver 530.41.03

[cp5555]

Sorry. SuperBenchmark doesn't officially support GeForce series at all. It would be great if you would like to contribute to it. Besides, we also don't know whether RTX 4090 supports FP8 or not as well.

[abuccts]

Hi @edisonchan, according to NVIDIA's cuBLASLt document:

FP8 was first introduced with Hopper GPUs (compute capability 9.0 or above) and is designed to further accelerate matrix multiplications.

RTX 4090 is built on Ada Lovelace arch with compute capability 8.9, which does not meet above requirement to use cuBLASLt FP8 currently. You can also check this discussion.

[edisonchan]

Hi @edisonchan, according to NVIDIA's cuBLASLt document:

FP8 was first introduced with Hopper GPUs (compute capability 9.0 or above) and is designed to further accelerate matrix multiplications.

RTX 4090 is built on Ada Lovelace arch with compute capability 8.9, which does not meet above requirement to use cuBLASLt FP8 currently. You can also check this discussion.

I am aware of that discussion, but according to another discussion on transformer-engine, the latest version of cudatoolkit should provide support. NVIDIA/TransformerEngine#15

[abuccts]

the latest version of cudatoolkit should provide support.

CUDA 12.1 update 1 was just released several days ago, and the nvcr container 23.04 is not released yet. You may run it if manually build the benchmark source with latest CUDA 12.1.1, we don't have RTX devices to try.

These versions won't be upgraded frequently neither since the main focus is on data-center GPUs.

[denglili0417]

How to reproduce it?

How to run it using the specified GPU and which parameters should be added?