[inference] RuntimeError: CUBLAS_STATUS_NOT_SUPPORTED on cards compute capability greater than 80

[letiantony]

2021-06-30 20:37:00.954006: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
initializing gpt tokenizer...
lightseq tokenizer pad token id: 0
huggingface tokenizer pad token id: 0
creating lightseq model...
finish initializing emb_wei from host to device
finish initializing enc_wei from host to device
finish initializing all weight from host to device
gpt2 buf_bytesize: 37281664
creating huggingface model...
====================START warmup====================
=========lightseq=========
lightseq generating...
Traceback (most recent call last):
File "ls_gpt.py", line 118, in
main()
File "ls_gpt.py", line 94, in main
warmup(ls_tokenizer, hf_tokenizer, ls_model, hf_model, sentences)
File "ls_gpt.py", line 56, in warmup
ls_generate(ls_model, ls_tokenizer, ls_inputs)
File "ls_gpt.py", line 33, in ls_generate
ls_res_ids, ls_time = ls_gpt2(model, inputs)
File "ls_gpt.py", line 12, in ls_gpt2
generated_ids = model.sample(inputs)
RuntimeError: [CUDA][ERROR] /tmp/build-via-sdist-ifaem8qq/lightseq-2.0.3/lightseq/inference/model/gpt_encoder.cc.cu(397): CUBLAS_STATUS_NOT_SUPPORTED

[letiantony]

env information:

NVIDIA-SMI 455.32.00
Driver Version: 455.32.00
CUDA Version: 11.1
|GPU Name GeForce RTX 3090

[Taka152]

@letiantony After googling it. It seems a problem with this CUDA version, could you try different CUDA versions?

[letiantony]

@Taka152 thanks for the reply, it worked with cuda 10.2，but still hope it support cuda 11.1

[Taka152]

@letiantony Thanks for your feedback, we have reproduced this error on CUDA 11. It seems to be a CUBLAS API deprecated in the newest CUDA. We'll find a way to solve this.

BTW, when you try CUDA 10.2, did you downgrade the driver version or it was still 455.32.00?

[letiantony]

@Taka152 NO, the Driver Version is 440.33.01

[Taka152]

@Taka152 NO, the Driver Version is 440.33.01

I see, I guess it is the driver version, not CUDA that is incompatible with LightSeq. Because CUDA is statically linked in LightSeq lib.

@letiantony Would you mind trying the 455 driver version with CUDA 10.2? That would help a lot to confirm my guess.

[Taka152]

BTW, our test on driver 450 is OK

[letiantony]

@Taka152 Sorry, because of the limits of authority，I can't upgrade or downgrade the version. I just use another GPU to try cuda 10.2.

[Taka152]

@Taka152 Sorry, because of the limits of authority，I can't upgrade or downgrade the version. I just use another GPU to try cuda 10.2.

That's ok, we'll find it

[stranger-cyber]

@Taka152 Hi Ying Xiong. Thank you for making this awesome library available to public. Actually I encountered the same issue as mentioned in this post. Now I am trying to downgrade the graphics driver as well as CUDA to see if it works. For your information, right now I am using NVIDIA RTX 3090 graphics card, and the versions of the graphics driver and CUDA are 460.56 and 11.2 respectively.

[Taka152]

We found on driver greater than 450, some cublas API has changed, so currently lightseq inference can't run driver greater than 450

[stranger-cyber]

We found on driver greater than 450, some cublas API has changed, so currently lightseq inference can't run driver greater than 450

Ying Xiong thanks for your prompt reply. I just checked the supported product list of the 450 driver, and it seems that this driver does not support NVIDIA RTX 3090 card. I was thinking if there would be another lightseq release to resolve the CUBLAS issue on more recent drivers in the near future. Thank you!

[Taka152]

Sure, we'll try to fix it.

[misska1]

Have you fix this issue?

[Taka152]

Here is the solution.#251 (comment). This problem is not caused by driver or cuda version. It is the card that not support pre-built lightseq.

[misska1]

How to fix this?

[misska1]

I had the same issue.

====================START warmup====================
=========lightseq=========
lightseq generating...
Traceback (most recent call last):
File "test/ls_bart.py", line 102, in
main()
File "test/ls_bart.py", line 83, in main
warmup(tokenizer, ls_model, hf_model, sentences)
File "test/ls_bart.py", line 54, in warmup
ls_generate(ls_model, tokenizer, inputs_id)
File "test/ls_bart.py", line 30, in ls_generate
ls_res_ids, ls_time = ls_bart(model, inputs_id)
File "test/ls_bart.py", line 12, in ls_bart
generated_ids = model.infer(inputs)
RuntimeError: [CUDA][ERROR] /tmp/build-via-sdist-t8nsnqiz/lightseq-2.2.1/lightseq/inference/model/encoder.cc.cu(168): CUBLAS_STATUS_NOT_SUPPORTED

NVIDIA-SMI 460.106.00
Driver Version: 460.106.00
CUDA Version: 11.2
|GPU Name GeForce RTX 3090