RuntimeError: CUBLAS error: CUBLAS_STATUS_NOT_INITIALIZED [BUG]

[fangd123]

running the example file fill_blank.py, it raise error as follows:

Loading model
Start
Input:  北京环球度假区相关负责人介绍，北京环球影城指定单日门票将采用____制度，即推出淡季日、平季日、旺季日和特定日门票。____价格为418元，____价格为528元，____价格为638元，____价格为____元。北京环球度假区将提供90天滚动价格日历，以方便游客提前规划行程。
Traceback (most recent call last):
  File "abc.py", line 28, in <module>
    main()
  File "abc.py", line 25, in main
    fill_blank(cpm2, input_text)
  File "abc.py", line 9, in fill_blank
    for result in cpm2.fill_blank(text,
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/bminf/models/cpm2.py", line 245, in fill_blank
    for token in res:
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/bminf/models/cpm2.py", line 129, in _gen_iter
    self._model.embedding(
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/bminf/arch/t5/model.py", line 165, in embedding
    self.input_embedding.embedding_forward(ctx, tensor_ids, x_out)
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/bminf/layers/embedding.py", line 27, in embedding_forward
    ck.embedding_forward(
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/kernels/embedding.py", line 25, in embedding_forward
    embedding_kernel.cu_embedding_forward(
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/kernels/base.py", line 48, in __call__
    func = self._prepare_func()
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/kernels/base.py", line 40, in _prepare_func
    self._module.get_module(), self._func_name
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/kernels/base.py", line 23, in get_module
    Device(curr_device).use()   # force initialize context
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/device/__init__.py", line 152, in use
    self._device.use()
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/device/__init__.py", line 120, in use
    self.cublasLtHandle = cublaslt.cublasLtCreate()
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/library/base.py", line 94, in wrapper
    return f(*args, **kwargs)
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/library/cublaslt.py", line 105, in cublasLtCreate
    checkCublasStatus(cublasLt.cublasLtCreate(ctypes.byref(handle)))
  File "/home/hmqf/miniconda3/envs/script_bert/lib/python3.8/site-packages/cpm_kernels/library/cublaslt.py", line 98, in checkCublasStatus
    raise RuntimeError("CUBLAS error: {}".format(
RuntimeError: CUBLAS error: CUBLAS_STATUS_NOT_INITIALIZED

Environment:
Python 3.8.10
cudatoolkit 11.3.1

[a710128]

Is there more information about GPU usage or nvidia-smi output?

[fangd123]

Is there more information about GPU usage or nvidia-smi output?

Ok, I used conda as my package manager.

My conda version : 4.10.3
Python version : 3.8.10
GPU : RTX3090
GPU Driver version : 495.29.05
CUDA Version: 11.3.1 ( by conda install )

Here is my nvidia-smi output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.29.05    Driver Version: 495.29.05    CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:2D:00.0 Off |                  N/A |
| 30%   24C    P8    16W / 350W |  20311MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1086      G   /usr/lib/xorg/Xorg                  8MiB |
|    0   N/A  N/A      1544      G   /usr/lib/xorg/Xorg                  8MiB |
|    0   N/A  N/A      1767      G   /usr/bin/gnome-shell                6MiB |
+-----------------------------------------------------------------------------+

[a710128]

20311MiB / 24268MiB

Your GPU doesn't have enough memory left.

[fangd123]

Oh，that's my fault.
Thank you !