[FT][ERROR] CUDA runtime error: CUBLAS_STATUS_EXECUTION_FAILED

[gongel]

Branch/Tag/Commit

nccl_dependent_refine

Docker Image Version

.

GPU name

V100

CUDA Driver

CUDA 10.1

Reproduced Steps

1. Background: Add XGLM model to FT(similar to GPT), and GPT model did not raise the error.
2. The main error message
   For fp16, I report an error in this line of code: https://github.com/NVIDIA/FasterTransformer/blob/nccl_dependent_refine/fastertransformer/open_decoder.h#L735; For fp32, there is no problem.
3. Search the same issue
   The error is same as CUDA runtime error: CUBLAS_STATUS_EXECUTION_FAILED for fp16 #181, but the weights of model are pretrained, I can not modify it.

Thank you very much for helping me.

[byshiue]

Can you try the GPT model of latest main branch? FT has supported XGLM and have some example in https://github.com/triton-inference-server/fastertransformer_backend/blob/main/docs/gpt_guide.md#run-xglm

[gongel]

Thanks for your reply. GPT model did not encounter the error in latest main branch and nccl_dependent_refine branch. But for fp16 in nccl_dependent_refine branch, XGLM will raise the error. " FT has supported XGLM " is great! Where is FT XGLM model code? I can't find it and I would appreciate it if you could provide me with the code. I will compare it with GPT model.

[byshiue]

XGLM shares same codes of GPT. It is supported in ParallelGpt class.

[gongel]

ok thanks

[gongel]

Fixed. Do 'bias_pad' will raise this error when in fp16 mode.