[Bug] RuntimeError: Triton Error [CUDA]: invalid argument #3322
Description
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
Describe the bug
报以下bug(issues里面有提到ptxas和显卡驱动的匹配问题，由于不知道ptxas与driver是如何匹配的，又是如何下载、安装的，所以没有尝试)
RuntimeError: Triton Error [CUDA]: invalid argument
运行环境：
Package Version Editable project location
--------------------------------- ------------- ----------------------------------------------
accelerate 1.4.0
addict 2.4.0
aiohappyeyeballs 2.4.6
aiohttp 3.11.12
aiohttp-cors 0.7.0
aiosignal 1.3.2
airportsdata 20241001
annotated-types 0.7.0
anthropic 0.46.0
anyio 4.8.0
astor 0.8.1
asttokens 3.0.0
async-timeout 5.0.1
attrs 25.1.0
baidu-aip 4.16.13
bcrypt 4.3.0
beautifulsoup4 4.13.3
bitsandbytes 0.45.3
blake3 1.0.4
blinker 1.9.0
Brotli 1.1.0
cachetools 5.5.2
certifi 2025.1.31
cffi 1.17.1
cfgv 3.4.0
chardet 5.2.0
charset-normalizer 3.4.1
click 8.1.8
cloudpickle 3.1.1
colorful 0.5.6
colossalai 0.4.9
compressed-tensors 0.9.1
contexttimer 0.3.3
contourpy 1.3.1
cryptography 44.0.2
cuda-bindings 12.8.0
cuda-python 12.8.0
cycler 0.12.1
datasets 3.3.2
decorator 5.1.1
decord 0.6.0
deepspeed 0.15.4
Deprecated 1.2.18
depyf 0.18.0
diffusers 0.29.0
dill 0.3.8
diskcache 5.6.3
distlib 0.3.9
distro 1.9.0
docstring_parser 0.16
einops 0.8.1
exceptiongroup 1.2.2
executing 2.2.0
fabric 3.2.2
fastapi 0.115.8
filelock 3.17.0
fire 0.7.0
flashinfer-python 0.2.1.post2
Flask 3.1.0
Flask-Cors 5.0.0
fonttools 4.56.0
frozenlist 1.5.0
fsspec 2024.12.0
galore-torch 1.0
gevent 24.11.1
gguf 0.10.0
gmpy2 2.1.5
google 3.0.0
google-api-core 2.24.1
google-auth 2.38.0
googleapis-common-protos 1.68.0
greenlet 3.1.1
grpcio 1.70.0
h11 0.14.0
h2 4.2.0
hf_transfer 0.1.9
hjson 3.1.0
hpack 4.1.0
httpcore 1.0.7
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.29.1
hyperframe 6.1.0
identify 2.6.9
idna 3.10
importlib_metadata 8.6.1
iniconfig 2.0.0
interegular 0.3.3
invoke 2.2.0
ipdb 0.13.13
ipython 8.32.0
itsdangerous 2.2.0
jedi 0.19.2
Jinja2 3.1.5
jiter 0.8.2
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
kiwisolver 1.4.8
lark 1.2.2
litellm 1.61.13
llvmlite 0.44.0
lm-format-enforcer 0.10.10
lmdeploy 0.7.1 /disk2/eliviate/lmdeploy_main_branch/lmdeploy
loguru 0.7.3
markdown-it-py 3.0.0
MarkupSafe 3.0.2
matplotlib 3.10.0
matplotlib-inline 0.1.7
mdurl 0.1.2
mistral_common 1.5.3
mmengine-lite 0.10.6
modelscope 1.23.1
mpmath 1.3.0
msgpack 1.1.0
msgspec 0.19.0
multidict 6.1.0
multiprocess 0.70.16
nest-asyncio 1.6.0
networkx 3.4.2
ninja 1.11.1.3
nodeenv 1.9.1
numba 0.61.0
numpy 1.26.4
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-ml-py 12.570.86
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
openai 1.63.2
opencensus 0.11.4
opencensus-context 0.1.3
opencv-python-headless 4.11.0.86
orjson 3.10.15
outlines 0.1.11
outlines_core 0.1.26
packaging 24.2
pandas 2.2.3
paramiko 3.5.1
parso 0.8.4
partial-json-parser 0.2.1.1.post5
peft 0.11.1
pexpect 4.9.0
pillow 11.1.0
pip 25.0.1
platformdirs 4.3.6
pluggy 1.5.0
plumbum 1.9.0
pre_commit 4.1.0
prometheus_client 0.21.1
prometheus-fastapi-instrumentator 7.0.2
prompt_toolkit 3.0.50
propcache 0.3.0
proto-plus 1.26.0
protobuf 5.29.3
psutil 7.0.0
ptyprocess 0.7.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
py-spy 0.4.0
pyairports 2.1.1
pyarrow 19.0.1
pyasn1 0.6.1
pyasn1_modules 0.4.1
pybind11 2.13.6
pycountry 24.6.1
pycparser 2.22
pydantic 2.10.6
pydantic_core 2.27.2
Pygments 2.19.1
PyNaCl 1.5.0
pynvml 12.0.0
pyparsing 3.2.1
PySocks 1.7.1
pytest 8.3.4
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-multipart 0.0.20
pytz 2025.1
PyYAML 6.0.2
pyzmq 26.2.1
ray 2.42.1
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
rich 13.9.4
rpds-py 0.23.1
rpyc 6.0.0
rsa 4.9
safetensors 0.4.5
sentencepiece 0.2.0
setproctitle 1.3.4
setuptools 75.8.0
sgl-kernel 0.0.3.post6
shortuuid 1.0.13
shtab 1.7.1
six 1.17.0
smart-open 7.1.0
sniffio 1.3.1
soupsieve 2.6
stack-data 0.6.3
starlette 0.45.3
sympy 1.13.1
termcolor 2.5.0
tiktoken 0.9.0
tokenizers 0.20.3
tomli 2.2.1
torch 2.5.1
torchao 0.8.0
torchaudio 2.5.1
torchvision 0.20.1
tqdm 4.67.1
traitlets 5.14.3
transformers 4.46.3
triton 3.1.0
trl 0.8.6
typeguard 4.4.2
typing_extensions 4.12.2
tyro 0.9.16
tzdata 2025.1
urllib3 2.3.0
uvicorn 0.29.0
uvloop 0.21.0
virtualenv 20.29.2
watchfiles 1.0.4
wcwidth 0.2.13
websockets 15.0
Werkzeug 3.1.3
wheel 0.45.1
wrapt 1.17.2
xformers 0.0.28.post3
xgrammar 0.1.10
xxhash 3.5.0
yapf 0.43.0
yarl 1.18.3
zipp 3.21.0
zope.event 5.0
zope.interface 7.2
zstandard 0.19.0
Reproduction
（nvidia-smi显示的显卡驱动 550.144.03； cuda驱动 CUDA Version: 12.4）：
lmdeploy serve api_server /disk2/elivate/DeepSeek/DeepSeek-R1 --tp 8 --backend pytorch --chat-template deepseek --cache-max-entry-count 0.8 --server-name 0.0.0.0 --server-port 23333
Environment
sys.platform: linux
Python: 3.10.16 | packaged by conda-forge | (main, Dec 5 2024, 14:16:10) [GCC 13.3.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1,2,3,4,5,6,7: NVIDIA H200
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
GCC: gcc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
PyTorch: 2.5.1+cu124
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 12.4
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
- CuDNN 90.1
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,
TorchVision: 0.20.1+cu124
LMDeploy: 0.7.1+
transformers: 4.46.3
gradio: Not Found
fastapi: 0.115.8
pydantic: 2.10.6
triton: 3.1.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE PIX PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE PIX PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE PIX PIX SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE PIX PIX SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS PIX PIX NODE NODE 48-95,144-191 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS PIX PIX NODE NODE 48-95,144-191 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS NODE NODE PIX PIX 48-95,144-191 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE NODE PIX PIX 48-95,144-191 1 N/A
NIC0 NODE NODE NODE NODE SYS SYS SYS SYS X PIX NODE NODE NODE NODE SYS SYS SYS SYS
NIC1 NODE NODE NODE NODE SYS SYS SYS SYS PIX X NODE NODE NODE NODE SYS SYS SYS SYS
NIC2 PIX PIX NODE NODE SYS SYS SYS SYS NODE NODE X PIX NODE NODE SYS SYS SYS SYS
NIC3 PIX PIX NODE NODE SYS SYS SYS SYS NODE NODE PIX X NODE NODE SYS SYS SYS SYS
NIC4 NODE NODE PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE X PIX SYS SYS SYS SYS
NIC5 NODE NODE PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE PIX X SYS SYS SYS SYS
NIC6 SYS SYS SYS SYS PIX PIX NODE NODE SYS SYS SYS SYS SYS SYS X PIX NODE NODE
NIC7 SYS SYS SYS SYS PIX PIX NODE NODE SYS SYS SYS SYS SYS SYS PIX X NODE NODE
NIC8 SYS SYS SYS SYS NODE NODE PIX PIX SYS SYS SYS SYS SYS SYS NODE NODE X PIX
NIC9 SYS SYS SYS SYS NODE NODE PIX PIX SYS SYS SYS SYS SYS SYS NODE NODE PIX X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_9
运行的是deepseek-r1
pytorch通过conda命令从官网安装Error traceback
(RayWorkerWrapper pid=3014724) 2025-03-24 17:28:16,700 - lmdeploy - ERROR - model_agent.py:391 - Task <ModelAgentLoop> failed
(RayWorkerWrapper pid=3014724) Traceback (most recent call last):
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 386, in _on_finish_callback
(RayWorkerWrapper pid=3014724) task.result()
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 374, in _async_loop_background
(RayWorkerWrapper pid=3014724) await self._async_step_background(
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 322, in _async_step_background
(RayWorkerWrapper pid=3014724) output = await self._async_model_forward(inputs,
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 248, in _async_model_forward
(RayWorkerWrapper pid=3014724) ret = await __long_context_single_forward(inputs)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 235, in __long_context_single_forward
(RayWorkerWrapper pid=3014724) tmp_out = await __forward(inp)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 220, in __forward
(RayWorkerWrapper pid=3014724) return await self.async_forward(inputs, swap_in_map=swap_in_map, swap_out_map=swap_out_map)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 538, in async_forward
(RayWorkerWrapper pid=3014724) output = self._forward_impl(inputs, swap_in_map=swap_in_map, swap_out_map=swap_out_map)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 521, in _forward_impl
(RayWorkerWrapper pid=3014724) output = model_forward(
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(RayWorkerWrapper pid=3014724) return func(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 75, in model_forward
(RayWorkerWrapper pid=3014724) output = model(**input_dict)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/backends/cuda/graph_runner.py", line 141, in __call__
(RayWorkerWrapper pid=3014724) return self.model(**kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(RayWorkerWrapper pid=3014724) return self._call_impl(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(RayWorkerWrapper pid=3014724) return forward_call(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/models/deepseek_v2.py", line 641, in forward
(RayWorkerWrapper pid=3014724) hidden_states = self.model(
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(RayWorkerWrapper pid=3014724) return self._call_impl(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(RayWorkerWrapper pid=3014724) return forward_call(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/models/deepseek_v2.py", line 593, in forward
(RayWorkerWrapper pid=3014724) hidden_states, residual = decoder_layer(
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(RayWorkerWrapper pid=3014724) return self._call_impl(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(RayWorkerWrapper pid=3014724) return forward_call(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/models/deepseek_v2.py", line 514, in forward
(RayWorkerWrapper pid=3014724) hidden_states = self.mlp(hidden_states)
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(RayWorkerWrapper pid=3014724) return self._call_impl(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(RayWorkerWrapper pid=3014724) return forward_call(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/models/deepseek_v2.py", line 394, in forward
(RayWorkerWrapper pid=3014724) out_states = self.experts(
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(RayWorkerWrapper pid=3014724) return self._call_impl(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(RayWorkerWrapper pid=3014724) return forward_call(*args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/nn/moe.py", line 457, in forward
(RayWorkerWrapper pid=3014724) ret = self.impl.forward(hidden_states, topk_weights, topk_ids, self.gate_up.weight, self.gate_up.scale,
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/backends/cuda/moe.py", line 213, in forward
(RayWorkerWrapper pid=3014724) output = fused_moe_blocked_fp8(input_quant,
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/kernels/cuda/blocked_fp8_fused_moe.py", line 294, in fused_moe_blocked_fp8
(RayWorkerWrapper pid=3014724) gate_cache = silu_and_mul(intermediate_cache1)
(RayWorkerWrapper pid=3014724) File "/disk2/eliviate/lmdeploy_main_branch/lmdeploy/lmdeploy/pytorch/kernels/cuda/activation.py", line 70, in silu_and_mul
(RayWorkerWrapper pid=3014724) _silu_and_mul_kernel[grid](gate_up,
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/triton/runtime/jit.py", line 345, in <lambda>
(RayWorkerWrapper pid=3014724) return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/triton/runtime/jit.py", line 691, in run
(RayWorkerWrapper pid=3014724) kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
(RayWorkerWrapper pid=3014724) File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/triton/backends/nvidia/driver.py", line 365, in __call__
(RayWorkerWrapper pid=3014724) self.launch(*args, **kwargs)
(RayWorkerWrapper pid=3014724) RuntimeError: Triton Error [CUDA]: invalid argument
(RayWorkerWrapper pid=3014787) 2025-03-22 15:22:46,531 - lmdeploy - WARNING - utils.py:18 - For higher performance, please install flash_mla https://github.com/deepseek-ai/FlashMLA [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)