ImportError: /usr/local/lib/python3.10/dist-packages/vllm/_C.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE

[xiaohaiqing]

Your current environment

accelerate                    0.28.0
aiohttp                       3.8.6
aiolimiter                    1.1.0
aioprometheus                 23.12.0
aiosignal                     1.3.1
altair                        5.1.2
annotated-types               0.6.0
antlr4-python3-runtime        4.9.3
anyio                         3.7.1
async-timeout                 4.0.3
attrs                         23.1.0
auto_gptq                     0.7.1
backoff                       2.2.1
bce-python-sdk                0.8.95
beautifulsoup4                4.12.2
blinker                       1.7.0
blis                          0.7.11
Brotli                        1.1.0
cachetools                    5.3.2
catalogue                     2.0.10
certifi                       2023.7.22
cffi                          1.16.0
chardet                       5.2.0
charset-normalizer            3.3.2
click                         8.1.7
cloudpathlib                  0.16.0
cmake                         3.27.7
coloredlogs                   15.0.1
confection                    0.1.3
contourpy                     1.2.0
cryptography                  41.0.5
cycler                        0.12.1
cymem                         2.0.8
dashscope                     1.13.2
dataclasses                   0.6
dataclasses-json              0.6.2
datasets                      2.14.7
decorator                     5.1.1
dill                          0.3.7
distro                        1.8.0
effdet                        0.4.1
einops                        0.7.0
emoji                         2.8.0
et-xmlfile                    1.1.0
exceptiongroup                1.1.3
faiss-cpu                     1.7.4
fastapi                       0.104.1
filelock                      3.13.1
filetype                      1.2.0
flatbuffers                   23.5.26
fonttools                     4.44.0
frozenlist                    1.4.0
fschat                        0.2.32
fsspec                        2023.10.0
future                        0.18.3
gekko                         1.0.7
gitdb                         4.0.11
GitPython                     3.1.40
google                        3.0.0
greenlet                      3.0.1
h11                           0.14.0
h2                            4.1.0
hpack                         4.0.0
httpcore                      1.0.2
httptools                     0.6.1
httpx                         0.25.1
huggingface-hub               0.21.4
humanfriendly                 10.0
hyperframe                    6.0.1
idna                          3.4
importlib-metadata            6.8.0
iniconfig                     2.0.0
iopath                        0.1.10
Jinja2                        3.1.2
joblib                        1.3.2
jsonpatch                     1.33
jsonpointer                   2.4
jsonschema                    4.19.2
jsonschema-specifications     2023.7.1
kiwisolver                    1.4.5
langchain                     0.0.334
langchain-experimental        0.0.40
langcodes                     3.3.0
langdetect                    1.0.9
langsmith                     0.0.63
layoutparser                  0.3.4
lit                           17.0.4
lxml                          4.9.3
Markdown                      3.5.1
markdown-it-py                3.0.0
markdown2                     2.4.10
markdownify                   0.11.6
MarkupSafe                    2.1.3
marshmallow                   3.20.1
matplotlib                    3.8.1
mdurl                         0.1.2
mpmath                        1.3.0
msg-parser                    1.2.0
msgpack                       1.0.7
multidict                     6.0.4
multiprocess                  0.70.15
murmurhash                    1.0.10
mypy-extensions               1.0.0
networkx                      3.2.1
nh3                           0.2.14
ninja                         1.11.1.1
nltk                          3.8.1
numexpr                       2.8.7
numpy                         1.24.4
nvidia-cublas-cu11            11.10.3.66
nvidia-cublas-cu12            12.1.3.1
nvidia-cuda-cupti-cu11        11.7.101
nvidia-cuda-cupti-cu12        12.1.105
nvidia-cuda-nvrtc-cu11        11.7.99
nvidia-cuda-nvrtc-cu12        12.1.105
nvidia-cuda-runtime-cu11      11.7.99
nvidia-cuda-runtime-cu12      12.1.105
nvidia-cudnn-cu11             8.5.0.96
nvidia-cudnn-cu12             8.9.2.26
nvidia-cufft-cu11             10.9.0.58
nvidia-cufft-cu12             11.0.2.54
nvidia-curand-cu11            10.2.10.91
nvidia-curand-cu12            10.3.2.106
nvidia-cusolver-cu11          11.4.0.1
nvidia-cusolver-cu12          11.4.5.107
nvidia-cusparse-cu11          11.7.4.91
nvidia-cusparse-cu12          12.1.0.106
nvidia-nccl-cu11              2.14.3
nvidia-nccl-cu12              2.19.3
nvidia-nvjitlink-cu12         12.3.52
nvidia-nvtx-cu11              11.7.91
nvidia-nvtx-cu12              12.1.105
olefile                       0.46
omegaconf                     2.3.0
onnx                          1.15.0
onnxruntime                   1.15.1
openai                        1.2.3
opencv-python                 4.8.1.78
openpyxl                      3.1.2
optimum                       1.17.1
orjson                        3.9.15
packaging                     23.2
pandas                        2.0.3
pathlib                       1.0.1
pathy                         0.10.3
pdf2image                     1.16.3
pdfminer.six                  20221105
pdfplumber                    0.10.3
peft                          0.7.1
Pillow                        9.5.0
pip                           24.0
pluggy                        1.3.0
portalocker                   2.8.2
preshed                       3.0.9
prompt-toolkit                3.0.40
protobuf                      4.25.0
psutil                        5.9.6
py                            1.11.0
pyarrow                       14.0.1
pyarrow-hotfix                0.6
pyclipper                     1.3.0.post5
pycocotools                   2.0.7
pycparser                     2.21
pycryptodome                  3.9.9
pydantic                      1.10.13
pydantic_core                 2.10.1
pydantic-settings             2.0.3
pydeck                        0.8.1b0
Pygments                      2.16.1
PyJWT                         2.8.0
PyMuPDF                       1.23.6
PyMuPDFb                      1.23.6
pypandoc                      1.12
pyparsing                     3.1.1
pypdfium2                     4.24.0
pytesseract                   0.3.10
pytest                        7.4.3
python-dateutil               2.8.2
python-decouple               3.8
python-docx                   1.1.0
python-dotenv                 1.0.0
python-iso639                 2023.6.15
python-magic                  0.4.27
python-multipart              0.0.6
python-pptx                   0.6.23
pytz                          2020.5
PyYAML                        6.0.1
qianfan                       0.1.3
quantile-python               1.1
rapidfuzz                     3.5.2
rapidocr-onnxruntime          1.3.8
ray                           2.8.0
referencing                   0.30.2
regex                         2023.10.3
requests                      2.31.0
retry                         0.9.2
rich                          13.6.0
rouge                         1.0.1
rpds-py                       0.12.0
safetensors                   0.4.2
scikit-learn                  1.3.2
scipy                         1.11.3
sentence-transformers         2.2.2
sentencepiece                 0.1.99
setuptools                    59.6.0
setuptools-scm                8.0.4
shapely                       2.0.2
shortuuid                     1.0.11
simplejson                    3.19.2
six                           1.16.0
smart-open                    6.4.0
smmap                         5.0.1
sniffio                       1.3.0
socksio                       1.0.0
soupsieve                     2.5
spacy                         3.6.1
spacy-legacy                  3.0.12
spacy-loggers                 1.0.5
spacy-pkuseg                  0.0.33
SQLAlchemy                    2.0.19
srsly                         2.4.8
starlette                     0.27.0
streamlit                     1.27.2
streamlit-aggrid              0.3.4.post3
streamlit-antd-components     0.2.3
streamlit-chatbox             1.1.11
streamlit-feedback            0.1.2
streamlit-option-menu         0.3.6
strsimpy                      0.2.1
svgwrite                      1.4.3
sympy                         1.12
tabulate                      0.9.0
tenacity                      8.2.3
thinc                         8.1.12
threadpoolctl                 3.2.0
tiktoken                      0.5.1
timm                          0.9.10
tinyec                        0.4.0
tokenizers                    0.15.2
toml                          0.10.2
tomli                         2.0.1
toolz                         0.12.0
torch                         2.2.1
torchaudio                    2.2.1
torchvision                   0.17.1
tornado                       6.3.3
tqdm                          4.66.1
transformers                  4.37.2
transformers-stream-generator 0.0.4
triton                        2.2.0
typer                         0.9.0
typing_extensions             4.8.0
typing-inspect                0.9.0
tzdata                        2023.3
tzlocal                       5.2
unstructured                  0.10.30
unstructured-inference        0.7.11
unstructured.pytesseract      0.3.12
urllib3                       2.0.7
uvicorn                       0.23.2
uvloop                        0.19.0
validators                    0.22.0
vllm                          0.2.5
volcengine                    1.0.114
wasabi                        1.1.2
watchdog                      3.0.0
watchfiles                    0.21.0
wavedrom                      2.0.3.post3
wcwidth                       0.2.9
weasel                        0.3.4
websockets                    12.0
wheel                         0.37.1
xformers                      0.0.25
xlrd                          2.0.1
XlsxWriter                    3.1.9
xxhash                        3.4.1
yarl                          1.9.2
zh-core-web-lg                3.6.0
zhipuai                       1.0.7
zipp                          3.17.0

How you are installing vllm

请问这是什么原因呢？

[youkaichao]

It seems you have a mixed installation of both cuda 11 and cuda 12. Please try a fresh new install with either cuda 11.8 or cuda 12.1 .

[xiaohaiqing]

目前cuda=12.1

[youkaichao]

nvidia-cuda-nvrtc-cu11        11.7.99
nvidia-cuda-nvrtc-cu12        12.1.105

You see here you have two cuda versions. You can only have one to make vllm work.

[Pumaly]

torch-2.2.1 and vllm-0.3.3, the error is _C.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE
torch-2.1.2 huggingface has error:RuntimeError: cutlassF: no kernel found to launch!

accelerate 0.28.0
aiofiles 23.2.1
aiohttp 3.9.3
aiosignal 1.3.1
altair 5.2.0
annotated-types 0.6.0
anyio 4.3.0
archspec 0.2.3
attrs 23.2.0
boltons 23.0.0
Brotli 1.0.9
certifi 2024.2.2
cffi 1.16.0
charset-normalizer 2.0.4
click 8.1.7
cloudpickle 3.0.0
conda 24.3.0
conda-content-trust 0.2.0
conda-libmamba-solver 23.12.0
conda-package-handling 2.2.0
conda_package_streaming 0.9.0
contourpy 1.2.0
cryptography 41.0.7
cupy-cuda12x 12.1.0
cycler 0.12.1
datasets 2.18.0
dill 0.3.8
diskcache 5.6.3
distro 1.8.0
docstring_parser 0.16
einops 0.7.0
fastapi 0.110.0
fastrlock 0.8.2
ffmpy 0.3.2
filelock 3.13.1
fire 0.6.0
fonttools 4.50.0
frozenlist 1.4.1
fsspec 2024.2.0
gmpy2 2.1.2
gradio 3.50.2
gradio_client 0.6.1
h11 0.14.0
httpcore 1.0.4
httptools 0.6.1
httpx 0.27.0
huggingface-hub 0.21.4
idna 3.4
importlib_resources 6.4.0
interegular 0.3.3
Jinja2 3.1.3
joblib 1.3.2
jsonpatch 1.32
jsonpointer 2.1
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
kiwisolver 1.4.5
lark 1.1.9
libmambapy 1.5.3
llvmlite 0.42.0
markdown-it-py 3.0.0
MarkupSafe 2.1.3
matplotlib 3.8.3
mdurl 0.1.2
menuinst 2.0.1
mkl-fft 1.3.8
mkl-random 1.2.4
mkl-service 2.4.0
mpmath 1.3.0
msgpack 1.0.8
multidict 6.0.5
multiprocess 0.70.16
nest-asyncio 1.6.0
networkx 3.1
ninja 1.11.1.1
numba 0.59.1
numpy 1.26.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.18.1
nvidia-nvjitlink-cu12 12.4.99
nvidia-nvtx-cu12 12.1.105
orjson 3.9.15
outlines 0.0.36
packaging 23.1
pandas 2.2.1
peft 0.10.0
pillow 10.2.0
pip 23.3.1
platformdirs 3.10.0
pluggy 1.0.0
prometheus_client 0.20.0
protobuf 5.26.0
psutil 5.9.8
pyarrow 15.0.2
pyarrow-hotfix 0.6
pycosat 0.6.6
pycparser 2.21
pydantic 2.6.4
pydantic_core 2.16.3
pydub 0.25.1
Pygments 2.17.2
pynvml 11.5.0
pyOpenSSL 23.2.0
pyparsing 3.1.2
PySocks 1.7.1
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-multipart 0.0.9
pytz 2024.1
PyYAML 6.0.1
ray 2.10.0
referencing 0.34.0
regex 2023.12.25
requests 2.31.0
rich 13.7.1
rpds-py 0.18.0
ruamel.yaml 0.17.21
safetensors 0.4.2
scipy 1.12.0
semantic-version 2.10.0
sentencepiece 0.2.0
setuptools 68.2.2
shtab 1.7.1
six 1.16.0
sniffio 1.3.1
sse-starlette 2.0.0
starlette 0.36.3
sympy 1.12
termcolor 2.4.0
tokenizers 0.15.2
toolz 0.12.1
torch 2.1.2
torchaudio 2.2.1
torchvision 0.17.1
tqdm 4.65.0
transformers 4.39.0
triton 2.1.0
trl 0.8.1
truststore 0.8.0
typing_extensions 4.9.0
tyro 0.7.3
tzdata 2024.1
urllib3 1.26.18
uvicorn 0.29.0
uvloop 0.19.0
vllm 0.3.3
watchfiles 0.21.0
websockets 11.0.3
wheel 0.41.2
xformers 0.0.23.post1
xxhash 3.4.1
yarl 1.9.4
zstandard 0.19.0

[hnyls2002]

@xiaohaiqing In my case, this is due to the older torch not being compatible with the newest torch.

1. Downgrade the torch version to torch==2.1.1
2. If Pip says there are conflicts with xformers, downgrade xformers==0.0.23.

Hoping this will help you.