训练微调出错 RuntimeError: r.nvmlDeviceGetNvLinkRemoteDeviceType_ #8018
Unanswered
Road2Redemption
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
我在我的机器上可以推理，但是不能训练模型，
这是报错信息：
[rank0]: RuntimeError: r.nvmlDeviceGetNvLinkRemoteDeviceType_ INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":33, please
report a bug to PyTorch. Can't find nvmlDeviceGetNvLinkRemoteDeviceType: /usr/lib64/libnvidia-ml.so.1: undefined symbol: nvmlD
eviceGetNvLinkRemoteDeviceType
下面是我的配置：
Mon May 12 09:26:06 2025
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.156.00 Driver Version: 450.156.00 CUDA Version: 12.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 A100-SXM4-40GB On | 00000000:13:00.0 Off | 0 |
| N/A 31C P0 55W / 400W | 0MiB / 40537MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 1 A100-SXM4-40GB On | 00000000:93:00.0 Off | 0 |
| N/A 32C P0 59W / 400W | 0MiB / 40537MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 2 A100-SXM4-40GB On | 00000000:99:00.0 Off | 0 |
| N/A 29C P0 58W / 400W | 0MiB / 40537MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 3 A100-SXM4-40GB On | 00000000:CB:00.0 Off | 0 |
| N/A 30C P0 57W / 400W | 0MiB / 40537MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
这是我的conda环境：
Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
accelerate 0.34.0 pypi_0 pypi
aiofiles 24.1.0 pypi_0 pypi
aiohappyeyeballs 2.6.1 pypi_0 pypi
aiohttp 3.11.18 pypi_0 pypi
aiohttp-cors 0.8.1 pypi_0 pypi
aiosignal 1.3.2 pypi_0 pypi
airportsdata 20250224 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
antlr4-python3-runtime 4.9.3 pypi_0 pypi
anyio 4.9.0 pypi_0 pypi
astor 0.8.1 pypi_0 pypi
async-timeout 5.0.1 pypi_0 pypi
attrs 25.3.0 pypi_0 pypi
audioread 3.0.1 pypi_0 pypi
av 14.3.0 pypi_0 pypi
blake3 1.0.4 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6
ca-certificates 2025.2.25 h06a4308_0
cachetools 5.5.2 pypi_0 pypi
certifi 2025.4.26 pypi_0 pypi
cffi 1.17.1 pypi_0 pypi
charset-normalizer 3.4.2 pypi_0 pypi
click 8.2.0 pypi_0 pypi
cloudpickle 3.1.1 pypi_0 pypi
colorful 0.5.6 pypi_0 pypi
compressed-tensors 0.9.1 pypi_0 pypi
contourpy 1.3.2 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
datasets 3.5.0 pypi_0 pypi
decorator 5.2.1 pypi_0 pypi
deepspeed 0.16.7 pypi_0 pypi
depyf 0.18.0 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
diskcache 5.6.3 pypi_0 pypi
distlib 0.3.9 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
docstring-parser 0.16 pypi_0 pypi
einops 0.8.1 pypi_0 pypi
exceptiongroup 1.3.0 pypi_0 pypi
fastapi 0.115.12 pypi_0 pypi
ffmpy 0.5.0 pypi_0 pypi
filelock 3.18.0 pypi_0 pypi
fire 0.7.0 pypi_0 pypi
fonttools 4.58.0 pypi_0 pypi
frozenlist 1.6.0 pypi_0 pypi
fsspec 2024.6.1 pypi_0 pypi
gguf 0.10.0 pypi_0 pypi
google-api-core 2.24.2 pypi_0 pypi
google-auth 2.40.1 pypi_0 pypi
googleapis-common-protos 1.70.0 pypi_0 pypi
gradio 5.25.0 pypi_0 pypi
gradio-client 1.8.0 pypi_0 pypi
groovy 0.1.2 pypi_0 pypi
grpcio 1.71.0 pypi_0 pypi
h11 0.16.0 pypi_0 pypi
hf-xet 1.1.0 pypi_0 pypi
hjson 3.1.0 pypi_0 pypi
httpcore 1.0.9 pypi_0 pypi
httptools 0.6.4 pypi_0 pypi
httpx 0.28.1 pypi_0 pypi
huggingface-hub 0.31.1 pypi_0 pypi
idna 3.10 pypi_0 pypi
importlib-metadata 8.7.0 pypi_0 pypi
interegular 0.3.3 pypi_0 pypi
jieba 0.42.1 pypi_0 pypi
jinja2 3.1.6 pypi_0 pypi
jiter 0.9.0 pypi_0 pypi
joblib 1.5.0 pypi_0 pypi
jsonschema 4.23.0 pypi_0 pypi
jsonschema-specifications 2025.4.1 pypi_0 pypi
kiwisolver 1.4.8 pypi_0 pypi
lark 1.2.2 pypi_0 pypi
lazy-loader 0.4 pypi_0 pypi
ld_impl_linux-64 2.40 h12ee557_0
libffi 3.4.4 h6a678d5_1
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
librosa 0.11.0 pypi_0 pypi
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h5eee18b_0
llamafactory 0.9.3.dev0 pypi_0 pypi
llvmlite 0.44.0 pypi_0 pypi
lm-format-enforcer 0.10.11 pypi_0 pypi
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 3.0.2 pypi_0 pypi
matplotlib 3.10.3 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mistral-common 1.5.4 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
msgpack 1.1.0 pypi_0 pypi
msgspec 0.19.0 pypi_0 pypi
multidict 6.4.3 pypi_0 pypi
multiprocess 0.70.16 pypi_0 pypi
ncurses 6.4 h6a678d5_0
nest-asyncio 1.6.0 pypi_0 pypi
networkx 3.4.2 pypi_0 pypi
ninja 1.11.1.4 pypi_0 pypi
nltk 3.9.1 pypi_0 pypi
numba 0.61.2 pypi_0 pypi
numpy 1.26.4 pypi_0 pypi
nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi
nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi
nvidia-curand-cu12 10.3.2.106 pypi_0 pypi
nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi
nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi
nvidia-ml-py 12.575.51 pypi_0 pypi
nvidia-nccl-cu12 2.21.5 pypi_0 pypi
nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
nvidia-nvtx-cu12 12.1.105 pypi_0 pypi
omegaconf 2.3.0 pypi_0 pypi
openai 1.78.0 pypi_0 pypi
opencensus 0.11.4 pypi_0 pypi
opencensus-context 0.1.3 pypi_0 pypi
opencv-python-headless 4.11.0.86 pypi_0 pypi
openssl 3.0.16 h5eee18b_0
orjson 3.10.18 pypi_0 pypi
outlines 0.1.11 pypi_0 pypi
outlines-core 0.1.26 pypi_0 pypi
packaging 25.0 pypi_0 pypi
pandas 2.2.3 pypi_0 pypi
partial-json-parser 0.2.1.1.post5 pypi_0 pypi
peft 0.15.1 pypi_0 pypi
pillow 11.2.1 pypi_0 pypi
pip 25.1 pyhc872135_2
platformdirs 4.3.8 pypi_0 pypi
pooch 1.8.2 pypi_0 pypi
prometheus-client 0.21.1 pypi_0 pypi
prometheus-fastapi-instrumentator 7.1.0 pypi_0 pypi
propcache 0.3.1 pypi_0 pypi
proto-plus 1.26.1 pypi_0 pypi
protobuf 6.30.2 pypi_0 pypi
psutil 7.0.0 pypi_0 pypi
py-cpuinfo 9.0.0 pypi_0 pypi
py-spy 0.4.0 pypi_0 pypi
pyarrow 20.0.0 pypi_0 pypi
pyasn1 0.6.1 pypi_0 pypi
pyasn1-modules 0.4.2 pypi_0 pypi
pycountry 24.6.1 pypi_0 pypi
pycparser 2.22 pypi_0 pypi
pydantic 2.10.6 pypi_0 pypi
pydantic-core 2.27.2 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pygments 2.19.1 pypi_0 pypi
pyparsing 3.2.3 pypi_0 pypi
python 3.10.16 he870216_1
python-dateutil 2.9.0.post0 pypi_0 pypi
python-dotenv 1.1.0 pypi_0 pypi
python-multipart 0.0.20 pypi_0 pypi
pytz 2025.2 pypi_0 pypi
pyyaml 6.0.2 pypi_0 pypi
pyzmq 26.4.0 pypi_0 pypi
ray 2.46.0 pypi_0 pypi
readline 8.2 h5eee18b_0
referencing 0.36.2 pypi_0 pypi
regex 2024.11.6 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
rich 14.0.0 pypi_0 pypi
rouge-chinese 1.0.3 pypi_0 pypi
rpds-py 0.24.0 pypi_0 pypi
rsa 4.9.1 pypi_0 pypi
ruff 0.11.9 pypi_0 pypi
safehttpx 0.1.6 pypi_0 pypi
safetensors 0.5.3 pypi_0 pypi
scikit-learn 1.6.1 pypi_0 pypi
scipy 1.15.3 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setuptools 78.1.1 pypi_0 pypi
shellingham 1.5.4 pypi_0 pypi
shtab 1.7.2 pypi_0 pypi
six 1.17.0 pypi_0 pypi
smart-open 7.1.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
soundfile 0.13.1 pypi_0 pypi
soxr 0.5.0.post1 pypi_0 pypi
sqlite 3.45.3 h5eee18b_0
sse-starlette 2.3.4 pypi_0 pypi
starlette 0.46.2 pypi_0 pypi
sympy 1.13.1 pypi_0 pypi
termcolor 3.1.0 pypi_0 pypi
threadpoolctl 3.6.0 pypi_0 pypi
tiktoken 0.9.0 pypi_0 pypi
tk 8.6.14 h39e8969_0
tokenizers 0.21.1 pypi_0 pypi
tomlkit 0.13.2 pypi_0 pypi
torch 2.5.1+cu121 pypi_0 pypi
torchaudio 2.5.1+cu121 pypi_0 pypi
torchvision 0.20.1+cu121 pypi_0 pypi
tqdm 4.67.1 pypi_0 pypi
transformers 4.49.0.dev0 pypi_0 pypi
triton 3.1.0 pypi_0 pypi
trl 0.9.6 pypi_0 pypi
typer 0.15.3 pypi_0 pypi
typing-extensions 4.13.2 pypi_0 pypi
typing-inspection 0.4.0 pypi_0 pypi
tyro 0.8.14 pypi_0 pypi
tzdata 2025.2 pypi_0 pypi
urllib3 2.4.0 pypi_0 pypi
uvicorn 0.34.2 pypi_0 pypi
uvloop 0.21.0 pypi_0 pypi
virtualenv 20.31.2 pypi_0 pypi
vllm 0.7.2 pypi_0 pypi
watchfiles 1.0.5 pypi_0 pypi
websockets 15.0.1 pypi_0 pypi
wheel 0.45.1 pypi_0 pypi
wrapt 1.17.2 pypi_0 pypi
xformers 0.0.28.post3 pypi_0 pypi
xgrammar 0.1.19 pypi_0 pypi
xxhash 3.5.0 pypi_0 pypi
xz 5.6.4 h5eee18b_1
yarl 1.20.0 pypi_0 pypi
zipp 3.21.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_1
这是我的yaml配置文件
model
model_name_or_path: ./Qwen2-VL-2B-Instruct
image_max_pixels: 262144
video_max_pixels: 16384
trust_remote_code: false
method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 4
lora_target: all
dataset
dataset: mllm_demo,identity,alpaca_en_demo # video: mllm_video_demo
template: qwen2_vl
cutoff_len: 2048
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 1
dataloader_num_workers: 1
output
output_dir: saves/qwen2_5vl-7b/lora/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none # choices: [none, wandb, tensorboard, swanlab, mlflow]
train
per_device_train_batch_size: 1
gradient_accumulation_steps: 1
learning_rate: 1.0e-4
num_train_epochs: 1.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null
我已经将batchsize减少到最小了，但是无论是单机单卡还是单机多卡都不能运行训练。
但是我可以进行推理。
请问有解决方案吗？因为是公司的机器我没法升级cuda driver
All reactions