[INFO|modeling_utils.py:4938] 2025-04-13 02:21:32,598 >> All the weights of Qwen2AudioForConditionalGeneration were initialized from the model checkpoint at /22085400531/LLM/Qwen2-Audio-7B-Instruct.
If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen2AudioForConditionalGeneration for predictions without further training.
[INFO|configuration_utils.py:1095] 2025-04-13 02:21:32,606 >> loading configuration file /22085400531/LLM/Qwen2-Audio-7B-Instruct/generation_config.json
[INFO|configuration_utils.py:1142] 2025-04-13 02:21:32,606 >> Generate config GenerationConfig {
"chat_format": "chatml",
"do_sample": true,
"eos_token_id": [
151643,
151645
],
"pad_token_id": 151643,
"repetition_penalty": 1.1,
"temperature": 0.7,
"top_k": 20,
"top_p": 0.5
}
[INFO|2025-04-13 02:21:32] llamafactory.model.model_utils.checkpointing:157 >> Gradient checkpointing enabled.
[INFO|2025-04-13 02:21:32] llamafactory.model.model_utils.attention:157 >> Using torch SDPA for faster training and inference.
[INFO|2025-04-13 02:21:32] llamafactory.model.adapter:157 >> Upcasting trainable params to float32.
[INFO|2025-04-13 02:21:32] llamafactory.model.adapter:157 >> Fine-tuning method: LoRA
[INFO|2025-04-13 02:21:32] llamafactory.model.model_utils.misc:157 >> Found linear modules: q_proj,k_proj,up_proj,o_proj,down_proj,gate_proj,v_proj
[INFO|2025-04-13 02:21:32] llamafactory.model.model_utils.visual:157 >> Set vision model not trainable: ['audio_tower'].
[INFO|2025-04-13 02:21:32] llamafactory.model.model_utils.visual:157 >> Set multi model projector not trainable: multi_modal_projector.
[INFO|2025-04-13 02:21:46] llamafactory.model.loader:157 >> trainable params: 19,988,480 || all params: 8,417,083,392 || trainable%: 0.2375
Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
[INFO|trainer.py:748] 2025-04-13 02:21:46,822 >> Using auto half precision backend
[WARNING|trainer.py:783] 2025-04-13 02:21:46,822 >> No label_names provided for model class `PeftModelForCausalLM`. Since `PeftModel` hides base models input arguments, if label_names is not given, label_names can't be set automatically within `Trainer`. Note that empty label_names list will be used instead.
[INFO|trainer.py:2414] 2025-04-13 02:21:47,526 >> ***** Running training *****
[INFO|trainer.py:2415] 2025-04-13 02:21:47,526 >> Num examples = 100
[INFO|trainer.py:2416] 2025-04-13 02:21:47,526 >> Num Epochs = 3
[INFO|trainer.py:2417] 2025-04-13 02:21:47,526 >> Instantaneous batch size per device = 4
[INFO|trainer.py:2420] 2025-04-13 02:21:47,526 >> Total train batch size (w. parallel, distributed & accumulation) = 32
[INFO|trainer.py:2421] 2025-04-13 02:21:47,526 >> Gradient Accumulation steps = 8
[INFO|trainer.py:2422] 2025-04-13 02:21:47,526 >> Total optimization steps = 9
[INFO|trainer.py:2423] 2025-04-13 02:21:47,531 >> Number of trainable parameters = 19,988,480
0%| | 0/9 [00:00<?, ?it/s][WARNING|logging.py:328] 2025-04-13 02:21:52,298 >> `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
[rank0]: Traceback (most recent call last):
[rank0]: File "/22085400531/LLM/LLaMA-Factory/src/llamafactory/launcher.py", line 23, in <module>
[rank0]: launch()
[rank0]: File "/22085400531/LLM/LLaMA-Factory/src/llamafactory/launcher.py", line 19, in launch
[rank0]: run_exp()
[rank0]: File "/22085400531/LLM/LLaMA-Factory/src/llamafactory/train/tuner.py", line 103, in run_exp
[rank0]: _training_function(config={"args": args, "callbacks": callbacks})
[rank0]: File "/22085400531/LLM/LLaMA-Factory/src/llamafactory/train/tuner.py", line 68, in _training_function
[rank0]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]: File "/22085400531/LLM/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 102, in run_sft
[rank0]: train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/transformers/trainer.py", line 2245, in train
[rank0]: return inner_training_loop(
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/transformers/trainer.py", line 2560, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/transformers/trainer.py", line 3736, in training_step
[rank0]: loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/transformers/trainer.py", line 3801, in compute_loss
[rank0]: outputs = model(**inputs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1643, in forward
[rank0]: else self._run_ddp_forward(*inputs, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1459, in _run_ddp_forward
[rank0]: return self.module(*inputs, **kwargs) # type: ignore[index]
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/accelerate/utils/operations.py", line 823, in forward
[rank0]: return model_forward(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/accelerate/utils/operations.py", line 811, in __call__
[rank0]: return convert_to_fp32(self.model_forward(*args, **kwargs))
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
[rank0]: return func(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/peft/peft_model.py", line 1577, in forward
[rank0]: return self.base_model(
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 188, in forward
[rank0]: return self.model.forward(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/transformers/models/qwen2_audio/modeling_qwen2_audio.py", line 1141, in forward
[rank0]: outputs = self.language_model(
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/transformers/utils/generic.py", line 965, in wrapper
[rank0]: output = func(self, *args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
[rank0]: return func(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 823, in forward
[rank0]: outputs: BaseModelOutputWithPast = self.model(
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/transformers/utils/generic.py", line 965, in wrapper
[rank0]: output = func(self, *args, **kwargs)
[rank0]: File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 537, in forward
[rank0]: layer_outputs = self._gradient_checkpointing_func(
[rank0]: File "/22085400531/LLM/LLaMA-Factory/src/llamafactory/model/model_utils/checkpointing.py", line 86, in custom_gradient_checkpointing_func
[rank0]: module: "torch.nn.Module" = func.__self__
[rank0]: AttributeError: 'functools.partial' object has no attribute '__self__'. Did you mean: '__call__'?
0%| | 0/9 [00:05<?, ?it/s]
[rank0]:[W413 02:21:53.001936277 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
E0413 02:21:54.888000 31978 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 32054) of binary: /root/anaconda3/envs/ds/bin/python3.10
Traceback (most recent call last):
File "/root/anaconda3/envs/ds/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
return f(*args, **kwargs)
File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/distributed/run.py", line 918, in main
run(args)
File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/distributed/run.py", line 909, in run
elastic_launch(
File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/anaconda3/envs/ds/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/22085400531/LLM/LLaMA-Factory/src/llamafactory/launcher.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-04-13_02:21:54
host : 42e4ogtgfhbgj-0
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 32054)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Reminder
System Info
Package Version Editable project location
absl-py 2.1.0
accelerate 1.2.1
aiofiles 23.2.1
aiohappyeyeballs 2.4.4
aiohttp 3.11.10
aiosignal 1.3.2
airportsdata 20250224
albucore 0.0.23
albumentations 2.0.5
aniso8601 10.0.0
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 4.9.0
astor 0.8.1
async-timeout 5.0.1
attrs 24.3.0
audioread 3.0.1
av 14.3.0
blake3 1.0.4
blinker 1.9.0
boto3 1.37.31
botocore 1.37.31
Brotli 1.1.0
cachetools 5.5.2
certifi 2025.1.31
cffi 1.17.1
charset-normalizer 3.4.1
click 8.1.8
cloudpickle 3.1.1
coloredlogs 15.0.1
colorlog 6.9.0
compressed-tensors 0.9.2
contourpy 1.3.1
cryptography 44.0.2
cupy-cuda12x 13.4.1
cycler 0.12.1
datasets 3.2.0
decorator 5.2.1
deepspeed 0.16.1
depyf 0.18.0
dill 0.3.8
diskcache 5.6.3
distro 1.9.0
dnspython 2.7.0
doclayout_yolo 0.0.2b1
docstring_parser 0.16
einops 0.8.0
email_validator 2.2.0
exceptiongroup 1.2.2
fast-langdetect 0.2.5
fastapi 0.115.12
fastapi-cli 0.0.7
fastrlock 0.8.3
fasttext-predict 0.9.2.4
ffmpy 0.5.0
filelock 3.18.0
fire 0.7.0
Flask 3.1.0
flask-restx 1.3.0
flatbuffers 25.2.10
fonttools 4.57.0
frozenlist 1.5.0
fsspec 2024.9.0
ftfy 6.3.1
gguf 0.10.0
gmpy2 2.1.5
gradio 4.44.1
gradio_client 1.3.0
groovy 0.1.2
grpcio 1.68.1
h11 0.14.0
h2 4.1.0
hf-xet 1.0.3
hjson 3.1.0
hpack 4.0.0
httpcore 1.0.7
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.30.2
humanfriendly 10.0
hyperframe 6.0.1
idna 3.10
importlib_metadata 8.6.1
importlib_resources 6.5.2
interegular 0.3.3
itsdangerous 2.2.0
jieba 0.42.1
Jinja2 3.1.6
jiter 0.9.0
jmespath 1.0.1
joblib 1.4.2
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
kiwisolver 1.4.8
lark 1.2.2
lazy_loader 0.4
librosa 0.11.0
llamafactory 0.9.2.dev0 /22085400531/LLM/LLaMA-Factory
llguidance 0.7.13
llvmlite 0.44.0
lm-format-enforcer 0.10.11
loguru 0.7.3
magic-pdf 1.3.1
Markdown 3.7
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.10.1
mdurl 0.1.2
mistral_common 1.5.4
modelscope 1.24.1
mpmath 1.3.0
msgpack 1.1.0
msgspec 0.19.0
multidict 6.1.0
multiprocess 0.70.16
nanobind 2.6.1
nest-asyncio 1.6.0
networkx 3.4.2
ninja 1.11.1.3
nltk 3.9.1
numba 0.61.0
numpy 1.26.4
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-cusparselt-cu12 0.6.2
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
omegaconf 2.3.0
onnxruntime 1.21.0
openai 1.71.0
opencv-python 4.11.0.86
opencv-python-headless 4.11.0.86
orjson 3.10.16
outlines 0.1.11
outlines_core 0.1.26
packaging 24.2
pandas 2.2.3
partial-json-parser 0.2.1.1.post5
pdfminer.six 20231228
peft 0.12.0
pillow 10.4.0
pip 24.3.1
platformdirs 4.3.7
pooch 1.8.2
prometheus_client 0.21.1
prometheus-fastapi-instrumentator 7.1.0
propcache 0.2.1
protobuf 3.20.3
psutil 6.1.0
py-cpuinfo 9.0.0
pyarrow 18.1.0
pyclipper 1.3.0.post6
pycountry 24.6.1
pycparser 2.22
pydantic 2.11.3
pydantic_core 2.33.1
pydub 0.25.1
Pygments 2.19.1
PyMuPDF 1.24.14
pyparsing 3.2.3
PySocks 1.7.1
python-dateutil 2.9.0.post0
python-dotenv 1.1.0
python-json-logger 3.3.0
python-multipart 0.0.20
pytz 2025.2
PyYAML 6.0.2
pyzmq 26.4.0
rapid-table 1.0.5
ray 2.43.0
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
rich 14.0.0
rich-toolkit 0.14.1
robust-downloader 0.0.2
rouge-chinese 1.0.3
rpds-py 0.24.0
ruff 0.11.5
s3transfer 0.11.4
safehttpx 0.1.6
safetensors 0.4.5
scikit-learn 1.6.1
scipy 1.15.2
seaborn 0.13.2
semantic-version 2.10.0
sentencepiece 0.2.0
setuptools 75.6.0
shapely 2.1.0
shellingham 1.5.4
shtab 1.7.1
simsimd 6.2.1
six 1.17.0
sniffio 1.3.1
soundfile 0.13.1
soxr 0.5.0.post1
sse-starlette 2.2.1
starlette 0.46.1
stringzilla 3.12.3
sympy 1.13.1
tensorboard 2.18.0
tensorboard-data-server 0.7.2
termcolor 3.0.1
thop 0.1.1.post2209072238
threadpoolctl 3.6.0
tiktoken 0.9.0
tokenizers 0.21.1
tomlkit 0.12.0
torch 2.6.0
torchaudio 2.6.0
torchvision 0.21.0
tqdm 4.67.1
transformers 4.51.2
triton 3.2.0
trl 0.9.6
typer 0.15.2
typing_extensions 4.13.2
typing-inspection 0.4.0
tyro 0.8.14
tzdata 2025.2
ultralytics 8.3.106
ultralytics-thop 2.0.14
urllib3 2.4.0
uvicorn 0.34.0
uvloop 0.21.0
vllm 0.8.3
watchfiles 1.0.5
wcwidth 0.2.13
websockets 12.0
Werkzeug 3.1.3
xformers 0.0.29.post2
xgrammar 0.1.17
xxhash 3.5.0
yarl 1.18.3
zipp 3.21.0
zstandard 0.23.0
Reproduction
Others
No response