swift3.0 internvl2 单机多卡训练报错 checkpoint use_reentrant #2590
Description
环境：5*A100 40 G
训练命令：
nproc_per_node=5
CUDA_VISIBLE_DEVICES=0,1,2,3,4 \
NPROC_PER_NODE=$nproc_per_node \
swift sft --model_type internvl2 --model /model_pretrained/InternVL2-8B --dataset train_swift.json --val_dataset val_swift.json --report_to wandb --num_train_epochs 1 --save_total_limit 5
/root/miniconda3/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:632: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.5 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
return fn(*args, **kwargs)
use_cache=True is incompatible with gradient checkpointing. Setting use_cache=False...
[rank2]: Traceback (most recent call last):
[rank2]: File "/root/autodl-tmp/ms-swift/swift/cli/sft.py", line 5, in
[rank2]: sft_main()
[rank2]: File "/root/autodl-tmp/ms-swift/swift/llm/train/sft.py", line 308, in sft_main
[rank2]: return SwiftSft(args).main()
[rank2]: File "/root/autodl-tmp/ms-swift/swift/llm/base.py", line 45, in main
[rank2]: result = self.run()
[rank2]: File "/root/autodl-tmp/ms-swift/swift/llm/train/sft.py", line 172, in run
[rank2]: return self.train(trainer)
[rank2]: File "/root/autodl-tmp/ms-swift/swift/llm/train/sft.py", line 225, in train
[rank2]: trainer.train(trainer.args.resume_from_checkpoint)
[rank2]: File "/root/autodl-tmp/ms-swift/swift/trainers/mixin.py", line 231, in train
[rank2]: return super().train(*args, **kwargs)
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2123, in train
[rank2]: return inner_training_loop(
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2481, in _inner_training_loop
[rank2]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 3612, in training_step
[rank2]: self.accelerator.backward(loss, **kwargs)
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/accelerate/accelerator.py", line 2248, in backward
[rank2]: loss.backward(**kwargs)
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
[rank2]: torch.autograd.backward(
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/torch/autograd/init.py", line 347, in backward
[rank2]: _engine_run_backward(
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
[rank2]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/torch/autograd/function.py", line 307, in apply
[rank2]: return user_fn(self, *args)
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 321, in backward
[rank2]: torch.autograd.backward(outputs_with_grad, args_with_grad)
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/torch/autograd/init.py", line 347, in backward
[rank2]: _engine_run_backward(
[rank2]: File "/root/miniconda3/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
[rank2]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[rank2]: RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the forward function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple checkpoint functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
[rank2]: Parameter at index 319 with name base_model.model.language_model.model.layers.31.feed_forward.w2.lora_B.default.weight has been marked as ready twice. This means that multiple autograd engine hooks have fired for this particular parameter during this iteration.