nohup: ignoring input
The following values were not passed to accelerate launch and had defaults used instead:
--num_processes was set to a value of 2
More than one GPU was found, enabling multi-GPU training.
If this was unintended please pass in --num_processes=1.
--num_machines was set to a value of 1
--mixed_precision was set to a value of 'no'
--dynamo_backend was set to a value of 'no'
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
/home/zhangkailin/.conda/envs/diff/lib/python3.12/site-packages/accelerate/utils/launch.py:236: UserWarning: Port 29500 is already in use. Accelerate will attempt to launch in a standalone-like mode by finding an open port automatically for this session. If this current attempt fails, or for more control in future runs, please specify a different port (e.g., --main_process_port <your_chosen_port>) or use --main_process_port 0 for automatic selection in your launch command or Accelerate config file.
warnings.warn(
[W718 20:29:59.662875029 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:00.332290007 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:01.086728483 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:01.663832546 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:02.003531155 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:05.547585849 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:10.156080575 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:18.263436069 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:28.254263791 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
训练脚本
DIFFSYNTH_SKIP_DOWNLOAD=True \
CUDA_VISIBLE_DEVICES=0,1 \
NPROC_PER_NODE=2 \
accelerate launch examples/qwen_image/model_training/train.py \
--dataset_base_path /appdata/zhangkailin/agent/wam/models_dd/guofeng \
--dataset_metadata_path /appdata/zhangkailin/agent/wam/models_dd/guofeng/metadata.csv \
--max_pixels 1572516 \
--dataset_repeat 50 \
--model_id_with_origin_paths "Qwen/Qwen-Image:transformer/diffusion_pytorch_model*.safetensors,Qwen/Qwen-Image:text_encoder/model*.safetensors,Qwen/Qwen-Image:vae/diffusion_pytorch_model.safetensors" \
--learning_rate 1e-4 \
--num_epochs 6 \
--remove_prefix_in_ckpt "pipe.dit." \
--save_steps 5 \
--output_path "./models/train/Qwen-Image-2512_lora_0518" \
--lora_base_model "dit" \
--lora_target_modules "to_q,to_k,to_v,add_q_proj,add_k_proj,add_v_proj,to_out.0,to_add_out,img_mlp.net.2,img_mod.1,txt_mlp.net.2,txt_mod.1" \
--lora_rank 16 \
--find_unused_parameters \
--dataset_num_workers 8 \
--use_gradient_checkpointing \
--gradient_accumulation_steps 4 \
--find_unused_parameters \
--wandb_project "qwen-image-guofeng" \
--wandb_name "qwenimage-lora-codex200-0518"
nohup: ignoring input
The following values were not passed to
accelerate launchand had defaults used instead:--num_processeswas set to a value of2More than one GPU was found, enabling multi-GPU training.
If this was unintended please pass in
--num_processes=1.--num_machineswas set to a value of1--mixed_precisionwas set to a value of'no'--dynamo_backendwas set to a value of'no'To avoid this warning pass in values for each of the problematic parameters or run
accelerate config./home/zhangkailin/.conda/envs/diff/lib/python3.12/site-packages/accelerate/utils/launch.py:236: UserWarning: Port
29500is already in use. Accelerate will attempt to launch in a standalone-like mode by finding an open port automatically for this session. If this current attempt fails, or for more control in future runs, please specify a different port (e.g.,--main_process_port <your_chosen_port>) or use--main_process_port 0for automatic selection in your launch command or Accelerate config file.warnings.warn(
[W718 20:29:59.662875029 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:00.332290007 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:01.086728483 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:01.663832546 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:02.003531155 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:05.547585849 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:10.156080575 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:18.263436069 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
[W718 20:30:28.254263791 socket.cpp:755] [c10d] The IPv6 network addresses of (I, 38455) cannot be retrieved (gai error: -2 - Name or service not known).
训练脚本