【GRPO】Qwen2.5-VL启用sequence_parallel时报错

[tcye]

基座：Qwen2.5-VL-7B-Instruct
GRPO时，启用 sequence_parallel_size>1 时，会报错：

[rank5]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1428, in _compute_loss_and_metrics
[rank5]: per_token_loss1 = coef_1 * advantages.unsqueeze(1)
[rank5]: ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
[rank5]: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:5 and cuda:4!

[hjh0119]

can you provide any repro script?

[tcye]

swift rlhf
--model_type qwen2_5_vl
--rlhf_type grpo
--torch_dtype bfloat16
--train_type full
--attn_impl flash_attn
--gradient_checkpointing true
--vit_gradient_checkpointing true
--freeze_vit true
--freeze_aligner true
--model $MODEL_PATH
--max_length $MAX_LEN
--dataset xxx.jsonl
--external_plugins
$SCRIPT_DIR/resource/audit_longcot.py
--reward_funcs audit_acc audit_format
--truncation_strategy delete
--use_vllm true
--max_completion_length $MAX_COM
--vllm_mode colocate
--vllm_gpu_memory_utilization 0.3
--vllm_max_model_len $((MAX_LEN + 1000))
--vllm_enable_prefix_caching true
--vllm_tensor_parallel_size 4
--num_train_epochs 1
--per_device_train_batch_size 4
--per_device_eval_batch_size 4
--gradient_accumulation_steps 1
--learning_rate 1e-6
--save_steps 100
--save_total_limit 3
--logging_steps 1
--output_dir /mnt/cephfs/tiancaiye/o1/output-s1-grpo
--warmup_ratio 0.05
--num_generations 8
--temperature 1.0
--deepspeed zero3
--log_completions true
--dynamic_sample true
--beta 0.001
--async_generate false
--offload_optimizer true
--offload_model true
--padding_free false
--sequence_parallel_size 4
--dataloader_drop_last true
--sleep_level 1
--dataloader_num_workers 4
--dataset_num_proc 16 \

启动脚本类似上面这样，参数设置都是参考 https://github.com/modelscope/ms-swift/blob/main/examples/train/sequence_parallel/sequence_parallel_grpo.sh

[hjh0119]

can't reproduce the issue, please retry with swift main branch