Skip to content

fix(train): align DDP no_sync with gradient accumulation - #3472

Merged
LauraGPT merged 1 commit into
modelscope:mainfrom
YuCeong-May:fix/ddp-no-sync-accumulation
Aug 5, 2026
Merged

fix(train): align DDP no_sync with gradient accumulation#3472
LauraGPT merged 1 commit into
modelscope:mainfrom
YuCeong-May:fix/ddp-no-sync-accumulation

Conversation

@YuCeong-May

Copy link
Copy Markdown
Contributor

Summary

Type of change

  • Bug fix
  • Documentation
  • Example or demo
  • Runtime or deployment
  • Benchmark or evaluation
  • Model/training change

Validation

  • python -m compileall funasr examples tests
  • Docs or links checked
  • Runtime/deployment command tested

User impact

Notes for reviewers

@LauraGPT LauraGPT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified against current main and with a two-rank CPU/Gloo DDP reproduction.

The old batch_idx % accum_grad condition synchronizes the first microbatch and suppresses synchronization on the optimizer-step microbatch, leaving rank-local accumulated gradients (4.0 vs 7.0 in the reproduction). The proposed (batch_idx + 1) % accum_grad condition keeps early microbatches inside no_sync() and synchronizes the final microbatch, producing identical accumulated gradients on both ranks (5.5 vs 5.5).

Additional checks:

  • synthetic merge with current main is clean and changes only this condition
  • python -m py_compile funasr/train_utils/trainer_ds.py
  • git diff --check

Approved.

@LauraGPT
LauraGPT merged commit d410a56 into modelscope:main Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants