New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE]: support multiple (partial) backward passes for zero #5601

Closed

ver217 opened this issue Apr 16, 2024 · 0 comments · Fixed by #5596

Assignees

Labels

Member

ver217 commented Apr 16, 2024

Describe the feature

In some vae training, users may use weight adaptive loss which may compute grad of some parameters twice, like

This will trigger backward hook twice.

Based on pytorch's document, we may use post-grad-accumulation hook to solve this problem.

ver217 added the enhancement label

ver217 self-assigned this

ver217 mentioned this issue

[zero] support multiple (partial) backward passes #5596

Merged

11 tasks

ver217 closed this as completed in #5596

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment