-
Notifications
You must be signed in to change notification settings - Fork 25.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DPT implementation contains unused parameters #30633
Comments
Hi @ducha-aiki, thanks for reporting! You are right, it looks like we can safely delete Would you mind sharing why this prevents DDP training? |
@qubvel I believe I shared this in:
That is a quote from the error crash message I am getting, when running with |
Thank you, I missed it 🙂 I am trying to understand why backbone unused weights are not blocking, while neck's block. Did you try training with a fix? |
@qubvel good point about the backbone. Probably because I have trained with a frozen backbone, which is kind of common. |
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
System Info
Kind of irrelevant, but:
Who can help?
The first (zeroth) fusion layer is never used and causing issues like when run on DDP:
@amyeroberts
Information
Reproduction
We take the code from DPT doc page
https://huggingface.co/docs/transformers/main/en/model_doc/dpt
make forward-backward pass and check unused parameters
Result:
This prevents DDP training. To fix that, one should add line into DPT model
Here is the same fix in the
mmsegmentation
https://github.com/open-mmlab/mmsegmentation/blob/main/mmseg/models/decode_heads/dpt_head.py#L271
I can submit a PR which makes this fix
Expected behavior
Not have unused parameters.
The text was updated successfully, but these errors were encountered: