Fix padding_idx logical error in Adaptive Input #1629
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi Huffon! Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file.In order for us to review and merge your code, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks! |
|
Thanks, nice catch! |
There was a problem hiding this comment.
@myleott has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
moussaKam
pushed a commit
to moussaKam/language-adaptive-pretraining
that referenced
this pull request
Sep 29, 2020
Summary: # Before submitting - [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements) - [x] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)? - [ ] Did you make sure to update the docs? - [ ] Did you write any new necessary tests? ## What does this PR do? I think if we keep pass **padding index of vocabulary** as `padding_idx` to adaptive embedding layers, there will be no chance to train some words. e.g. If `cut_off` is (20000,60000) and vocab is larger than 60000, we can't learn[**20,000+padding_idx**]th word and [**60,000+padding_idx**]th word. Because those words' ids will be **padding_idx** by subtraction logic and eventually get zero tensors. So, I changed `self.padding_idx` to `None` after assign vocab's `padding_idx` **for the first time at head embedding representation**. Pull Request resolved: facebookresearch#1629 Differential Revision: D19557340 Pulled By: myleott fbshipit-source-id: e0c3b38862374d422a46dc62c248b2ecfbf08fd2
facebook-github-bot
pushed a commit
that referenced
this pull request
Feb 17, 2021
Summary: # Before submitting - [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements) - [ ] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)? - [ ] Did you make sure to update the docs? - [ ] Did you write any new necessary tests? ## What does this PR do? Fixes # (issue). ## PR review Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in Github issues there's a high chance it will not be merged. ## Did you have fun? Make sure you had fun coding � Pull Request resolved: fairinternal/fairseq-py#1629 Reviewed By: myleott Differential Revision: D26484942 Pulled By: sshleifer fbshipit-source-id: 9dcbab5c404c14d8f35628d823102ad9ce59dffd
harkash
pushed a commit
to harkash/fairseq
that referenced
this pull request
Feb 23, 2021
Summary: # Before submitting - [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements) - [ ] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)? - [ ] Did you make sure to update the docs? - [ ] Did you write any new necessary tests? ## What does this PR do? Fixes # (issue). ## PR review Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in Github issues there's a high chance it will not be merged. ## Did you have fun? Make sure you had fun coding � Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/1629 Reviewed By: myleott Differential Revision: D26484942 Pulled By: sshleifer fbshipit-source-id: 9dcbab5c404c14d8f35628d823102ad9ce59dffd
sshleifer
added a commit
that referenced
this pull request
Apr 7, 2021
Summary: # Before submitting - [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements) - [ ] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)? - [ ] Did you make sure to update the docs? - [ ] Did you write any new necessary tests? ## What does this PR do? Fixes # (issue). ## PR review Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in Github issues there's a high chance it will not be merged. ## Did you have fun? Make sure you had fun coding � Pull Request resolved: fairinternal/fairseq-py#1629 Reviewed By: myleott Differential Revision: D26484942 Pulled By: sshleifer fbshipit-source-id: 9dcbab5c404c14d8f35628d823102ad9ce59dffd
Harleen8118
pushed a commit
to Harleen8118/IBERT
that referenced
this pull request
Jun 26, 2025
Summary: # Before submitting - [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements) - [x] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)? - [ ] Did you make sure to update the docs? - [ ] Did you write any new necessary tests? ## What does this PR do? I think if we keep pass **padding index of vocabulary** as `padding_idx` to adaptive embedding layers, there will be no chance to train some words. e.g. If `cut_off` is (20000,60000) and vocab is larger than 60000, we can't learn[**20,000+padding_idx**]th word and [**60,000+padding_idx**]th word. Because those words' ids will be **padding_idx** by subtraction logic and eventually get zero tensors. So, I changed `self.padding_idx` to `None` after assign vocab's `padding_idx` **for the first time at head embedding representation**. Pull Request resolved: facebookresearch/fairseq#1629 Differential Revision: D19557340 Pulled By: myleott fbshipit-source-id: e0c3b38862374d422a46dc62c248b2ecfbf08fd2
caltia
pushed a commit
to caltia/fairseq
that referenced
this pull request
Jul 8, 2025
Summary: # Before submitting - [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements) - [x] Did you read the [contributor guideline](https://github.com/pytorch/fairseq/blob/master/CONTRIBUTING.md)? - [ ] Did you make sure to update the docs? - [ ] Did you write any new necessary tests? ## What does this PR do? I think if we keep pass **padding index of vocabulary** as `padding_idx` to adaptive embedding layers, there will be no chance to train some words. e.g. If `cut_off` is (20000,60000) and vocab is larger than 60000, we can't learn[**20,000+padding_idx**]th word and [**60,000+padding_idx**]th word. Because those words' ids will be **padding_idx** by subtraction logic and eventually get zero tensors. So, I changed `self.padding_idx` to `None` after assign vocab's `padding_idx` **for the first time at head embedding representation**. Pull Request resolved: facebookresearch/fairseq#1629 Differential Revision: D19557340 Pulled By: myleott fbshipit-source-id: e0c3b38862374d422a46dc62c248b2ecfbf08fd2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before submitting
What does this PR do?
I think if we keep pass padding index of vocabulary as
padding_idxto adaptive embedding layers,there will be no chance to train some words.
e.g. If
cut_offis (20000,60000) and vocab is larger than 60000,we can't learn[20,000+padding_idx]th word and [60,000+padding_idx]th word.
Because those words' ids will be padding_idx by subtraction logic and eventually get zero tensors.
So, I changed
self.padding_idxtoNoneafter assign vocab'spadding_idxfor the first time at head embedding representation.