Add whole word masking for SentencepieceBPE #1292
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@myleott is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
Contributor
moussaKam
pushed a commit
to moussaKam/language-adaptive-pretraining
that referenced
this pull request
Sep 29, 2020
Summary: Models seem to train fine with this modification. I checked that the mask for beginning of words is correct but didn't check if the actual masking worked correctly. Pull Request resolved: facebookresearch#1292 Differential Revision: D18338307 Pulled By: myleott fbshipit-source-id: eae9e29d6ab648e768d70921694a898554496704
yzpang
pushed a commit
to yzpang/gold-off-policy-text-gen-iclr21
that referenced
this pull request
Feb 19, 2021
Summary: Models seem to train fine with this modification. I checked that the mask for beginning of words is correct but didn't check if the actual masking worked correctly. Pull Request resolved: facebookresearch/fairseq#1292 Differential Revision: D18338307 Pulled By: myleott fbshipit-source-id: eae9e29d6ab648e768d70921694a898554496704
yzpang
pushed a commit
to yzpang/gold-off-policy-text-gen-iclr21
that referenced
this pull request
Feb 19, 2021
Summary: Models seem to train fine with this modification. I checked that the mask for beginning of words is correct but didn't check if the actual masking worked correctly. Pull Request resolved: facebookresearch/fairseq#1292 Differential Revision: D18338307 Pulled By: myleott fbshipit-source-id: eae9e29d6ab648e768d70921694a898554496704
yfyeung
pushed a commit
to yfyeung/fairseq
that referenced
this pull request
Dec 6, 2023
Harleen8118
pushed a commit
to Harleen8118/IBERT
that referenced
this pull request
Jun 26, 2025
Summary: Models seem to train fine with this modification. I checked that the mask for beginning of words is correct but didn't check if the actual masking worked correctly. Pull Request resolved: facebookresearch/fairseq#1292 Differential Revision: D18338307 Pulled By: myleott fbshipit-source-id: eae9e29d6ab648e768d70921694a898554496704
caltia
pushed a commit
to caltia/fairseq
that referenced
this pull request
Jul 8, 2025
Summary: Models seem to train fine with this modification. I checked that the mask for beginning of words is correct but didn't check if the actual masking worked correctly. Pull Request resolved: facebookresearch/fairseq#1292 Differential Revision: D18338307 Pulled By: myleott fbshipit-source-id: eae9e29d6ab648e768d70921694a898554496704
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Models seem to train fine with this modification. I checked that the mask for beginning of words is correct but didn't check if the actual masking worked correctly.