Generate: consistently handle special tokens as tensors #30624
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
cc @zucchini-nlp this one is the same as #29788, which you've already reviewed in its early state. Note that because we've merged other PRs first (e.g. removing the decoding functions from the public API), the diff is much smaller 💛 |
ArthurZucker
reviewed
May 6, 2024
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
|
@ArthurZucker addressed your comments :D (let's see this fast CI going brr) |
ArthurZucker
approved these changes
May 9, 2024
Comment on lines
420
to
+421
| torch.zeros(input_ids.shape[:2], dtype=torch.int64, layout=input_ids.layout, device=input_ids.device) | ||
| + model._get_decoder_start_token_id() | ||
| + generation_config.decoder_start_token_id |
There was a problem hiding this comment.
do we have to do the + and not just use the decoder start toekn id?
There was a problem hiding this comment.
I think the + here makes it a tensor of all decoder_start_token_id (as opposed to concatenation).
zucchini-nlp
pushed a commit
to zucchini-nlp/transformers
that referenced
this pull request
May 10, 2024
…30624) * tmp commit * [test_all] mvp * missing not * [test_all] final test fixes * fix musicgen_melody and rag * [test_all] empty commit * PR comments * Update src/transformers/generation/utils.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
itazap
pushed a commit
that referenced
this pull request
May 14, 2024
* tmp commit * [test_all] mvp * missing not * [test_all] final test fixes * fix musicgen_melody and rag * [test_all] empty commit * PR comments * Update src/transformers/generation/utils.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
This was referenced May 28, 2024
eginhard
added a commit
to idiap/coqui-ai-TTS
that referenced
this pull request
Jun 16, 2024
….41.1 Fixes #31. The handling of special tokens in `transformers` was changed in huggingface/transformers#30624 and huggingface/transformers#30746. This updates the XTTS streaming code accordingly.
eginhard
added a commit
to idiap/coqui-ai-TTS
that referenced
this pull request
Jun 16, 2024
….41.1 Fixes #31. The handling of special tokens in `transformers` was changed in huggingface/transformers#30624 and huggingface/transformers#30746. This updates the XTTS streaming code accordingly.
eginhard
added a commit
to idiap/coqui-ai-TTS
that referenced
this pull request
Jun 17, 2024
….41.1 Fixes #31. The handling of special tokens in `transformers` was changed in huggingface/transformers#30624 and huggingface/transformers#30746. This updates the XTTS streaming code accordingly.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
(reopened from #29788, requirements were merged and made this PR simpler)
To enable
torch.compilewithgenerate, some special token-related operations have to be rewritten into torch operations. That requires special tokens to be tensors instead of integers or a list of integers. (See #29374 for a working prototype)This PR reworks special token usage in
generateto consistently treat them as a tensor, as opposed to e.g. keeping track ofeos_token_idin integer and in tensor form.👉 Review suggestion: start by reading
_prepare_special_tokensand how it fits ingenerate.Tests ran locally:
pytest --doctest-modules src/transformers/generation/logits_process.py -vv)pytest --doctest-modules src/transformers/generation/utils.py -vv)RUN_SLOW=1 py.test tests/generation/ -vv)RUN_SLOW=1 py.test tests/test_cache_utils.py -vv)RUN_SLOW=1 py.test tests/models/llama/test_modeling_llama.py -vv)RUN_SLOW=1 py.test tests/models/whisper/test_modeling_whisper.py -vv) -- same failures as inmain