-
Notifications
You must be signed in to change notification settings - Fork 26.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Unexpected overflowing_tokens in tokenizer.encode_plus #8028
Comments
I confirm the issue. It was ok with transformers 3.0.0, but from 3.1.0 it is changed. |
And the code: transformers/src/transformers/tokenization_utils_base.py Lines 2558 to 2571 in 6b4c617
looks bugged, despite above: ids = ids[:-1] should be ids = ids[:-window_len] .
|
* Exposing prepare_for_model for both slow & fast tokenizers * Update method signature * The traditional style commit * Hide the warnings behind the verbose flag * update default truncation strategy and prepare_for_model * fix tests and prepare_for_models methods Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>
Pinging @thomwolf |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Environment info
transformers
version: 3.4.0Who can help
tokenizers: @mfuntowicz
Information
When I am using BERT tokenizer, I get unexpected
overflowing_tokens
. Here is a example code to reproduce:To reproduce
The output is:
[1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 1038, 1039, 1040, 1041, 1042, 1043, 1044, 1045, 1046, 1047, 1048, 1049, 1048, 1047, 1046, 1045, 1044, 1043, 1042, 1041, 1040, 1039, 1038]
Expected behavior
The expected behavior I want is:
[1018, 1019, 1020, 1021, 1022, 1023, 1024, 1025, 1026, 1027, 1028, 1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 1038, 1039, 1040, 1041, 1042, 1043, 1044, 1045, 1046, 1047, 1048, 1049]
The current output contains
[1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 1038, 1039, 1040, 1041, 1042, 1043, 1044, 1045, 1046, 1047, 1048, 1049]
and an additional reversed Tensor of[1048, 1047, 1046, 1045, 1044, 1043, 1042, 1041, 1040, 1039, 1038]
, which I think is wrong.When I dig into the code, I find that:
transformers/src/transformers/tokenization_utils_base.py
Lines 2556 to 2564 in 6b4c617
I wonder why there is a for loop in it and I think I need
truncation_strategy = TruncationStrategy.ONLY_FIRST
. However, I failed to turn the truncation_stractegy toonly_first
because the code here turn the truncation strategy tolongest_first
.transformers/src/transformers/tokenization_utils_base.py
Lines 1750 to 1759 in 6b4c617
Can you give me any help?
The text was updated successfully, but these errors were encountered: