Skip to content

Commit

Permalink
More explicit error when failing to tensorize overflowing tokens (#5633)
Browse files Browse the repository at this point in the history
  • Loading branch information
LysandreJik committed Jul 9, 2020
1 parent b9d8af0 commit 3cc23ee
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions src/transformers/tokenization_utils_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,11 @@ def convert_to_tensors(self, tensor_type: Union[None, str, TensorType], prepend_

self[key] = tensor
except: # noqa E722
if key == "overflowing_tokens":
raise ValueError(
"Unable to create tensor returning overflowing tokens of different lengths. "
"Please see if a fast version of this tokenizer is available to have this feature available."
)
raise ValueError(
"Unable to create tensor, you should probably activate truncation and/or padding "
"with 'padding=True' 'truncation=True' to have batched tensors with the same length."
Expand Down

0 comments on commit 3cc23ee

Please sign in to comment.