Skip to content

Commit

Permalink
fix deprecated ref to tokenizer.max_len (#10220)
Browse files Browse the repository at this point in the history
This is to fix deprecated reference to `tokenizer.max_len` with `tokenizer.model_max_length` - similar to [issue 8739](#8739) and [PR 8604](#8604). 
Example [here](https://colab.research.google.com/gist/poedator/f8776349e5c625ce287fc6fcd312fa1e/tokenizer-max_len-error-in-transformers_glue.ipynb). The error happens when `glue_convert_examples_to_features` is called without `max_length` parameter specified. In that case line 119 with wrong reference gets called. This simple fix should  do it.
  • Loading branch information
poedator committed Feb 24, 2021
1 parent cdcdd5f commit 5f2a3d7
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/transformers/data/processors/glue.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ def _glue_convert_examples_to_features(
output_mode=None,
):
if max_length is None:
max_length = tokenizer.max_len
max_length = tokenizer.model_max_length

if task is not None:
processor = glue_processors[task]()
Expand Down

0 comments on commit 5f2a3d7

Please sign in to comment.