Skip to content

Commit

Permalink
Tokenizers docs: Specify which class contains __call__ method (#14379)
Browse files Browse the repository at this point in the history
* Update tokenizer.rst

* Apply `make fixup`
  • Loading branch information
Xing Han Lu authored Nov 28, 2021
1 parent 69511cd commit ebbe8cc
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion docs/source/main_classes/tokenizer.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@ methods for using all the tokenizers:
- Managing special tokens (like mask, beginning-of-sentence, etc.): adding them, assigning them to attributes in the
tokenizer for easy access and making sure they are not split during tokenization.

:class:`~transformers.BatchEncoding` holds the output of the tokenizer's encoding methods (``__call__``,
:class:`~transformers.BatchEncoding` holds the output of the
:class:`~transformers.tokenization_utils_base.PreTrainedTokenizerBase`'s encoding methods (``__call__``,
``encode_plus`` and ``batch_encode_plus``) and is derived from a Python dictionary. When the tokenizer is a pure python
tokenizer, this class behaves just like a standard python dictionary and holds the various model inputs computed by
these methods (``input_ids``, ``attention_mask``...). When the tokenizer is a "Fast" tokenizer (i.e., backed by
Expand Down

0 comments on commit ebbe8cc

Please sign in to comment.