We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
38a85b2
is_pretokenized
trim_offsets
Decoders
http
WordLevel
SentencePieceUnigramTokenizer
UnigramTrainer
Tokenizer.from_pretrained