Skip to content

Bug fix update to load the pretrained `TransfoXLModel` from s3, added fallback for OpenAIGPTTokenizer when SpaCy is not installed

Choose a tag to compare

View all tags
@thomwolf thomwolf released this 13 Feb 09:21
v0.5.1
4e56da3
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Mostly a bug fix update for loading the TransfoXLModel from s3:

  • Fixes a bug in the loading of the pretrained TransfoXLModel from the s3 dump (which is a converted TransfoXLLMHeadModel) in which the weights were not loaded.
  • Added a fallback of OpenAIGPTTokenizer on BERT's BasicTokenizer when SpaCy and ftfy are not installed. Using BERT's BasicTokenizer instead of SpaCy should be fine in most cases as long as you have a relatively clean input (SpaCy+ftfy were included to exactly reproduce the paper's pre-processing steps on the Toronto Book Corpus) and this also let us use the never_split option to avoid splitting special tokens like [CLS], [SEP]... which is easier than adding the tokens after tokenization.
  • Updated the README on the tokenizers options and methods which was lagging behind a bit.
Assets 2
Loading