Skip to content

[YANKED] Rust v0.12.0
Compare
Choose a tag to compare
@Narsil Narsil released this 31 Mar 09:10
· 217 commits to main since this release
v0.12.0
0eb7455
This commit was created on GitHub.com and signed with GitHub鈥檚 verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

[0.12.0]

Bump minor version because of a breaking change.

The breaking change was causing more issues upstream in transformers than anticipated:
huggingface/transformers#16537 (comment)

The decision was to rollback on that breaking change, and figure out a different way later to do this modification

  • [#938] Breaking change. Decoder trait is modified to be composable. This is only breaking if you are using decoders on their own. tokenizers should be error free.

  • [#939] Making the regex in ByteLevel pre_tokenizer optional (necessary for BigScience)

  • [#952] Fixed the vocabulary size of UnigramTrainer output (to respect added tokens)

  • [#954] Fixed not being able to save vocabularies with holes in vocab (ConvBert). Yell warnings instead, but stop panicking.

  • [#961] Added link for Ruby port of tokenizers

  • [#960] Feature gate for cli and its clap dependency

Assets 2