Skip to content

splintr 0.21.1

Choose a tag to compare

View all tags
@github-actions github-actions released this 11 Oct 12:06
· 5 commits to main since this release
v0.21.1
3e35bda

Added

  • SpecialPolicy::with_leading_bos and AnyTokenizer::with_leading_bos set, replace or remove the token that encode prepends. Tokens after the content, the pair template and the EOS id stay as they are. A caller whose model files state BOS apart from the vocabulary uses it, such as a tokenizer_config.json add_bos_token beside a bundled vocabulary.
  • SpecialPolicy::leading_special returns the special token the single-sequence template places before the content.

Full diff: v0.21.0...v0.21.1

Assets 11
Loading