Skip to content

Releases: mlverse/tok

v0.1.2

27 Jun 12:21
b315535
Compare
Choose a tag to compare
  • Added support for authentication. (#7)
  • Added support for training tokenizers from scratch. (#8)
  • Added support for serializing tokenizers. (#8)
  • Implemented support for configuring padding and truncation. (#9)
  • Added support for decoders, post_processors and normalizers. (#10)
  • Added support for get_vocab_size(). (#10)
  • Updated extendr version to avoid non-API function calls. (#11)

v0.1.1

18 Aug 12:56
7eeef0c
Compare
Choose a tag to compare
  • Dependencies are now vendored to avoid versioning problems on CRAN (#5)

v0.1.0

06 Jul 17:35
ff3d530
Compare
Choose a tag to compare
  • Initial CRAN submission.