Skip to content

v2025.11.24

@bauwenst bauwenst tagged this 24 Nov 16:23
- Added a base class Specials that forces special tokens to be defined as fields of (nested) dataclasses.
- Fixed long-time broken TokeniserFactory classes, removed any baked-in vocabularies, and added convertor from TkTkT specials to HF specials.
Assets 2
Loading