Repository navigation
Removed
HfJsonError::InvalidByteLevel. No load returns it any more. Amatchthat names it no longer compiles.
Fixed
- A byte-level
tokenizer.jsonwhose vocabulary holds literal entries outsideadded_tokensnow loads. Cohere's holds 1403, such as™and emoji pieces. Encode never emits such an entry, and decode renders its literal text, astokenizersdoes. The load failed withInvalidByteLevelbefore, and that error variant is removed.
Full diff: v0.21.1...v0.22.0