Repository navigation
b12507
* vocab : extract bpe_rank loading into function This commit extracts the BPE rank loading logic into a separate function named `load_bpe_ranks`. The motivation for this is that the came code is currently used by both gpt2/hybriddna/whitespace tokenizer models and gemma4 and this avoids the duplication of code to improve readability and maintainability. * change signature of load_bpe_ranks