Skip to content

b12507

  • b12507
  • 7144124
  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
  • Choose a tag to compare

  • b12507
  • 7144124
  • Choose a tag to compare

  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
@danbev danbev tagged this 18 Nov 10:33
* vocab : extract bpe_rank loading into function

This commit extracts the BPE rank loading logic into a separate function
named `load_bpe_ranks`.

The motivation for this is that the came code is currently used by both
gpt2/hybriddna/whitespace tokenizer models and gemma4 and this avoids
the duplication of code to improve readability and maintainability.

* change signature of load_bpe_ranks
Assets 2
Loading