vocab : keep DNA k-mer ids distinct from colliding BPE tokens

[kashif]

Overview

Follow-up to #23410. The HybridDNA tokenizer gives every DNA k-mer its own id, but one 6-mer (CCCCCC) also exists as a Qwen3 BPE token. Because get_vocab() is keyed by text, the DNA id (154402) was dropped in favor of the BPE id (91443) and written out as an unused pad — so <dna>…CCCCCC…</dna> encoded to the wrong id and 154402 detokenized to [PAD154402], diverging from the Python tokenizer.

A naive conversion fix can't work: llama.cpp's vocab is a 1:1 text↔id map, so two tokens named CCCCCC won't load. transformers avoids this by resolving k-mers through a dedicated DNA map in <dna> context. This PR does the same in src/llama-vocab.cpp only: inside <dna> a k-mer resolves to its own id by product-order index (not the shared text→id map), and at load the colliding k-mer's text is restored from its index so it detokenizes correctly.

Result matches transformers both ways: DNA CCCCCC → 154402, plain CCCCCC → 91443, both detokenize to CCCCCC. Verified with full token-id parity against AutoTokenizer(..., trust_remote_code=True).

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES for comments and refactoring

[kashif]

@CISC would something like this be a valid solution?

[CISC]

To not essentially reimplement the vocab I wonder if it would make more sense to f.ex. postfix the DNA tokens with a special character at conversion, which we can then easily strip from id_to_token without replacing the strings?

[kashif]

thanks for the suggestion @CISC let me try that

[kashif]

@CISC should be ready for intial review

[CISC]

Nice, but don't modify token_to_piece, simply update id_to_tokens DNA entries text by erasing the marker in-place (only for hybriddna tokenizer, similar to what you did before).

[kashif]

@CISC thanks for the suggestion. Fixed

[CISC]

I'll check it out later today (I added a test for CCCCCC in the vocab files).

Is it perhaps worth only checking the tokens after <oov>?

[kashif]

thanks @CISC checking! Also, hope you can get some sleep!!

[CISC]

thanks @CISC checking! Also, hope you can get some sleep!!

Sleep is overrated. :)

[CISC]

@kashif I guess update the GGUFs ASAP, I'll add the vocab test files.

[kashif]

Thanks @CISC, all weights updated and re-uploaded

[CISC]

BTW, I think you forgot to delete chat_template.jinja, all the GGUFs have chat templates.

[kashif]

opps! fixing

[kashif]

good catch @CISC removed and reuploaded weights

[CISC]

@ServeurpersoCom Thanks, holding for Release fix first.

[kashif]

thanks @CISC