Make _added_tokens_decoder the single source of truth for added-token state

[ishan-1010]

What this fixes

Fixes #47439. Two bugs and the structure that produced them.

Added tokens live in _added_tokens_decoder (source of truth) plus a hand-maintained _added_tokens_encoder cache, and the public properties rebuild sorted views on every access. Five model tokenizers mutate the decoder directly; cpmant and wav2vec2 forget the encoder, so convert_tokens_to_ids returns ids that no longer exist:

• cpmant: the popped </_> / </n> markers keep their old added ids in the cache; before Fix convert_tokens_to_ids performance regression for slow tokenizers (#46315) #46323 they fell through to unk. Restored that behavior.
• wav2vec2: after set_target_lang, a removed added token still encodes to its old id, which now decodes to an unrelated vocabulary token from the new language.

The change

• _sync_added_tokens() on the python backend: rebuilds the encoder cache from the decoder and drops the cached sorted view. Direct decoder mutations call it.
• added_tokens_encoder / added_tokens_decoder properties serve a cached sorted view instead of re-sorting per access. They still return a new dict per call, so existing code that mutates a returned dict (cpmant did) keeps working.
• Cache invalidation added at every decoder write: base __init__, the added_tokens_decoder setter, _add_tokens in both the python and sentencepiece backends.
• The five direct mutators (cpmant, wav2vec2, tapas, plbart, bartpho) go through the helper. The first two are the bug fixes; the other three were consistent but now cannot drift.

The fast (tokenizers) backend is untouched; it has its own added-token handling.

Numbers

Correctness is the point; the cached view is a side benefit: 500 property accesses with 4000 added tokens go from 420 ms to 159 ms (M1, CPU).

Tests

Three regression tests: cpmant marker consistency, wav2vec2 stale-entry after set_target_lang, and a base test that the views follow every mutation path (add_tokens, the setter, and mutation of a returned dict staying harmless).

python -m pytest tests/tokenization/test_tokenization_utils.py tests/models/{cpmant,wav2vec2,tapas,plbart,bartpho,byt5,ctrl,marian,siglip,m2m_100,speech_to_text}/test_tokenization_*.py
# remaining failures are identical on unmodified main (network/fixture dependent)

Notes

Not a duplicate: no open PR touches these paths. AI-assisted; I reviewed every line, wrote the repros, and ran the tests and benchmark above.

[Rocketknight1]

cc @itazap

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: bartpho, cpmant, plbart, tapas, wav2vec2

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 30189558582:2
Result: failure | Jobs: 16 | Tests: 174,846 | Failures: 2 | Duration: 16h 29m

[ishan-1010]

For whoever picks this up: the red shard is unrelated to this diff. tests_processors shard 2 failed with two httpx.RemoteProtocolError: Server disconnected errors in tests/models/idefics2/test_processing_idefics2.py (911 passed, 2 errors), which is a network drop while fetching a fixture, not a tokenizer failure. The diff here touches only tokenization files and all 8 tests_tokenization shards plus tests_custom_tokenizers are green.

I cannot re-run the job from my side. Happy to rebase to force a fresh run if that is easier than a re-run.

[itazap]

I'm not sure if the sync function is what we're looking for here, there should be a direct way to fix the used _added_tokens_decoder / _added_tokens_encoder in the models that touch them

[ishan-1010]

The catch is that the properties read a cached sorted view now (L479 and L487), so anything that mutates _added_tokens_decoder has to clear that cache too, not just update _added_tokens_encoder. Inlining that at each site is two statements instead of one call, so it's easier to get wrong, not harder.

If the cache goes as well then direct fixes are enough, and then it's only the two that are actually broken. cpmant L151 pops self.added_tokens_encoder, which is the property's throwaway dict, so the real one keeps the marker ids. wav2vec2 L202 only deletes from the decoder. tapas, plbart and bartpho are already consistent on main so they'd come out of the PR.

That loses the per-access re-sort, which wasn't the point here anyway. I'll push it that way unless you'd rather keep the cache and inline the invalidation.