Preserve CLS/SEP for cls/sep-style tokenizers on transformers v5

[yuriihavrylko]

Addresses the remaining (ModernBERT-family) case of #324.

Problem

#326 fixed the transformers-v5 "uniform low scores" issue for spm-based backbones by setting add_bos_token/add_eos_token in _set_tokenizer_spec_tokens. That helper assigns:

tokenizer.add_bos_token = tokenizer.bos_token_id is not None

On transformers 4.x, cls/sep-style tokenizers don't expose add_bos_token, so the hasattr gate skips them. On transformers v5 the unified tokenizer backend exposes these attributes on every tokenizer — and for cls/sep-style tokenizers (ModernBERT, ettin, etc.) bos_token_id is None, so the helper assigns False. On v5 that assignment rebuilds the backend post-processor, which strips [CLS]/[SEP] from all encodings:

tok = AutoTokenizer.from_pretrained("knowledgator/modern-gliner-bi-base-v1.0")
tok(["Paris"], is_split_into_words=True)["input_ids"]   # [50281, 36062, 50282]
tok.add_bos_token = tok.bos_token_id is not None         # False
tok(["Paris"], is_split_into_words=True)["input_ids"]   # [36062]  <- CLS/SEP gone

The model then sees inputs without special tokens and returns uniformly near-zero scores — no crash, no warning. Affected checkpoints include knowledgator/modern-gliner-bi-base/large-v1.0 and knowledgator/gliner-pii-edge-v1.0 (weights, config and rope translation are all fine — we verified tensor-identical weight loading and identical raw-tokenizer output before isolating this).

Fix

Only opt in to bos/eos when the tokenizer actually defines them; never assign False. This preserves the #326 behavior for spm models (ids exist → set True) and leaves cls/sep tokenizers' post-processors intact.

Verification (transformers 5.6.2, this branch)

Test input: "Marie Curie worked at the University of Paris in France." with labels ["person", "organization", "location"], threshold 0.3.

model	before	after
modern-gliner-bi-base-v1.0	[] (all scores ≤0.01)	[('Marie Curie','person',0.85), ('University of Paris','organization',0.88), ('France','location',0.45)] — identical to gliner 0.2.16 + transformers 4.48.3
gliner-pii-edge-v1.0	[]	correct PII extractions, identical to the 0.2.16/4.48.3 reference
urchade/gliner_small-v2.1 (DeBERTa/spm — the #326 case)	works	works — no regression
mmBERT-backbone bi-encoder (bos/eos ids defined)	works	works — unchanged

[Ingvarstep]

@yuriihavrylko , awesome catch, thank you for fixing it!