You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Strip tokenizer padding before inference (served scores were wrong)
A tokenizer.json with a `padding` section — which real HuggingFace repos ship,
including our own published phishing classifiers — makes the Rust tokenizer pad
every encoding to a fixed width. transformers does not pad by default. We
forwarded those [PAD] ids to the model under an all-ones attention mask, so it
attended to ~500 padding tokens as if they were text.
Impact: every score served for the published catalog models was wrong. On the
probe 'Account notice: sign in to review recent activity.' the real model
returned LABEL_0=0.0005 where HuggingFace returns 0.7474. Affects 0.2.0-0.3.2
(0.3.2 was never released).
- drop trailing pad ids (pad_id read from tokenizer.json) before building the
inputs, reproducing the HF reference exactly (now 0.7474327 vs 0.747434)
- new tiny-pad fixture: tiny-clf with padding enabled in tokenizer.json; its
golden scores are tiny-clf's, so any regression that lets pad ids through
fails immediately. The existing fixtures could not catch this — their
tokenizers declare no padding, which is why it shipped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>