Reject unimplemented BPE options instead of ignoring them silently - #2
Open
bjo4 wants to merge 1 commit into
Open
Reject unimplemented BPE options instead of ignoring them silently#2bjo4 wants to merge 1 commit into
bjo4 wants to merge 1 commit into
Conversation
BPE and BpeTrainer accepted a set of HuggingFace-compatible options, stored them, and in the model's case serialized them back out, without ever acting on any of them. A caller who set one got wrong tokenization with no signal, and a round-tripped tokenizer.json looked unchanged, so nothing appeared wrong. Options that would change tokenization now raise UnsupportedFeatureError: dropout in (0, 1], and non-empty continuing_subword_prefix / end_of_word_suffix. A dropout outside [0, 1] raises ModelError instead, since that is invalid rather than unimplemented, matching HuggingFace's own rejection at construction. Default values stay silent. None, 0.0 and "" are genuine no-ops in HuggingFace too, so files that tokenize correctly today keep loading -- GPT-2-lineage exports ship "" for both affix options. BpeTrainer gets the same treatment for its two affix options, which it also stored and never read. Fixing only the model would have left half of the same defect in place. cache_capacity was being discarded entirely, without even an attribute assignment. It is now retained, but not rejected: a cache is a pure optimization, so its absence cannot change output. HuggingFace does not serialize it into tokenizer.json, so to_dict() stays faithful by omitting it. The rejection rules live in a shared module because they must stay identical in the model and the trainer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1
BPEandBpeTraineraccept a set of HuggingFace-compatible options, storethem, and in the model's case serialize them back out, without ever acting on
any of them. Setting one gives you tokenization that differs from HuggingFace
with no signal, and a round-tripped
tokenizer.jsonlooks unchanged, so nothingappears wrong.
This PR makes the ones I am not implementing fail loudly. The four that change
token output —
unk_token,fuse_unk,byte_fallback,ignore_merges— areimplemented in the follow-up PR, which is stacked on this branch.
What changes
dropoutNone,0.0(0, 1]→UnsupportedFeatureError; outside[0, 1]→ModelErrorcontinuing_subword_prefixNone,""UnsupportedFeatureErrorend_of_word_suffixNone,""UnsupportedFeatureErrorDefaults stay silent on purpose.
None,0.0and""are genuine no-ops inHuggingFace too — I checked — so files that tokenize correctly today keep
loading, and GPT-2-lineage exports ship
""for both affix options. Anout-of-range
dropoutis invalid rather than unimplemented, which is why it isModelError; HuggingFace also rejects it at construction.BpeTrainergets the same treatment for its two affix options, which it alsostored and never read. Fixing only the model would have left half of the same
defect in place, and the inconsistency would be the first thing a reviewer noticed.
cache_capacitywas being discarded entirely —models/bpe.py:41accepted itwithout even an attribute assignment. It is now retained but not rejected,
because a cache is a pure optimization: its absence cannot change output. There
is a test asserting it stays out of
to_dict(), since HuggingFace does notserialize it into
tokenizer.jsoneither.The rejection rules live in
_validation.pyrather than being duplicated,because they have to stay identical in the model and the trainer.
Still stored and unread, deliberately out of scope
Listing these so the inventory is complete rather than looking like an oversight:
decoders/byte_level.py—trim_offsets,add_prefix_space,use_regexpre_tokenizers/byte_level.py—trim_offsetsThey are an offsets concern rather than a token-identity one, and they live in
different components. Happy to take them in a separate PR if you want them.
Testing
25 new tests in
tests/test_unsupported_flags.py, covering each option'saccepted and rejected values, the exception hierarchy, message quality, the
cache_capacityretention and serialization behaviour, and theTokenizer.from_dictpath.Full suite: 78 passed (53 before this change, 25 new). No existing test
changed.