Widen the empty-array probe to merged item-start tokens - #212
Merged
mattt merged 2 commits intoSep 4, 2026
Conversation
Byte-pair vocabularies put most item-start probability on merged tokens, such as a quote fused with the first word or carrying a leading space. The probe offered only the bare single-character tokens, so the comparison against the closing bracket was noise and small models closed arrays the prompt asked them to fill. Admit every token whose first non-whitespace character starts the item type, and treat any token that trims to the closing bracket as a close.
There was a problem hiding this comment.
🟡 Changes recommended
The probe-widening logic is applied to string/object/array but boolean/number item-start tokens remain inconsistent with the new “first non-whitespace” approach, which can keep the probe noisy for those array element types.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR improves the “empty array” decision probe in ConstrainedJSONGenerator so models using BPE vocabularies can more reliably choose between closing [] vs starting the first element by accounting for merged/whitespace-prefixed tokens.
Changes:
- Expands the close decision to treat any token that trims to
]as a close candidate. - Broadens item-start token sets for
.string/.object/.arrayby including tokens whose first non-whitespace character is the relevant structural prefix. - Adds helper utilities to derive token sets (
tokensMatchingTrimmed,tokensStarting(with:)) used by the probe.
File summaries
| File | Description |
|---|---|
| Sources/AnyLanguageModel/Shared/StructuredGeneration.swift | Widens empty-array probe token candidates for both close (]) and item-starts to better reflect BPE merged tokens. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
638
to
642
| case .array: | ||
| return [try Self.singleToken(for: "[", backend: backend)] | ||
| return tokensStarting(with: "[") | ||
| case .boolean: | ||
| var tokens = Set<Int>() | ||
| for literal in ["true", "false"] { |
Comment on lines
591
to
+602
| private mutating func sampleWhetherToCloseEmptyArray( | ||
| items: GenerationSchema.Node | ||
| ) async throws -> Bool { | ||
| let closeToken = try Self.singleToken(for: "]", backend: backend) | ||
| let closeTokens = tokensMatchingTrimmed("]") | ||
| var allowed = try itemStartTokens(for: items) | ||
| allowed.insert(closeToken) | ||
| guard !allowed.isEmpty else { | ||
| allowed.formUnion(closeTokens) | ||
| guard !allowed.isEmpty, !closeTokens.isEmpty else { | ||
| return false | ||
| } | ||
| let token = try await backend.sample(from: allowed) | ||
| return token == closeToken | ||
| return closeTokens.contains(token) | ||
| } |
Comment on lines
+604
to
+610
| /// Tokens whose text equals `text` after trimming surrounding whitespace. | ||
| private func tokensMatchingTrimmed(_ text: String) -> Set<Int> { | ||
| var tokens = Set<Int>() | ||
| for token in 0 ..< backend.vocabSize { | ||
| if backend.isSpecialToken(token) { continue } | ||
| guard let tokenText = backend.tokenText(token) else { continue } | ||
| if tokenText.trimmingCharacters(in: .whitespacesAndNewlines) == text { |
Boolean starts admit any token that is a prefix of true or false after leading whitespace, and number starts look at the first non-whitespace character, matching the string, object, and array cases. Adds a test with merged item-start and close tokens.
Collaborator
|
Merging now. I pushed a follow-up that applies the same rule to boolean and number items and adds a test with merged tokens. Caching the vocab scans can be a follow-up if it shows up in profiles. Thanks, @james-333i! |
james-333i
added a commit
to james-333i/AnyLanguageModel
that referenced
this pull request
Sep 4, 2026
Brings in the merged huggingface#195, huggingface#205, huggingface#212 and huggingface#217 along with the follow-ups applied on merge (all-text prompt drop, boolean and number probe items). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Byte-pair vocabularies put most item-start probability on merged tokens, such as a quote fused with the first word or carrying a leading space. The probe offered only the bare single-character tokens, so the comparison against the closing bracket was noise and small models closed arrays the prompt asked them to fill.
Admit every token whose first non-whitespace character starts the item type, and treat any token that trims to the closing bracket as a close.