Guard prompt tokens against context window overflow#113
Conversation
When visual context, clipboard, and prefix text combine to exceed the 2048-token context window, llama_decode() hits an out-of-bounds tensor access and ggml_abort crashes the app. This is easily triggered in Xcode where dense code fills the OCR capture. Both generate() and summarize() now cap prompt tokens to contextWindowTokens − maxPredictionTokens, trimming from the front to preserve the tail closest to the caret. Front-trimming also invalidates the KV prefix cache since the byte-level overlap no longer holds. Also fixes the Release build's Sparkle public key placeholder that prevented update checks from working. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
decodePrompt was feeding all prompt tokens to llama_decode in a single call. When the prompt exceeded 512 tokens (batchCapacity / n_batch), ggml hit an out-of-bounds tensor access and aborted. This is the actual crash trigger — the context window guard alone wasn't sufficient since 512 < 2048. Now decodes in chunks of batchCapacity, only requesting logits on the final token of the last chunk. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| private static func activeVisualContextSummary(rawSummary: String?) -> String? { | ||
| guard let rawSummary else { | ||
| return nil | ||
| } | ||
|
|
||
| let sanitizedSummary = PromptContextSanitizer.sanitize(rawSummary) | ||
| guard !sanitizedSummary.isEmpty, | ||
| PromptContextSanitizer.containsAlphanumericSignal(sanitizedSummary) | ||
| else { | ||
| return nil | ||
| } | ||
|
|
||
| return clippedText(trimmed, maxCharacters: maxClipboardContextCharacters) | ||
| return sanitizedSummary | ||
| } |
There was a problem hiding this comment.
activeClipboardContext caps the returned string via clippedText(sanitizedContext, maxCharacters: maxClipboardContextCharacters), but activeVisualContextSummary returns the sanitized summary without any character limit. If a caller ever passes a summary that bypasses ScreenshotContextGenerator's boundedSummaryText, the factory layer applies no cap, making the token guard in LlamaRuntimeCore the only safety net. Adding a character cap here would make the two context inputs symmetric.
| private static func activeVisualContextSummary(rawSummary: String?) -> String? { | |
| guard let rawSummary else { | |
| return nil | |
| } | |
| let sanitizedSummary = PromptContextSanitizer.sanitize(rawSummary) | |
| guard !sanitizedSummary.isEmpty, | |
| PromptContextSanitizer.containsAlphanumericSignal(sanitizedSummary) | |
| else { | |
| return nil | |
| } | |
| return clippedText(trimmed, maxCharacters: maxClipboardContextCharacters) | |
| return sanitizedSummary | |
| } | |
| private static let maxVisualContextSummaryCharacters = 2_000 | |
| private static func activeVisualContextSummary(rawSummary: String?) -> String? { | |
| guard let rawSummary else { | |
| return nil | |
| } | |
| let sanitizedSummary = PromptContextSanitizer.sanitize(rawSummary) | |
| guard !sanitizedSummary.isEmpty, | |
| PromptContextSanitizer.containsAlphanumericSignal(sanitizedSummary) | |
| else { | |
| return nil | |
| } | |
| return clippedText(sanitizedSummary, maxCharacters: maxVisualContextSummaryCharacters) | |
| } |
If maxPredictionTokens ever equals or exceeds contextWindowTokens, the subtraction goes zero or negative. Array.suffix with a negative value crashes, and suffix(0) leaves the KV cache empty for sampling. Addresses Greptile review feedback on both generate() and summarize(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
| let promptBytes = Array(prompt.utf8) | ||
| let contextRequest = PromptContextRequest( | ||
| promptBytes: Array(prompt.utf8), | ||
| promptBytes: promptBytes, | ||
| promptTokens: promptTokens, | ||
| samplingFingerprint: SamplingFingerprint(options: options), | ||
| cachedPrefixBytes: cachedPrefixBytes | ||
| cachedPrefixBytes: adjustedCachedPrefixBytes | ||
| ) |
There was a problem hiding this comment.
promptBytes stored in cache after a front-trim
When front-trimming fires, promptTokens is the suffix of allPromptTokens, but promptBytes is still set to the full Array(prompt.utf8). Both values are stored together in promptCache via rebuildPromptContext. On a subsequent request that doesn't overflow (e.g., the user deletes text), preparePromptContext computes a large confirmedCommonBytes from the full-bytes comparison, then falls through to a token-level commonPrefixCount between the stored trimmed-suffix tokens and the new full tokens. If even one token matches at position 0 (possible with shared prompt-template prefixes), reusableTokenCount > 0 and KV cache entries built from the trimmed-suffix tokens are treated as valid prefix state for the new full prompt, producing incorrect completions. The fix is to store an empty (or sentinel) promptBytes in the trim branch so no byte-prefix match can grant cache access.
1 participant
Summary
decodePromptwas feeding all prompt tokens tollama_decodein a single call. When prompts exceeded the 512-token batch limit (n_batch/n_ubatch), ggml hit an out-of-bounds tensor access andabort()ed. Now decodes in chunks ofbatchCapacity.contextWindowTokens − maxPredictionTokensbefore decoding, trimming from the front to preserve the tail closest to the caret. Applies to bothgenerate()andsummarize().maxPromptTokensis clamped to at least 1 so a pathologicalmaxPredictionTokens >= contextWindowTokensdoesn't crashArray.suffixor leave the KV cache empty.Context
When visual context (OCR up to 2000 chars), clipboard (1200 chars), and prefix text (1000 chars) combine into a large prompt, two things go wrong:
ggml_abort(the primary crash)ggml_abortXcode windows with dense code are a reliable trigger due to high OCR token density.
Test plan
xcodebuild -scheme tabby build)🤖 Generated with Claude Code
Greptile Summary
This PR addresses a hard crash caused by
llama_decode()receiving more tokens than the context window allows when visual context, clipboard, and prefix text combine to overflow the 2048-token limit. It also introduces a sharedPromptContextSanitizerto strip prompt-injection artifacts (ANSI escapes, separators, shell syntax) from clipboard and OCR inputs before they reach the prompt.max(1, contextWindowTokens − maxPredictionTokens)front-trim guard in bothgenerate()andsummarize(), keeping the tail of the prompt closest to the caret.decodePromptto process tokens inbatchCapacity-sized chunks, preventingllama_decodefrom exceedingn_batch/n_ubatchon a single call.PromptContextSanitizeras a pureSupport/helper and wires it intoClipboardContextProvider,ScreenshotContextGenerator, andSuggestionRequestFactory; includes two new factory tests covering ANSI and punctuation stripping.Confidence Score: 3/5
The crash fix and batch-chunking refactor are sound, but a cache-consistency gap in the trim path needs to be closed before merging.
When a front-trim fires, rebuildPromptContext stores the full original promptBytes alongside the trimmed promptTokens in promptCache. On any subsequent request that no longer overflows, preparePromptContext computes a large confirmedCommonBytes (full-bytes comparison succeeds), then falls through to a token-level commonPrefixCount between the stored trimmed-suffix tokens and the new full tokens. If even one token matches at position 0, reusableTokenCount > 0 and KV cache entries built from the trimmed-suffix are treated as valid prefix state for the new full prompt, producing incorrect completions.
tabby/Services/Runtime/LlamaRuntimeCore.swift — specifically the PromptContextRequest construction in the trim branch and how rebuildPromptContext stores that value in promptCache.
Important Files Changed
Reviews (2): Last reviewed commit: "Clamp maxPromptTokens to at least 1" | Re-trigger Greptile