model going over max context

[acoliver]

What happened?

(main*) Context: 136,565/131,134 tokens | TPM: 45.00k | Wait: 0ms | 18:29:59
~/projects/llxprt/branch-1/llxprt-code [no sandbox (see /docs)] qwen3-next-80b-a3b-instruct-mlx | Tokens: 5,797,852 | ✖ 1 error (ctrl+o for details)

A compress was not triggered. its just going over and letting lmstudio handle it.

What did you expect to happen?

there is a compression ratio that should have fired long ago. Also, there is a guard that is supposed to prevent it from going way over.

Client information

v0.8.0-nightly.260102.9e1b7f4f6
was running lmstudio provider alias and the listed model

Login information

No response

Anything else we need to know?

No response

[acoliver]

@CodeRabbit look into this

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Search for compression-related code and context limit management
echo "=== Searching for compression ratio and trigger logic ==="
rg -n -C3 "compression.*ratio|compress.*trigger" --type-add 'code:*.{ts,js,py}' --type=code

echo -e "\n=== Searching for context limit guards and validation ==="
rg -n -C3 "context.*limit|max.*context|guard.*context" --type-add 'code:*.{ts,js,py}' --type=code

echo -e "\n=== Searching for token budget and overflow handling ==="
rg -n -C3 "over.*limit|exceed.*context|budget.*exceed" --type-add 'code:*.{ts,js,py}' --type=code

echo -e "\n=== Finding files related to context management ==="
fd -e ts -e js -e py . | rg -i "context|budget|compress|token"

Length of output: 153733

---

🏁 Script executed:

#!/bin/bash
# Find the enforceContextWindow implementation and compression trigger logic
echo "=== enforceContextWindow implementation ==="
ast-grep --pattern 'enforceContextWindow($$$) {
  $$$
}'

echo -e "\n=== Compression threshold check logic ==="
rg -n -A10 "COMPRESSION_TOKEN_THRESHOLD|compressionThreshold" packages/core/src/core/geminiChat.ts

echo -e "\n=== tokenLimit function implementation ==="
cat packages/core/src/core/tokenLimits.ts

echo -e "\n=== lmstudio provider configuration ==="
rg -n "lmstudio" packages/cli/src/providers/ --type-add 'code:*.{ts,js}' --type=code

Length of output: 3679

---

🏁 Script executed:

#!/bin/bash
# Get the full enforceContextWindow method implementation
echo "=== enforceContextWindow full implementation ==="
rg -n -A50 "async enforceContextWindow" packages/core/src/core/geminiChat.ts

echo -e "\n=== Compression check and trigger flow ==="
rg -n -B5 -A20 "shouldCompress\s*=" packages/core/src/core/geminiChat.ts

echo -e "\n=== getEffectiveTokenCount implementation ==="
rg -n -A15 "getEffectiveTokenCount\(\)" packages/core/src/core/geminiChat.ts | head -40

Length of output: 4053

---

🏁 Script executed:

#!/bin/bash
# Find TOKEN_SAFETY_MARGIN and compression call sites
echo "=== TOKEN_SAFETY_MARGIN definition ==="
rg -n "TOKEN_SAFETY_MARGIN\s*=" packages/core/src/core/geminiChat.ts

echo -e "\n=== ensureCompressionBeforeSend calls ==="
rg -n "ensureCompressionBeforeSend" packages/core/src/core/geminiChat.ts

echo -e "\n=== Send and stream methods that should call compression ==="
rg -n -B3 -A20 "async send\(|async stream\(" packages/core/src/core/geminiChat.ts | head -100

echo -e "\n=== Check for any conditions that skip compression ==="
rg -n -B5 -A5 "skip.*compress|bypass.*compress|isCompressing" packages/core/src/core/geminiChat.ts

Length of output: 473

---

🏁 Script executed:

#!/bin/bash
# Get the full ensureCompressionBeforeSend implementation
echo "=== ensureCompressionBeforeSend full implementation ==="
rg -n -A40 "private async ensureCompressionBeforeSend" packages/core/src/core/geminiChat.ts

echo -e "\n=== Check shouldTriggerCompression implementation ==="
rg -n -B10 -A30 "private shouldTriggerCompression" packages/core/src/core/geminiChat.ts

echo -e "\n=== Check if there are any early returns or bypasses ==="
rg -n -B5 -A10 "return.*before.*compress|skip.*compression|disable.*compression" packages/core/src/core/geminiChat.ts

echo -e "\n=== Look for isCompressing flag usage ==="
rg -n -C5 "this\.isCompressing" packages/core/src/core/geminiChat.ts

Length of output: 1619

---

🏁 Script executed:

#!/bin/bash
# Find shouldCompress method (not shouldTriggerCompression)
echo "=== shouldCompress method implementation ==="
rg -n -A30 "private shouldCompress\(" packages/core/src/core/geminiChat.ts

echo -e "\n=== Full flow from send() including compression and guard ==="
rg -n -A35 "^\s*async send\(" packages/core/src/core/geminiChat.ts | head -80

echo -e "\n=== Full flow from stream() including compression and guard ==="
rg -n -A35 "^\s*async stream\(" packages/core/src/core/geminiChat.ts | head -80

echo -e "\n=== Check cachedCompressionThreshold initialization ==="
rg -n "cachedCompressionThreshold" packages/core/src/core/geminiChat.ts | head -20

Length of output: 1923

---

🏁 Script executed:

#!/bin/bash
# Find contextLimit() implementation
echo "=== contextLimit implementation in runtime context ==="
rg -n -A10 "contextLimit\(\)" packages/core/src/runtime/createAgentRuntimeContext.ts

echo -e "\n=== Check compression threshold default ==="
rg -n "COMPRESSION_TOKEN_THRESHOLD|compressionThreshold\(\)" packages/core/src/core/compression-config.ts

echo -e "\n=== Look for UI context display and where 131,134 might come from ==="
rg -n "Context:.*tokens" packages/cli/src/ui/

echo -e "\n=== Check ephemerals methods in runtime context ==="
rg -n -B3 -A8 "compressionThreshold\(\)|contextLimit\(\)" packages/core/src/runtime/createAgentRuntimeContext.ts | head -60

Length of output: 598

---

🏁 Script executed:

#!/bin/bash
# Get Footer.tsx to see how limit is calculated
echo "=== Footer.tsx context limit display ==="
rg -n -B10 -A10 "Context:.*historyTokenCount" packages/cli/src/ui/components/Footer.tsx

echo -e "\n=== Find ephemerals structure definition ==="
rg -n -B5 -A15 "ephemerals.*:" packages/core/src/runtime/createAgentRuntimeContext.ts | head -80

echo -e "\n=== Check tokenLimit usage in codebase ==="
rg -n "tokenLimit\(" packages/core/src/core/geminiChat.ts

echo -e "\n=== Find where provider model info gets context limits ==="
rg -n "maxContextTokens|contextSize|context.*window" packages/core/src/providers/lmstudio/

Length of output: 1411

---

🏁 Script executed:

#!/bin/bash
# Find where Footer is used and where limit comes from
echo "=== Footer usage with limit prop ==="
rg -n -B5 -A5 "<Footer|ResponsiveContextDisplay" packages/cli/src/ui/ | grep -A5 -B5 "limit"

echo -e "\n=== Check how providers are structured ==="
fd -t d . packages/core/src/providers/ | head -20

echo -e "\n=== Look for model info and context window info ==="
rg -n "modelInfo|getModelInfo|maxTokens.*context" packages/core/src/providers/ --type-add 'code:*.ts' --type=code -g '!*.test.ts' | head -30

echo -e "\n=== Check runtime context ephemerals methods ==="
cat packages/core/src/runtime/createAgentRuntimeContext.ts | grep -A50 "ephemerals:"

Length of output: 1795

---

🏁 Script executed:

#!/bin/bash
# Get the full ephemerals object from runtime context
echo "=== Full createAgentRuntimeContext.ts file (ephemerals section) ==="
cat packages/core/src/runtime/createAgentRuntimeContext.ts

echo -e "\n=== Separator ==="
echo "=== END OF FILE ==="

Length of output: 4531

---

@acoliver I've investigated issue #982 and found multiple problems with the context management system. Here's what I discovered:

Root Causes

1. Compression Threshold Not Triggering

With context-limit=131,134:

• Compression should trigger at 85% = 111,464 tokens
• Guard should block at 131,134 - 256 = 130,878 tokens
• But the system reached 136,565 tokens (5,687 over!)

2. Critical Code Flow Issues

The context enforcement happens in this sequence:

// In geminiChat.ts line 1791
private async enforceContextWindow(pendingTokens, promptId, provider) {
  const completionBudget = this.getCompletionBudget(provider);
  const limit = tokenLimit(this.runtimeState.model, userContextLimit);
  const marginAdjustedLimit = limit - 256;
  
  const projected = this.getEffectiveTokenCount() + pendingTokens + completionBudget;
  
  if (projected <= marginAdjustedLimit) {
    return; // No enforcement needed
  }
  
  // Compress and check again...
}

3. The Bugs

Bug #1: shouldCompress() calculation uses ephemerals.contextLimit() which for unknown models returns DEFAULT_TOKEN_LIMIT (1,048,576) instead of the user-set value when compression threshold is cached:

// Line 1607 - cachedCompressionThreshold calculation
this.cachedCompressionThreshold = threshold * contextLimit;

If contextLimit() returns the wrong value on first call, the threshold is cached incorrectly for the entire session.

Bug #2: The guard in enforceContextWindow uses tokenLimit(model, userContextLimit) directly, but for the model "qwen3-next-80b-a3b-instruct-mlx", the tokenLimit() function has no case for it, so it returns DEFAULT_TOKEN_LIMIT (1,048,576) if userContextLimit is undefined.

Bug #3: Token counting happens asynchronously via historyService.waitForTokenUpdates(), creating race conditions where tokens can increase between the check and the actual send.

Bug #4: The model can return MORE tokens than max_tokens requested, causing overflow after the guard check passes.

4. Why It Happened

Looking at your provider alias (lmstudio with qwen model), the system likely:

1. Never properly initialized the compression threshold
2. Used default 1M token limit instead of your 131,134 setting
3. Let the request through because projected was calculated against the wrong limit
4. Model response added more tokens, pushing history over

Recommended Fixes

---

🏁 Script executed:

#!/bin/bash
# Create a focused investigation into the specific model and settings
echo "=== Check how lmstudio provider is configured ==="
rg -n "lmstudio|qwen" packages/cli/src/providers/providerAliases.ts

echo -e "\n=== Check if there are any provider-specific context limits ==="
rg -n "qwen.*context|qwen.*token.*limit" packages/ --type-add 'code:*.{ts,json}' --type=code

echo -e "\n=== Verify getCompletionBudget implementation ==="
rg -n -A10 "getCompletionBudget\(" packages/core/src/core/geminiChat.ts

Length of output: 1163

---

The Real Issues

After deeper analysis, the code logic appears correct, which means the bugs are more subtle:

Issue 1: Silent Failure in Compression/Guard

The error message says "A compress was not triggered" and "it's just going over and letting lmstudio handle it." This suggests:

• Either ensureCompressionBeforeSend() wasn't called
• Or enforceContextWindow() wasn't called
• Or both functions silently returned without enforcing

Issue 2: Possible Async Race Condition

// Line 703 in send() and 1042 in stream()
await this.ensureCompressionBeforeSend(prompt_id, pendingTokens, 'send');
// ... some code ...
await this.enforceContextWindow(pendingTokens, prompt_id, provider);

Between these calls, if the history service updates token counts asynchronously, the protection could be bypassed.

Issue 3: LMStudio Over-Generation

LMStudio (and other local providers) may return MORE tokens than max_tokens specifies, especially if the response completes a sentence. The guard calculates a completion budget, but if the actual generation exceeds that budget, the overflow happens.

Recommended Fixes

Fix 1: Add Guard Enforcement Logging

private async enforceContextWindow(...) {
  this.logger.warn('enforceContextWindow called', { 
    projected, 
    marginAdjustedLimit,
    currentTokens: this.getEffectiveTokenCount() 
  });
  
  if (projected <= marginAdjustedLimit) {
    this.logger.debug('Within limit, returning');
    return;
  }
  
  // ... rest of enforcement
}

Fix 2: Hard Block Instead of Just Compression

After compression fails, instead of throwing an error, the system should:

1. Truncate the oldest messages
2. Retry compression with more aggressive settings
3. Refuse to send if still over limit

Fix 3: Add Post-Send Validation

After receiving the response, verify the total context hasn't exceeded the limit, and trigger emergency compression if it has.

Fix 4: Guard Against Model Over-Generation

Increase the safety margin or add a post-generation check:

private static readonly TOKEN_SAFETY_MARGIN = 10000; // Much larger buffer

Immediate Debugging Steps

Run the application with debug logging to see where the enforcement is failing:

DEBUG=llxprt:* llxprt

Look for these log messages:

• "Compression needed"
• "enforceContextWindow called"
• "Projected token usage exceeds context limit"

If these logs are missing, the enforcement code isn't being reached, indicating a deeper architectural problem.

Suggested Investigation

Would you like me to:

1. Create a test case that reproduces this exact scenario?
2. Add comprehensive logging to track the token flow?
3. Implement the hard-block safety measures?
4. Investigate why LMStudio specifically might be bypassing these checks?

---

🧠 Learnings used

Learnt from: acoliver
Repo: vybestack/llxprt-code PR: 0
File: :0-0
Timestamp: 2025-11-21T23:05:45.164Z
Learning: Every provider example in quick-reference should include geometry budgeting: set both 'context-limit' (ephemeral) and 'max_tokens' (model param), with a safety margin note.

[acoliver]

@CodeRabbit even weirder: ───────────────╮
│ > /compress │
╰───────────────╯

✦ Nothing to compress.
█
╭──────────────────────────────────────────────╮ █
│ > how close are you to finding the answer? │ █
╰──────────────────────────────────────────────╯ █
█
█
INFO: Request cancelled. █

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ > Type your message or @path/to/file │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
(main*) Context: 245,030/131,134 tokens | TPM: 43.00k | Wait: 0ms | 21:09:04
~/projects/llxprt/branch-1/llxprt-code [no sandbox (see /docs)] qwen3-next-80b-a3b-instruct-mlx | Tokens: 8,561,839 | ✖ 1 error (ctrl+o for details

I told it to compress and even though it is 245030/131134 tokens...it said there was nothing to compress - it ha been running for many hours... just tool call after tool call (lmstudio is truncating on its side)...

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Find the /compress command implementation and "Nothing to compress" message
echo "=== Find 'Nothing to compress' message ==="
rg -n "Nothing to compress" packages/

echo -e "\n=== Find /compress command handler ==="
rg -n -B5 -A20 "compress.*command|handleCompress|/compress" packages/cli/src/commands/

echo -e "\n=== Find compress eligibility check in CLI ==="
rg -n -B10 -A20 "canCompress|shouldCompress|isCompressible" packages/cli/src/

echo -e "\n=== Check if there's a compression state flag ==="
rg -n "isCompressing|compressionInProgress|compressing.*flag" packages/core/src/core/geminiChat.ts

Length of output: 378

---

🏁 Script executed:

#!/bin/bash
# Get the context around the "Nothing to compress" logic
echo "=== Get compress() method implementation ==="
rg -n -B30 -A10 "Nothing to compress" packages/core/src/core/geminiChat.ts

echo -e "\n=== Find where compress command is called ==="
rg -n -B5 -A10 "\.compress\(\)|chat\.compress|geminiChat\.compress" packages/cli/src/

echo -e "\n=== Check CompressionMessage component ==="
cat packages/cli/src/ui/components/messages/CompressionMessage.tsx

echo -e "\n=== Check if there's a minimum messages requirement ==="
rg -n "minimum.*messages|messages.*length.*compress|history.*empty" packages/core/src/core/geminiChat.ts | head -20

Length of output: 4286

---

🏁 Script executed:

#!/bin/bash
# Find getCompressionSplit implementation
echo "=== getCompressionSplit implementation ==="
rg -n -A40 "getCompressionSplit\(\)" packages/core/src/core/geminiChat.ts

echo -e "\n=== Check recentMessagesCount configuration ==="
rg -n "recentMessagesCount|recent.*messages" packages/core/src/core/geminiChat.ts | head -20

echo -e "\n=== Check history service message retrieval ==="
rg -n "getHistory|getAllMessages|getMessages" packages/core/src/core/geminiChat.ts | head -20

echo -e "\n=== Find where /compress command triggers performCompression ==="
rg -n -B5 -A10 "performCompression|/compress" packages/cli/src/

Length of output: 12314

---

🏁 Script executed:

#!/bin/bash
# Get the compressCommand implementation
echo "=== compressCommand.ts implementation ==="
cat packages/cli/src/ui/commands/compressCommand.ts

echo -e "\n=== Check getCurated method in HistoryService ==="
rg -n -A20 "getCurated\(\)" packages/core/src/history/

echo -e "\n=== Check if curated history might be empty ==="
rg -n "curated.*length|getCurated" packages/core/src/core/geminiChat.ts | head -30

Length of output: 2933

---

🏁 Script executed:

#!/bin/bash
# Get the full adjustForToolCallBoundary method
echo "=== Full adjustForToolCallBoundary method ==="
rg -n -A50 "adjustForToolCallBoundary" packages/core/src/core/geminiChat.ts

echo -e "\n=== Find tryCompressChat method that /compress calls ==="
rg -n -A30 "tryCompressChat" packages/core/src/core/geminiChat.ts

echo -e "\n=== Check HistoryService getCurated implementation ==="
fd "HistoryService" packages/core/src/ --type f -x cat {}

Length of output: 83910

---

🏁 Script executed:

#!/bin/bash
# Find ContentValidation.hasContent implementation
echo "=== ContentValidation.hasContent implementation ==="
rg -n -A30 "hasContent" packages/core/src/history/ --type-add 'code:*.ts' --type=code

echo -e "\n=== Find tryCompressChat implementation ==="
rg -n -A40 "tryCompressChat" packages/core/src/core/geminiChat.ts

echo -e "\n=== Check preserveThreshold configuration ==="
rg -n "preserveThreshold" packages/core/src/

Length of output: 4626

---

🏁 Script executed:

#!/bin/bash
# Find ContentValidation implementation
echo "=== ContentValidation implementation ==="
rg -n -A30 "class ContentValidation|ContentValidation.*=" packages/core/src/ --type-add 'code:*.ts' --type=code

echo -e "\n=== Find hasContent method ==="
rg -n -A20 "hasContent.*function|hasContent.*=.*\(" packages/core/src/ --type-add 'code:*.ts' --type=code

echo -e "\n=== Find where tryCompressChat is defined ==="
rg -n "tryCompressChat.*\(" packages/core/src/ --type-add 'code:*.ts' --type=code

echo -e "\n=== Check if there's a manual compress command handler ==="
rg -n -A20 "manual.*compress|user.*compress|command.*compress" packages/core/src/core/geminiChat.ts

Length of output: 5464

---

🏁 Script executed:

#!/bin/bash
# Get the full tryCompressChat implementation from client.ts
echo "=== tryCompressChat implementation in client.ts ==="
rg -n -A80 "async tryCompressChat" packages/core/src/core/client.ts

echo -e "\n=== Check if there's any condition that prevents compression ==="
rg -n -B5 -A5 "NOOP|Nothing to compress" packages/core/src/core/

Length of output: 12815

---

@acoliver This is a critical bug in the compression boundary adjustment logic! I found the root cause:

The Problem

When you run /compress manually at 245,030/131,134 tokens, here's what happens:

1. getCompressionSplit() calculates split point (Line 1903):

   let splitIndex = Math.floor(curated.length * (1 - 0.2)); // Keep last 20%

2. adjustForToolCallBoundary() moves the boundary (Line 1906):
   This method is supposed to avoid splitting tool call/response pairs, BUT with your history being "just tool call after tool call" for hours, it keeps moving the split index forward:

   // Line 1927-1929: Skip over tool responses
   while (index < history.length && history[index].speaker === 'tool') {
     index++;  // Keeps moving forward!
   }
   
   // Line 1932-1960: Check for unmatched tool calls and move back

3. Result: With hours of continuous tool execution, nearly ALL your recent history is tool-related. The boundary adjustment pushes splitIndex so far forward that there's nothing left to compress!

4. Line 1909 check fails:

   if (splitIndex < 4) {
     return { toCompress: [], toKeep: curated };  // "Nothing to compress"
   }

Why This Happens

Your scenario is perfect storm:

• 🔴 245K tokens (186% over limit)
• 🔴 Hours of tool execution = history dominated by tool call/response pairs
• 🔴 20% preserve threshold = keeps last 20% of messages
• 🔴 Boundary adjustment = moves split forward to avoid breaking tool pairs
• 🔴 Result: Boundary gets pushed to near the end, leaving nothing to compress

The Bugs

Bug #1: Overly Aggressive Tool Boundary Protection

The adjustForToolCallBoundary() method is too conservative. It should allow compression of OLD tool call/response pairs, but currently it's preserving too much.

Bug #2: No Fallback When Boundary Adjustment Fails

When boundary adjustment leaves nothing to compress, the system should:

1. Try a more aggressive split (compress older tool pairs)
2. Or warn the user that compression can't help with current settings

Bug #3: Context Limit Not Enforced BEFORE Reaching 245K

You shouldn't have been able to reach 245K with a 131K limit. The earlier bug I identified (wrong limit being used) is why you got this far.

Recommended Fixes

Fix 1: Relax Tool Boundary Protection for Old Messages

private adjustForToolCallBoundary(history: IContent[], index: number): number {
  // If we're trying to compress more than 50% of history, allow splitting some tool pairs
  const compressionRatio = index / history.length;
  
  if (compressionRatio > 0.5) {
    // Only protect the most recent tool pairs
    const minProtectIndex = Math.floor(history.length * 0.3);
    if (index < minProtectIndex) {
      // Don't move forward anymore - allow compression of old tool pairs
      return index;
    }
  }
  
  // Normal boundary adjustment logic...
}

Fix 2: Add Emergency Compression Mode

if (toCompress.length === 0 && curated.length > 10) {
  this.logger.warn('Compression boundary adjustment left nothing to compress - using emergency mode');
  
  // Force compress at least 50% of history
  const emergencyIndex = Math.floor(curated.length * 0.5);
  return {
    toCompress: curated.slice(0, emergencyIndex),
    toKeep: curated.slice(emergencyIndex)
  };
}

Fix 3: Show Better Error Message

if (toCompress.length === 0) {
  if (curated.length > 0) {
    this.logger.warn('Cannot compress - history dominated by recent tool activity', {
      historyLength: curated.length,
      tokenCount: this.historyService.getTotalTokens()
    });
    throw new Error('Cannot compress history dominated by recent tool calls. Consider using /clear or reducing tool usage.');
  }
  this.logger.debug('Nothing to compress - history too short');
  return;
}

Immediate Workaround

For now, you can:

1. Run /clear to reset history
2. Cancel the long-running tool execution (Ctrl+C)
3. Restart with better context management:
   • Use shorter context-limit setting
   • Set max_tokens lower to reduce model output
   • Enable automatic compression at lower threshold

Would you like me to:

1. ✅ Create an issue to track this bug?
2. ✅ Generate a test case that reproduces this scenario?
3. ✅ Implement the emergency compression mode fix?
4. ✅ Add logging to diagnose why the guard didn't prevent overflow?

---

🧠 Learnings used

Learnt from: e2720pjk
Repo: vybestack/llxprt-code PR: 583
File: packages/core/src/providers/openai/OpenAIProvider.ts:935-959
Timestamp: 2025-11-16T22:51:26.374Z
Learning: In the llxprt-code codebase (packages/core/src/providers/openai/OpenAIProvider.ts), tools like `run_shell_command` use internal streaming only for real-time UI updates during execution, but each tool execution produces exactly ONE final `ToolResponseBlock` containing the full result. The streaming chunks are never sent to the LLM and are not converted into multiple tool messages. The OpenAI Chat Completions API requires that each tool call (tool_call_id) corresponds to exactly one message with role 'tool', so duplicate tool response detection that removes subsequent tool messages with the same tool_call_id is correct and necessary for API compliance.

---

Learnt from: acoliver
Repo: vybestack/llxprt-code PR: 0
File: :0-0
Timestamp: 2025-11-21T23:05:45.164Z
Learning: Every provider example in quick-reference should include geometry budgeting: set both 'context-limit' (ephemeral) and 'max_tokens' (model param), with a safety margin note.