Skip to content

Local LLM Limitation mitigations

Jump to bottom
Justus Brugman edited this page Aug 22, 2026 · 6 revisions

When running narrative generation on local, mid-weight LLMs (such as Gemma 26B/12B or Qwen 30B), application code must actively compensate for foundational transformer architecture limits.

This document maps the Big 5 Local LLM Limitations directly to the design patterns implemented inside Storyteller.

Overview Matrix

Limitation Class Severity in Storytelling Storyteller Status Primary Mitigation Mechanism
1. Entity Contagion & Drift High ⚠️ Partially Mitigated Structured canonical-state.yaml & synchronous validator evaluation (ALLOW / REPLACE).
2. Lexical Corruption & Quantization Noise Medium ❌ Not Mitigated (Backend Dependent) Left to model quantization choice (Q6_K / Q8_0) and backend decoding settings.
3. Context Degradation ("Lost in the Middle") High ✅ Actively Mitigated Multi-layer memory shaping (summary.md, recent-summary.md) & single-system-prompt stacking.
4. Instruction Drift & Guardrail Collapse Critical ⚠️ Best-Effort Mitigation Periodic cache-busting resets (cacheBuster.interval), turn-level system injections, and JSON Schema constrained validation.
5. Context Saturation & Overhead Critical ✅ Actively Mitigated Fixed turn-window limits (chat.maxRecentTurns) & the sequential DerivedMemoryTaskQueue.

1. Entity Contagion & Feature Bleeding

The Problem

In mid-weight local models, attributes associated with one character decay across narrative distance and accidentally bind to another active character in the scene ("entity bleeding").

  • Example: The prompt defines Chris as the guitarist and Mike as a childhood friend who does not play music. After a scene set at a music venue, the LLM outputs:

    "Mike unstrapped his guitar and leaned it against the amplifier."

Storyteller Implementation: ⚠️ Partial Mitigation

Storyteller limits entity bleeding through state boundary enforcement, but cannot alter the underlying transformer forward-pass logic:

  1. Explicit Canon Context: PromptAssemblyService injects fixed_protagonists.yml and canonical-state.yaml directly into the system prompt to anchor character identities.
  2. Validator Guardrail: When enabled (validation.enabled=true), ResponseGuard routes the draft through a secondary validator pass. If the validator detects a breach of canonical character constraints, it issues a REPLACE decision to override the invalid turn before display.
  • Remaining Risk: Highly dense multi-character scenes in a single generation pass can still trigger momentary attribute bleeding if the validator misses subtle context cues.

2. Lexical Corruption & Quantization Noise

The Problem

Aggressive weight quantization (such as 4-bit or 6-bit K-quants) blurs the distance between subword embeddings. This causes token-level hallucinations or gibberish word completions.

  • Example: Instead of generating "She furrowed her eyebrows in frustration," a quantized 4-bit model might generate:

    "She furrowed her highbrows in frustration" or "She furrowed her eyebrows wish."

Storyteller Implementation: ❌ Not Mitigated

Storyteller currently relies on raw text output for the primary story generation pipeline:

  • Current Architecture: ResponseSanitizer handles formatting and escape cleanups, but does not perform dictionary parsing or spelling/grammar correction on story text.
  • Recommended Setup Workaround: To avoid lexical noise, run larger models at Q6_K or Q8_0 precision on high-unified-memory machines (e.g., Apple Silicon 64 GB), or lower decoding temperature (0.2–0.5) at the backend level.

3. Context Degradation ("Lost in the Middle")

The Problem

Transformers exhibit a U-shaped attention curve: tokens placed in the middle of long prompts (e.g., tokens 10,000 to 25,000 in a 32K context) suffer from exponential retrieval decay (Needle in a Haystack).

  • Example: Important rules or plot events established 15 turns ago get ignored because they sit in the low-attention center of the context window.

Storyteller Implementation: ✅ Active Mitigation

Storyteller completely avoids sending raw, unmanaged turn histories to the model:

  • Memory Hierarchy: Long-term events are continuously condensed into summary.md and recent-summary.md.
  • Single System Message Stacking: All active state and background context are assembled into a single combined first system message. This ensures critical world constraints remain pinned at the very top of the context window where attention weights are highest.

4. Instruction Drift & Guardrail Degradation

The Problem

During extended roleplay or storytelling sessions, accumulated narrative context can reduce the model’s attention to earlier system instructions. As the conversation grows, constraints such as “Never speak as the user” may be followed less consistently.

  • Example: The model may begin writing dialogue for the user’s protagonist or stop following the required character formatting.

Storyteller Implementation: ⚠️ Best-Effort Mitigation

Because model output is probabilistic, instruction drift cannot be eliminated entirely at the application layer. Storyteller uses several best-effort safeguards to detect, limit, and recover from drift:

  1. Periodic Cache-Buster Requests (cacheBuster.interval): After every N persisted story turns (default: 5), Storyteller sends a transient reset request with a unique token prepended to the system prompt. The response is discarded, and the request is not added to history.json.

    The changed prefix prevents exact-prefix cache reuse for that individual request on cache-sensitive OpenAI-compatible backends. It does not flush the backend’s cache, modify persistent model state, or guarantee that later requests will follow the rules more closely.

  2. User-Triggered Resets: Ctrl-W sends the same transient reset request without modifying history.json. Ctrl-U removes the most recent persisted turn, sends the reset request, and restores the removed user prompt to the input buffer for editing and retrying.

  3. Structured Validator Output: When validation is enabled, ValidationClient requests a constrained JSON Schema response containing an ALLOW or REPLACE decision.

    With validation.outputMode=auto, schema-constrained output is used when the backend supports it. If the backend rejects structured output, Storyteller falls back to tolerant text parsing for the remainder of the active session. Invalid, empty, or unparseable validator responses are handled using the configured fail-closed response.

  • Remaining Risk: These safeguards reduce the impact of instruction drift but cannot guarantee that it will not occur. Cache-buster requests do not constitute a documented KV-cache flush, and schema enforcement constrains only the validator’s decision format—not the original story generation. In long sessions, users may still need to use Ctrl-W or Ctrl-U when drift becomes visible.

5. Context Saturation & Performance Degradation

The Problem

As prompt size approaches the maximum context limit (e.g., 32K–65K tokens), Key-Value (KV) cache memory consumption explodes. This causes severe Time-To-First-Token (TTFT) latency spikes and hardware swapping.

  • Example: Running simultaneous background summarization requests alongside user-facing generation causes local GPU/VRAM to thrash, dropping generation speed to under 1 token/sec.

Storyteller Implementation: ✅ Active Mitigation

Storyteller treats local compute constraints as a core design boundary:

  1. Strict Context Window Bounding: chat.maxRecentTurns (default 2) restricts raw narrative history to a short, high-speed window.
  2. Sequential Derived-Memory Queue (DerivedMemoryTaskQueue): When a story turn completes, three background memory updates are triggered (summary, recent-summary, canonical-state). To prevent three concurrent LLM inference calls from overwhelming local VRAM/CPU resources, Storyteller submits these jobs to a single-threaded daemon queue.
  3. Non-Blocking Architecture: Background memory updates run asynchronously and sequentially without blocking the user interface or triggering parallel backend contention.

Clone this wiki locally