Skip to content

Local LLM Limitation mitigations

Jump to bottom
Justus Brugman edited this page Aug 22, 2026 · 6 revisions

When running narrative generation on local, mid-weight LLMs (such as Gemma 26B/12B or Qwen 30B), application code must actively compensate for foundational transformer architecture limits.

This document maps the Big 5 Local LLM Limitations directly to the design patterns implemented inside Storyteller.

Overview Matrix

Limitation Class Severity in Storytelling Storyteller Status Primary Mitigation Mechanism
1. Entity Contagion & Drift High ⚠️ Partially Mitigated Structured canonical-state.yaml & synchronous validator evaluation (ALLOW / REPLACE).
2. Lexical Corruption & Quantization Noise Medium ❌ Not Mitigated (Backend Dependent) Left to model quantization choice (Q6_K / Q8_0) and backend decoding settings.
3. Context Degradation ("Lost in the Middle") High ✅ Actively Mitigated Multi-layer memory shaping (summary.md, recent-summary.md) & single-system-prompt stacking.
4. Instruction Drift & Guardrail Collapse Critical ✅ Actively Mitigated Periodic cache-busting resets (cacheBuster.interval), turn-level system injections, and JSON Schema constrained validation.
5. Context Saturation & Overhead Critical ✅ Actively Mitigated Fixed turn-window limits (chat.maxRecentTurns) & the sequential DerivedMemoryTaskQueue.

1. Entity Contagion & Feature Bleeding

The Problem

In mid-weight local models, attributes associated with one character decay across narrative distance and accidentally bind to another active character in the scene ("entity bleeding").

  • Example: The prompt defines Chris as the guitarist and Mike as a childhood friend who does not play music. After a scene set at a music venue, the LLM outputs:

    "Mike unstrapped his guitar and leaned it against the amplifier."

Storyteller Implementation: ⚠️ Partial Mitigation

Storyteller limits entity bleeding through state boundary enforcement, but cannot alter the underlying transformer forward-pass logic:

  1. Explicit Canon Context: PromptAssemblyService injects fixed_protagonists.yml and canonical-state.yaml directly into the system prompt to anchor character identities.
  2. Validator Guardrail: When enabled (validation.enabled=true), ResponseGuard routes the draft through a secondary validator pass. If the validator detects a breach of canonical character constraints, it issues a REPLACE decision to override the invalid turn before display.
  • Remaining Risk: Highly dense multi-character scenes in a single generation pass can still trigger momentary attribute bleeding if the validator misses subtle context cues.

2. Lexical Corruption & Quantization Noise

The Problem

Aggressive weight quantization (such as 4-bit or 6-bit K-quants) blurs the distance between subword embeddings. This causes token-level hallucinations or gibberish word completions.

  • Example: Instead of generating "She furrowed her eyebrows in frustration," a quantized 4-bit model might generate:

    "She furrowed her highbrows in frustration" or "She furrowed her eyebrows wish."

Storyteller Implementation: ❌ Not Mitigated

Storyteller currently relies on raw text output for the primary story generation pipeline:

  • Current Architecture: ResponseSanitizer handles formatting and escape cleanups, but does not perform dictionary parsing or spelling/grammar correction on story text.
  • Recommended Setup Workaround: To avoid lexical noise, run larger models at Q6_K or Q8_0 precision on high-unified-memory machines (e.g., Apple Silicon 64 GB), or lower decoding temperature (0.2–0.5) at the backend level.

3. Context Degradation ("Lost in the Middle")

The Problem

Transformers exhibit a U-shaped attention curve: tokens placed in the middle of long prompts (e.g., tokens 10,000 to 25,000 in a 32K context) suffer from exponential retrieval decay (Needle in a Haystack).

  • Example: Important rules or plot events established 15 turns ago get ignored because they sit in the low-attention center of the context window.

Storyteller Implementation: ✅ Active Mitigation

Storyteller completely avoids sending raw, unmanaged turn histories to the model:

  • Memory Hierarchy: Long-term events are continuously condensed into summary.md and recent-summary.md.
  • Single System Message Stacking: All active state and background context are assembled into a single combined first system message. This ensures critical world constraints remain pinned at the very top of the context window where attention weights are highest.

4. Instruction Drift & Guardrail Collapse

The Problem

During extended roleplay or storytelling, autoregressive momentum causes narrative prose to overwhelm static system rules. Negative constraints ("Never speak as the user") collapse after 10–20 turns.

  • Example: The model gradually forgets its system rules and begins outputting dialogue for the user's protagonist or breaking character formatting.
graph TD
    A["Main System Prompt + Fixed Protagonists + Canonical State + Long-Term Summary + Recent Summary"] --> B["Last N Raw Turns (chat.maxRecentTurns)"]
    B --> C["Latest User Prompt"]
Loading

Storyteller Implementation: ✅ Active Mitigation

Storyteller employs a multi-tiered defense against instruction drift:

  1. Periodic Cache-Busting Resets (cacheBuster.interval): Every $N$ turns (default 5), Storyteller sends a transient request with a unique token prepended to the prompt. This deliberate prefix change forces prefix-caching backends (like LM Studio or llama.cpp) to re-evaluate system prompt rules rather than relying on stale cached attention states.
  2. User-Triggered Manual Resets: Shortcuts like Ctrl-W and Ctrl-U trigger an immediate transient reset-with-cache-buster to re-anchor model attention without corrupting history.json.
  3. JSON Schema Validator Enforcement: The validator pass (ValidationClient) enforces structured output via JSON Schema (validation.outputMode=auto). Constraining the decoding sampling layer physically prevents the model from drifting into invalid decision formats.

5. Context Saturation & Performance Degradation

The Problem

As prompt size approaches the maximum context limit (e.g., 32K–65K tokens), Key-Value (KV) cache memory consumption explodes. This causes severe Time-To-First-Token (TTFT) latency spikes and hardware swapping.

  • Example: Running simultaneous background summarization requests alongside user-facing generation causes local GPU/VRAM to thrash, dropping generation speed to under 1 token/sec.

Storyteller Implementation: ✅ Active Mitigation

Storyteller treats local compute constraints as a core design boundary:

  1. Strict Context Window Bounding: chat.maxRecentTurns (default 2) restricts raw narrative history to a short, high-speed window.
  2. Sequential Derived-Memory Queue (DerivedMemoryTaskQueue): When a story turn completes, three background memory updates are triggered (summary, recent-summary, canonical-state). To prevent three concurrent LLM inference calls from overwhelming local VRAM/CPU resources, Storyteller submits these jobs to a single-threaded daemon queue.
  3. Non-Blocking Architecture: Background memory updates run asynchronously and sequentially without blocking the user interface or triggering parallel backend contention.

Clone this wiki locally