Skip to content

Local LLM Limitation mitigations

Jump to bottom
Justus Brugman edited this page Aug 22, 2026 · 6 revisions

When running narrative generation on local, mid-sized LLMs (such as Gemma-4-26B-A4B or Qwen 30B-class models), application code can compensate for several model, context-management, and local-inference limitations.

This document maps the Big 5 Local LLM Limitations directly to the design patterns implemented inside Storyteller.

Overview Matrix

Limitation Class Severity in Storytelling Storyteller Status Primary Mitigation Mechanism
1. Entity Contagion & Drift High ⚠️ Partially Mitigated Canonical context in story generation plus synchronous validation of explicit rules and fixed-protagonist constraints (ALLOW / REPLACE).
2. Lexical Corruption & Quantization Noise Medium ❌ Not Mitigated (Backend Dependent) No lexical correction; output quality depends on the selected model, quantization method, inference backend, and decoding configuration.
3. Context Degradation ("Lost in the Middle") High ✅ Actively Mitigated Multi-layer memory shaping (summary.md, recent-summary.md) & single-system-prompt stacking.
4. Instruction Drift & Guardrail Degradation Critical ⚠️ Best-Effort Mitigation Transient cache-buster reset requests, user-triggered reset/undo actions, and JSON Schema-constrained validation when supported.
5. Context Saturation & Overhead Critical ✅ Actively Mitigated Fixed turn-window limits (chat.maxRecentTurns) & the sequential DerivedMemoryTaskQueue.

1. Entity Contagion & Feature Bleeding

The Problem

In mid-weight local models, attributes associated with one character decay across narrative distance and accidentally bind to another active character in the scene ("entity bleeding").

  • Example: The prompt defines Chris as the guitarist and Mike as a childhood friend who does not play music. After a scene set at a music venue, the LLM outputs:

    "Mike unstrapped his guitar and leaned it against the amplifier."

Storyteller Implementation: ⚠️ Partial Mitigation

Storyteller limits entity bleeding through state boundary enforcement, but cannot alter the underlying transformer forward-pass logic:

  1. Explicit Canon Context: PromptAssemblyService injects fixed_protagonists.yml and canonical-state.yaml directly into the system prompt to anchor character identities.
  2. Validator Guardrail: When enabled (validation.enabled=true), ResponseGuard routes the draft through a synchronous validator pass. The validator checks the supplied rules and fixed-protagonist constraints and can issue a REPLACE decision before the response is displayed. It does not currently receive or validate canonical-state.yaml.
  • Remaining Risk: Highly dense multi-character scenes in a single generation pass can still trigger momentary attribute bleeding if the validator misses subtle context cues.

2. Lexical Corruption & Quantization Noise

The Problem

Aggressive weight quantization reduces model precision and can increase generation errors. Depending on the model, quantization method, inference backend, and decoding configuration, this may appear as malformed wording, inappropriate token choices, repetition, or less coherent prose.

  • Example: Instead of generating “She furrowed her eyebrows in frustration,” a heavily quantized model might generate:

    “She furrowed her highbrows in frustration” or “She furrowed her eyebrows wish.”

Storyteller Implementation: ❌ Not Mitigated

Storyteller does not attempt to detect or correct lexical errors in generated story text.

ResponseSanitizer trims responses and decodes visible JSON-style escape sequences, but does not perform spelling, grammar, dictionary, or semantic correction. The validator checks explicit story rules and fixed-protagonist constraints; it is not designed to detect lexical corruption.

Any lexical degradation introduced by the selected model, quantization method, inference backend, or decoding configuration may therefore remain present in the final response.

3. Context Degradation ("Lost in the Middle")

The Problem

Models can retrieve information less reliably when it appears deep inside a long context, particularly when relevant information is surrounded by large amounts of unrelated text. This position-dependent degradation is commonly described as the “Lost in the Middle” problem.

Needle in a Haystack evaluations expose this limitation by placing a specific fact—the “needle”—at different positions inside a much larger context and testing whether the model can retrieve it. Performance varies by model, prompt structure, information position, and context length; a model supporting a large context window does not necessarily use every part of that window equally reliably.

  • Example: Important rules, character details, or plot events established many turns earlier may be ignored or recalled incorrectly when they appear deep inside the assembled context.

Storyteller Implementation: ✅ Active Mitigation

Storyteller avoids sending the complete raw conversation history to the model. Instead, it assembles each story request from a bounded window of recent raw turns, periodically generated memory summaries, canonical state, fixed-protagonist definitions, and the active system instructions.

  • Memory Hierarchy: Older events are periodically condensed into recent-summary.md, summary.md, and canonical-state.yaml after their configured batch thresholds are reached. These asynchronous derived-memory updates may temporarily lag behind the latest persisted turn.
  • Single System Message Stacking: Active instructions, fixed protagonists, canonical state, and summaries are assembled into one first system message. This gives the backend a stable prompt structure, although it does not guarantee that every included fact will be retrieved or followed.
  • Recent Raw-Turn Window: Only the most recent chat.maxRecentTurns story turns are included as raw conversation messages. Older turns remain persisted in history.json but are represented to the model through the derived-memory layers.

4. Instruction Drift & Guardrail Degradation

The Problem

During extended roleplay or storytelling sessions, accumulated narrative context can reduce the model’s attention to earlier system instructions. As the conversation grows, constraints such as “Never speak as the user” may be followed less consistently.

  • Example: The model may begin writing dialogue for the user’s protagonist or stop following the required character formatting.

Storyteller Implementation: ⚠️ Best-Effort Mitigation

Because model output is probabilistic, instruction drift cannot be eliminated entirely at the application layer. Storyteller uses several best-effort safeguards to detect, limit, and recover from drift:

  1. Periodic Cache-Buster Requests (cacheBuster.interval): After every N persisted story turns (default: 5), Storyteller sends a transient reset request with a unique token prepended to the system prompt. The response is discarded, and the request is not added to history.json.

    The changed prefix prevents exact-prefix cache reuse for that individual request on cache-sensitive OpenAI-compatible backends. It does not flush the backend’s cache, modify persistent model state, or guarantee that later requests will follow the rules more closely.

  2. User-Triggered Resets: Ctrl-W sends the same transient reset request without modifying history.json. When a persisted turn is available, Ctrl-U removes the most recent turn, sends the reset request, and restores the removed user prompt to the input buffer for editing and retrying.

  3. Structured Validator Output: When validation is enabled, ValidationClient requests a constrained JSON Schema response containing an ALLOW or REPLACE decision.

    With validation.outputMode=auto, schema-constrained output is used when the backend supports it. If the backend rejects structured output, Storyteller falls back to tolerant text parsing for the remainder of the active session. Invalid, empty, or unparseable validator responses are handled using the configured fail-closed response.

  • Remaining Risk: These safeguards reduce the impact of instruction drift but cannot guarantee that it will not occur. Cache-buster requests do not constitute a documented KV-cache flush, and schema enforcement constrains only the validator’s decision format—not the original story generation. In long sessions, users may still need to use Ctrl-W or Ctrl-U when drift becomes visible.

5. Context Saturation & Performance Degradation

The Problem

As prompt size approaches the configured context limit, KV-cache memory consumption grows with sequence length and prompt prefill becomes more expensive. On memory-constrained systems, this can cause latency spikes, reduced generation speed, or swapping.

  • Example: A background memory update that overlaps with user-facing generation may cause both requests to compete for local GPU, VRAM, unified memory, or CPU resources.

Storyteller Implementation: ✅ Active Mitigation

Storyteller reduces prompt growth and serializes derived-memory work, but does not impose a hard token limit on the complete assembled prompt:

  1. Raw-History Bounding: chat.maxRecentTurns (default: 2) restricts the number of raw narrative turns included in foreground story requests. This reduces prompt growth but does not limit the combined size of system instructions, fixed-protagonist definitions, canonical state, and summaries.

  2. Sequential Derived-Memory Queue (DerivedMemoryTaskQueue): After a story turn completes, Storyteller checks whether the long-term summary, recent summary, or canonical state requires an update. Eligible jobs are submitted to a single-threaded daemon queue, preventing derived-memory inference calls from running concurrently with one another.

  3. Asynchronous Background Updates: Eligible memory updates run asynchronously and sequentially, so they do not block completion of the story turn that scheduled them. A background update may still overlap with a later foreground story request when both use the same inference backend.

Clone this wiki locally