Skip to content

Local LLM Limitation mitigations

Jump to bottom
Justus Brugman edited this page Aug 25, 2026 · 6 revisions

When running narrative generation on local, mid-sized LLMs (such as Gemma-4-26B-A4B or Qwen 30B-class models), application code can compensate for several model, context-management, and local-inference limitations.

This document maps the Big 5 Local LLM Limitations directly to the design patterns implemented inside Storyteller.

Overview Matrix

Limitation Class Severity in Storytelling Storyteller Status Primary Mitigation Mechanism
1. Entity Contagion & Drift High ⚠️ Actively, but Partially Mitigated Canonical context, selectively injected authoritative knowledge-graph facts, and synchronous validation (ALLOW / REPLACE).
2. Lexical Corruption & Quantization Noise Medium ❌ Not Mitigated (Backend Dependent) No lexical correction; output quality depends on the selected model, quantization method, inference backend, and decoding configuration.
3. Context Degradation ("Lost in the Middle") High ✅ Actively Mitigated Multi-layer memory shaping, selective graph grounding, and single-system-prompt stacking.
4. Instruction Drift & Guardrail Degradation Critical ⚠️ Best-Effort Mitigation Transient cache-buster reset requests, user-triggered reset/undo actions, and JSON Schema-constrained validation when supported.
5. Context Saturation & Overhead Critical ✅ Actively Mitigated Fixed turn-window limits, selective graph retrieval, and the sequential DerivedMemoryTaskQueue.

1. Entity Contagion & Feature Bleeding

The Problem

In mid-weight local models, attributes associated with one character decay across narrative distance and accidentally bind to another active character in the scene ("entity bleeding").

  • Example: The prompt defines Chris as the guitarist and Mike as a childhood friend who does not play music. After a scene set at a music venue, the LLM outputs:

    "Mike unstrapped his guitar and leaned it against the amplifier."

Relationships, capabilities, possessions, and locations can bleed in the same way. A model may make Valerie fall in love with Chris even though the established relationship is with Mike, assign one character another character's skill, or move a character into the wrong home.

Storyteller Implementation: ⚠️ Active, but Partial Mitigation

Storyteller limits entity bleeding through explicit context, deterministic fact selection, and synchronous validation, but cannot alter the underlying transformer forward-pass logic:

  1. Explicit Canon Context: PromptAssemblyService injects fixed_protagonists.yml and canonical-state.yaml into the system prompt to anchor character identities and current narrative state.

  2. Validated Knowledge Graph: memory/knowledge-graph.json stores typed entities and explicit positive or negative facts. Facts can describe capabilities, possessions, relationships, cohabitation, and residence. Predicate definitions are configuration-driven through systemprompts/graph-predicates.json, including permitted subject and object entity types and their positive and negative prompt text.

  3. Selective Story Grounding: Before story generation, Storyteller resolves entity names and configured aliases in the current user input. It injects only ACTIVE, hard=true facts connected to those entities into the combined system message. These facts are marked as authoritative and explicitly instruct the model not to transfer traits between characters.

  4. Validator Guardrail: When enabled (validation.enabled=true), ResponseGuard routes the draft through a synchronous validator pass. Storyteller resolves entities across both the user input and draft response and adds the relevant authoritative graph facts to the validation request. The validator checks those facts alongside the supplied rules and fixed-protagonist constraints and can issue a REPLACE decision before the response is displayed. It does not currently receive the complete canonical-state.yaml.

  5. Safe Runtime Loading: The graph is loaded automatically. Valid changes to the configured graph file become visible without restarting Storyteller. If the file is temporarily incomplete or invalid during editing, the runtime continues serving the last valid snapshot.

  6. Explicit Graph Management: /graph inspects the current graph without contacting the model. /graph -generate creates and immediately loads a minimal empty graph without contacting the model. /graph -fill uses the model only to extract a graph from the complete configured fixed_protagonists.yml, after which Java normalizes, validates, atomically persists, and publishes the result.

Remaining Risks and Boundaries

  • Normal story turns are read-only with respect to the graph. Generated story events do not automatically create, change, or retire facts.
  • Retrieval requires an entity name or configured alias to appear in the current user input or generated draft. Storyteller does not currently perform semantic entity recognition.
  • The graph does not infer unstated relationships, transitive facts, or symmetric relationships. For example, LOVES(A, B) does not imply LOVES(B, A).
  • Facts that are not both ACTIVE and hard=true are not used for prompt grounding.
  • Dense multi-character scenes can still trigger attribute bleeding if the relevant entity is not resolved or the model or validator ignores a supplied fact.

2. Lexical Corruption & Quantization Noise

The Problem

Aggressive weight quantization reduces model precision and can increase generation errors. Depending on the model, quantization method, inference backend, and decoding configuration, this may appear as malformed wording, inappropriate token choices, repetition, or less coherent prose.

  • Example: Instead of generating “She furrowed her eyebrows in frustration,” a heavily quantized model might generate:

    “She furrowed her highbrows in frustration” or “She furrowed her eyebrows wish.”

Storyteller Implementation: ❌ Not Mitigated

Storyteller does not attempt to detect or correct lexical errors in generated story text.

ResponseSanitizer trims responses and decodes visible JSON-style escape sequences, but does not perform spelling, grammar, dictionary, or semantic correction. The validator checks explicit story rules, fixed-protagonist constraints, and relevant knowledge-graph facts; it is not designed to detect lexical corruption.

Any lexical degradation introduced by the selected model, quantization method, inference backend, or decoding configuration may therefore remain present in the final response.

3. Context Degradation ("Lost in the Middle")

The Problem

Models can retrieve information less reliably when it appears deep inside a long context, particularly when relevant information is surrounded by large amounts of unrelated text. This position-dependent degradation is commonly described as the “Lost in the Middle” problem.

Needle in a Haystack evaluations expose this limitation by placing a specific fact—the “needle”—at different positions inside a much larger context and testing whether the model can retrieve it. Performance varies by model, prompt structure, information position, and context length; a model supporting a large context window does not necessarily use every part of that window equally reliably.

  • Example: Important rules, character details, or plot events established many turns earlier may be ignored or recalled incorrectly when they appear deep inside the assembled context.

Storyteller Implementation: ✅ Active Mitigation

Storyteller avoids sending the complete raw conversation history to the model. Instead, it assembles each story request from a bounded window of recent raw turns, periodically generated memory summaries, canonical state, fixed-protagonist definitions, and the active system instructions.

  • Memory Hierarchy: Older events are periodically condensed into recent-summary.md, summary.md, and canonical-state.yaml after their configured batch thresholds are reached. These asynchronous derived-memory updates may temporarily lag behind the latest persisted turn.
  • Single System Message Stacking: Active instructions, fixed protagonists, canonical state, and summaries are assembled into one first system message. This gives the backend a stable prompt structure, although it does not guarantee that every included fact will be retrieved or followed.
  • Recent Raw-Turn Window: Only the most recent chat.maxRecentTurns story turns are included as raw conversation messages. Older turns remain persisted in history.json but are represented to the model through the derived-memory layers.
  • Selective Graph Grounding: Instead of injecting the complete knowledge graph, Storyteller retrieves active hard facts connected to entities mentioned in the latest user input. These facts are placed in the combined system message, reducing dependence on retrieving critical relationships, capabilities, or locations from older summaries.

4. Instruction Drift & Guardrail Degradation

The Problem

During extended roleplay or storytelling sessions, accumulated narrative context can reduce the model’s attention to earlier system instructions. As the conversation grows, constraints such as “Never speak as the user” may be followed less consistently.

  • Example: The model may begin writing dialogue for the user’s protagonist or stop following the required character formatting.

Storyteller Implementation: ⚠️ Best-Effort Mitigation

Because model output is probabilistic, instruction drift cannot be eliminated entirely at the application layer. Storyteller uses several best-effort safeguards to detect, limit, and recover from drift:

  1. Periodic Cache-Buster Requests (cacheBuster.interval): After every N persisted story turns (default: 5), Storyteller sends a transient reset request with a unique token prepended to the system prompt. The response is discarded, and the request is not added to history.json.

    The changed prefix prevents exact-prefix cache reuse for that individual request on cache-sensitive OpenAI-compatible backends. It does not flush the backend’s cache, modify persistent model state, or guarantee that later requests will follow the rules more closely.

  2. User-Triggered Resets: Ctrl-W sends the same transient reset request without modifying history.json. When a persisted turn is available, Ctrl-U removes the most recent turn, sends the reset request, and restores the removed user prompt to the input buffer for editing and retrying.

  3. Structured Validator Output: When validation is enabled, ValidationClient requests a constrained JSON Schema response containing an ALLOW or REPLACE decision.

    With validation.outputMode=auto, schema-constrained output is used when the backend supports it. If the backend rejects structured output, Storyteller falls back to tolerant text parsing for the remainder of the active session. Invalid, empty, or unparseable validator responses are handled using the configured fail-closed response.

  • Remaining Risk: These safeguards reduce the impact of instruction drift but cannot guarantee that it will not occur. Cache-buster requests do not constitute a documented KV-cache flush, and schema enforcement constrains only the validator’s decision format—not the original story generation. In long sessions, users may still need to use Ctrl-W or Ctrl-U when drift becomes visible.

5. Context Saturation & Performance Degradation

The Problem

As prompt size approaches the configured context limit, KV-cache memory consumption grows with sequence length and prompt prefill becomes more expensive. On memory-constrained systems, this can cause latency spikes, reduced generation speed, or swapping.

  • Example: A background memory update that overlaps with user-facing generation may cause both requests to compete for local GPU, VRAM, unified memory, or CPU resources.

Storyteller Implementation: ✅ Active Mitigation

Storyteller reduces prompt growth and serializes derived-memory work, but does not impose a hard token limit on the complete assembled prompt:

  1. Raw-History Bounding: chat.maxRecentTurns (default: 2) restricts the number of raw narrative turns included in foreground story requests. This reduces prompt growth but does not limit the combined size of system instructions, fixed-protagonist definitions, canonical state, and summaries.

  2. Sequential Derived-Memory Queue (DerivedMemoryTaskQueue): After a story turn completes, Storyteller checks whether the long-term summary, recent summary, or canonical state requires an update. Eligible jobs are submitted to a single-threaded daemon queue, preventing derived-memory inference calls from running concurrently with one another.

  3. Asynchronous Background Updates: Eligible memory updates run asynchronously and sequentially, so they do not block completion of the story turn that scheduled them. A background update may still overlap with a later foreground story request when both use the same inference backend.

  4. Selective Graph Retrieval: Storyteller does not add the complete knowledge graph to every prompt. It includes only active hard facts connected to entities resolved in the current input, or in the input and draft during validation. This limits normal graph overhead, although a highly connected entity can still add many facts because no separate fact-count or token cap is currently applied.

Clone this wiki locally