-
Notifications
You must be signed in to change notification settings - Fork 0
Local LLM Limitations: Entity Contagion & Drift
While open-source local LLMs (even in the mid-weight 26B–35B parameter range) have improved drastically, they still hit hard architectural and capacity ceilings when tracking detailed, multi-entity states over long contexts.
Entity Contagion & Latent Feature BleedingIn mid-weight local LLMs (26B–35B), entity tracking relies on dynamic attention routing across context windows. When narrative tokens intervene between an entity and its assigned attributes, attention weights decay. When the model encounters overlapping semantic concepts (e.g., music, relationships, male leads), a failure mode called Entity Contagion (or Latent Feature Bleeding) occurs.
Rather than maintaining a isolated database of characters, the model calculates the next token based on statistical proximity. The associative bridge between Band -> Guitarist -> Male Character collapses into the most active entity in the working memory.Illustrative Example:
Chris: [Role: Guitarist, Band Member]
Mike: [Role: Childhood Friend, Non-Musician]
Valerie: [Role: Lead/Protagonist]Valerie stepped off the stage, her heart still pounding from the set. The guitars had sounded incredible tonight. She grabbed a water bottle and saw Mike standing near the exit..."
graph TD
Sub1[Chris = Guitarist] -->|Decay over Distance| Trait[Floating Trait: Guitarist]
Context[Valerie thinks of Mike...] --> Subject[Active Subject: Mike]
Trait -->|Attention Bleeding| Bridge[High Token Proximity]
Subject -->|Direct Context Proximity| Bridge
Bridge -->|Faulty Binding| Result[Failure: Mike becomes Guitarist]
Mechanism: As the model decodes the scene involving Mike, the high-activation latent features for [Band] and [Guitarist] are floating unanchored in short-term context memory. Because Mike is the active subject, the attention mechanism erroneously binds these floating traits to him.
Resulting Output:
"Mike adjusted the strap of his guitar and smiled at Valerie as she approached." (Failure: Chris's role permanently bled into Mike's entity state).
Key Technical Takeaways
-
Positional Distance vs. Semantic Association: The model prioritizes immediate proximity of entities (
Mike) over distant structured constraints (Chris = Guitarist). -
Soft Bounding Failures: Negative rules (e.g., "Do not make Mike the guitarist") fail because transformer attention attends heavily to the words Mike and Guitarist, inadvertently reinforcing the incorrect connection.
-
Limited Parameter Capacity for Entity Mapping: A 26B/35B model has a restricted number of attention heads dedicated to high-dimensional state tracking. When storing complex YAML-like relational trees, the model frequently suffers from "entity bleeding," where attributes (e.g., "guitarist") disconnect from their primary entity ("Chris") and attach to the most active subject in the immediate generation window ("Mike").
-
Superficial Context Retrieval (Lost in the Middle): Local models often process system prompts or long YAML blocks as background tokens rather than strict database constraints. Information tucked in the middle of long prompts gets diluted by the attention mechanism, causing the model to prioritize conversational flow and local token probability over hard structured data.
-
Instruction Drift & Rule Overriding: System prompts and explicit guardrails are soft constraints in transformers, not hard programming rules. Under creative text generation, auto-regressive decoding favors narrative momentum and high-probability language patterns over strict negative constraints, rendering system rules ineffective over extended outputs.
-
Supervised Fine-Tuning (SFT) Bias: Most local models are heavily fine-tuned on general storytelling or chat datasets where common tropes (e.g., "the male lead is in the band") are mathematically overrepresented. When a model experiences even a slight drop in attention accuracy, it defaults to these training tropes.
-
Lack of Active State-Verification: Unlike multi-agent enterprise setups, local models generate output in a single forward pass. They lack built-in secondary checking passes to validate generated attributes against the source schema before outputting text.
Flatten the State Schema: Instead of nested YAML structures, use explicit, flat sentences in the context (e.g., "Chris is the ONLY guitarist in the band. Mike is NOT a guitarist.").
Pre-fill / Post-Processing Logic: Run a lightweight secondary local pass (e.g., a fast 8B model or regex validator) specifically tuned to flag character trait inconsistencies before displaying the output.
Beyond logic and state tracking errors, local models—especially quantized variants (e.g., 6-bit or 4-bit)—frequently suffer from token corruption and word-level hallucinations (e.g., generating "she knitted her highbrows" or "she furrowed her eyebrows wish" instead of "she furrowed her eyebrows").
-
Loss of Precision in Embeddings (Quantization Impact): Compressing a model to 6-bit rounds off fine-grained probability weights. The subtle mathematical distance between related subword tokens shrinks, causing the model to accidentally swap a correct token for a visually or semantically neighboring one.
-
Subword Tokenization Glitches: LLMs do not output full words; they predict subword fragments (tokens). If the activation weight for the correct fragment (e.g.,
-brows) dips even slightly, the decoding algorithm picks an adjacent high-probability fragment (e.g.,-wishor-high), breaking the word or phrase mid-generation. -
*Sampling/Temperature Noise(: Even low sampling temperatures (
$T = 0.7$ ) combined with Top-P/Top-K can accidentally push a corrupted token to the top of the probability queue if the model's confidence in the correct token drops slightly or decays.
- Upgrade Quantization Precision: Upgrade from lower-precision formats to Q8_0 or unquantized FP16 if your VRAM allows it. Higher precision restores the fine mathematical distances between subword token embeddings.
-
Tweak Decoding Parameters: Lower the Temperature (e.g.,
0.2–0.5) and tighten Top-P / Min-P settings. This restricts the candidate token pool and prevents low-probability, corrupted subwords from creeping into the output stream. -
Repetition & Bad-Words Penalty: Increase the
repetition_penaltyslightly or apply a frequency/presence penalty to stop the model from repeating a glitched token sequence once it begins. - System Prompt Constraints: Add explicit style rules in the system prompt emphasizing formal grammar and vocabulary (e.g., "Maintain strict adherence to proper grammar, real dictionary words, and precise character anatomy descriptions").
- Automated Post-Processing Pass: Implement a fast post-processing regex or dictionary filter (like Hunspell or LanguageTool), or run a lightweight secondary LLM pass (e.g., a fast 8B model) dedicated exclusively to proofreading and fixing corrupted tokens before rendering the final text.
Even when a local model supports large context windows (e.g., 32k to 128k tokens), its ability to recall specific facts hidden deep within that context degrades significantly. This is commonly known as the Needle in a Haystack (NIAH) retrieval decay.
- U-Shaped Attention Curve: Transformer attention mechanisms naturally pay higher attention to tokens at the very beginning (system prompt) and the very end (latest user turn) of the prompt window. Information placed in the middle receives exponentially lower attention weight.
- Context Dilution: As the conversation grows, noisy or narrative tokens overwhelm the model’s dynamic memory, diluting key entity schemas and rules.
- Dynamic Context Pruning: Strip out old, non-essential narrative turns and keep only the latest working state and active character schemas.
- Anchor Key Rules at Both Ends: Repeat critical rules or entity constraints at the very bottom of the prompt (just before the generation trigger) rather than relying solely on the system prompt at the top.
In extended sessions or multi-turn storytelling, local models exhibit Instruction Drift. System prompts, negative rules (e.g., "Never write from X perspective"), and structural output constraints gradually collapse as the conversation lengthens.
- Autoregressive Bias: The model predicts text based on the entire preceding context. As the context fills with narrative prose, the statistical momentum of the story text overwhelms the static instructions in the system prompt.
- Lack of Hard Constraints: System prompts are soft probabilistic weights, not compiled code. Under creative generation, the model prioritizes narrative continuity over rule compliance.
- System Prompt Injection per Turn: Re-inject system instructions invisibly at every turn instead of relying on a single initial system message.
-
Structured Output Enforcers: Use constrained sampling libraries (such as
GBNF grammarsorOutlines) to physically block forbidden tokens or force specific output structures at the decoding layer.
As the context length expands toward its maximum limit, local hardware faces drastic performance slowdowns and potential output degradation.
- *KV-Cache Memory Pressure: Maintaining the Key-Value (KV) cache for large contexts consumes immense VRAM. When VRAM fills up, the system offloads computations to system RAM, causing severe latency spikes (Time-To-First-Token).
- *Attention Fragmentation: Beyond a certain context length, attention distributions become overly diffuse, leading to repetitive loops, rambling outputs, or abrupt mid-sentence cutoffs.
- Sliding Window Context: Restrict the active context window to a manageable length (e.g., 8k–16k tokens) rather than filling the maximum theoretical limit.
-
KV-Cache Quantization: Quantize the KV-cache (e.g.,
FP8orQ4_0cache) to reduce memory overhead and maintain high token processing speeds.