Skip to content

perf: distillation loses specific identifiers at 400K+ token sessions #417

@BYK

Description

@BYK

Problem

At 400K tokens, Lore scores 3.5 vs compaction's 4.2 on CM-1. The gap is specifically on medium-difficulty questions where distillation dropped specific details (exact error messages, alternative approaches considered, config file paths).

Root Cause

Lore's two-stage compression pipeline loses more information than compaction's single-pass summary:

  1. Gen-0 compression: Each 16K segment gets 8*sqrt(16384) = 1024 tokens. This is 16:1 compression — specific identifiers (error messages, file paths, version numbers, rejected alternatives) are dropped.
  2. Meta-distillation: 20 gen-0 segments (~20K tokens) compressed to ~5-6K tokens. Another 3-4:1 compression compounding the loss.
  3. Total: 320K → 20K → 5-6K = ~53-64:1 compression vs compaction's 320K → 4K single-pass (80:1 but higher quality due to seeing raw content).

Evidence

CM-1 inflated to 400K tokens:

Baseline Easy Medium Hard Overall
Lore 5.0 2.3 3.3 3.5
Tail-window 4.7 1.3 1.4 2.5
Compaction 4.7 3.9 4.1 4.2

Failed medium questions: model says 'no alternative approach was explicitly discussed' (it was), 'I don't have information about an integration test failure' (there was one).

Possible Fixes

  1. Increase gen-0 budget multiplier from 8 to 10-12 (more tokens per segment)
  2. Raise the CAP from 4096 to 6144 or 8192 (allows larger segments more budget)
  3. Skip meta-distillation for recent segments (keep gen-0 detail for the most recent N segments)
  4. Improve observer prompt to prioritize specific identifiers over chronological event logging
  5. Increase tool output truncation limit from 2K chars (distillation input) — specific error messages in tool results get truncated before the LLM sees them

Context

Discovered during #410 investigation. The eval fix (#414) correctly represents Lore's behavior — this gap is a real Lore limitation, not an eval bug.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions