Skip to content

fix(eval): build realistic Lore context for QA questions#414

Merged
BYK merged 1 commit into
mainfrom
fix-eval-lore-context
May 20, 2026
Merged

fix(eval): build realistic Lore context for QA questions#414
BYK merged 1 commit into
mainfrom
fix-eval-lore-context

Conversation

@BYK
Copy link
Copy Markdown
Owner

@BYK BYK commented May 20, 2026

Summary

Fixes the CM-1 eval to test Lore the way it actually works in production — with distillation prefix + raw tail window + LTM + recall, using separate budgets that match the gradient context manager.

Problem

The eval's Lore baseline created a fresh gateway session per QA question. This new session had:

  • No distillation prefix — distillations are session-scoped (gradient.ts:691), and the QA session had a different ID
  • No context-bound LTM — gated by isFirstTurn (pipeline.ts:2900)
  • Only the recall tool — which returned compressed/truncated results

In real Lore usage, the gradient system allocates separate budgets: 25% of usable context for distillation prefix, 40% for raw tail window, with LTM injected separately. The eval was testing recall-in-isolation rather than the full Lore experience.

Fix

Adds buildLoreContext() to the eval harness that mimics the gradient context manager:

  1. Separate budgets — distillation and raw tail don't compete for the same pool (matching config.ts defaults)
  2. Usable context — subtracts output reserve (32K) and overhead (~8K) before computing budgets (matching gradient.ts logic)
  3. Non-archived distillations onlyarchived = 0 filter matches what the gradient system includes in-context
  4. No LTM duplication — LTM is left to the gateway's own system prompt injection
  5. Built once per baseline — not per question (performance)

The recall tool is still available via the gateway for additional detail.

Results

Base CM-1 (~16K tokens):

Baseline Before After
Lore 1.9 4.8
Tail-window 4.9 4.9
Compaction 4.9 4.9

Inflated CM-1 (400K tokens) — Lore beats tail-window by +50%:

Baseline Score
Lore 3.9
Tail-window 2.6
Compaction 4.1

Remaining gap vs compaction (3.9 vs 4.1) is due to distillation compression losses — tracked in #417.

Changes

packages/core/eval/harness.ts

  • Add buildLoreContext(turns, contextWindow) — builds distillation + raw tail context with separate budgets matching gradient defaults
  • Update askQuestionViaGateway() to accept and inject Lore context into the QA request
  • Hoist context building outside the question loop (built once per baseline)
  • Remove unused sessionHeaders parameter

Related to #410

The CM-1 eval was comparing Lore unfairly: tail-window got the full
conversation in context, while Lore got a brand new empty session with
no context — only the recall tool.

In real usage, Lore's gradient system allocates separate budgets:
- 25% of usable context for distillation prefix
- 40% for raw tail window (independent of distillation)
- LTM injected separately by the gateway

Fix: add buildLoreContext() that mimics the gradient context manager
with separate budgets computed from usable context (after subtracting
output reserve and overhead). Only non-archived distillations are
included (matching gradient.ts). LTM is left to the gateway's own
injection to avoid double-counting. Context is built once per baseline,
not per question.

Results (base CM-1 ~16K): Lore 4.8, tail-window 4.9, compaction 4.9
Results (400K inflated): Lore 3.9 > tail-window 2.6 (+50%)
Compaction leads slightly at 4.1 — tracked in #417.

Related to #410
@BYK BYK force-pushed the fix-eval-lore-context branch from 0a3c7e0 to d69634c Compare May 20, 2026 09:52
@BYK BYK enabled auto-merge (squash) May 20, 2026 09:53
@BYK BYK merged commit 8257bd3 into main May 20, 2026
10 checks passed
@BYK BYK deleted the fix-eval-lore-context branch May 20, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant