Discussion: reducing tokens across memory generation and context delivery

[hidb4ai]

This discussion proposes reducing total tokens across memory generation and later use as a product direction for PowerContext, while preserving durable facts and task quality. The design is tracked in Issue #1989 and RFC PR #1990. Benefits remain to be established.

PowerContext owns two relevant boundaries: input, after Source is saved and before Atomic/Topic extraction, and output, inside existing prepare_context after Memory is recalled. Smaller prepare bodies alone do not reduce extraction costs already incurred.

Consider three messages: “Read-only idempotent queries may retry three times,” “Financial operations must never retry automatically,” and “Starting next week, reduce the limit to two; other restrictions remain unchanged.” A host saves snapshots [m1], [m1,m2], and [m1,m2,m3].

At input, remove duplicate body copies within one request while retaining metadata and field roles. Share repeated message bodies within the same authorized window, retaining every complete snapshot, original Source identity, time, order, and occurrence count. Sending only m3 would lose context; the proposed view retains all three messages. The server checks complete reconstruction, and the model reads the shared view directly. Shared views never overwrite original Sources or official Artifact content.

At output, group identical values within an Experience, avoid duplicate summary/snippet text within a Topic, or replace repeated table keys with one header. Retain roles, conditions, citations, types, values, and duplicate rows. Unknown or unvalidated formats, failed checks, and representations that are not smaller fall back to legacy.

Each side independently defaults to off and supports off / shadow / active. Shadow follows legacy without another extraction call. The proposal adds no tables, columns, migrations, endpoints, MCP operations, or public fields. Agents need no history adaptation or new state.

A separate admission strategy would skip extraction for inputs judged to need no memory. It remains shadow-only and never advances cursors based on that judgment. Activation requires durable-fact and future-question quality checks plus production recovery: disabling input cannot repair omissions from already consumed Sources.

Evaluation uses two generated Memory snapshots, input off and active, each reused with output off and active: four cells, without re-extraction between output pairs. Use actual capture→processing→durable readback. Existing LongMemEval insertion bypasses extraction; static Readers cannot establish input quality. Measure all actual provider usage, stages, retries, and later queries; missing usage is unknown. Critical missing counts make full-cohort cost indeterminate. Check severe conditional/temporal errors, source support, restart readback, and held-out future questions.

Community review:

1. Which Source formats show duplicate bodies or growing snapshots in real deployments?
2. Which directly readable snapshot representation should we qualify first, especially for repeated messages and distinct times?
3. What durable-fact, future-question, latency, and recovery thresholds should be required before activation?