Long-horizon context: can history retrieval make history recursively self-referential?

[deephbz]

I've been looking at the new token-budget / history / notes / new_context approach and wanted to make one possible long-horizon failure mode explicit for discussion.

Summary

My current mental model is:

• A thread can span multiple fresh context windows.
• notes preserves a small model-authored working checkpoint.
• history preserves old window items and lets later windows retrieve exact items.
• Stable window/item IDs let notes point back to canonical historical evidence.
• This avoids recursively summarizing the whole conversation during rollover.
• However, retrieving history appears to itself create a new tool call/result in the current window.
• Consider an original 50K-token tool result in window W1.
• W2 retrieves that result because it is still needed.
• W3 later retrieves the same original result again.
• Logically there is one piece of evidence, but the event history may now contain several materialized copies.
• If later history.search_contents indexes those retrieval results as ordinary history, search can potentially find the original plus its echoes.
• This seems different from active-context growth: fresh windows solve that part well.
• The concern is instead durable-history cleanliness, canonical identity, provenance, and retrieval quality over very long threads.
• Stable IDs may already mitigate much of this if models consistently retain and dereference the original item ID.
• But large tool outputs also look somewhat different from conversational events: they are durable artifacts with potentially many references.
• A filesystem analogy would separate transcript.jsonl (events), notes.md (checkpoint), and immutable files/artifacts (bulk content).
• History entries could then refer to an artifact rather than making each retrieval conceptually become another copy of the artifact.
• I am not proposing that specific implementation; I'm mainly trying to make the distinction explicit.
• I'm curious whether the existing system already canonicalizes/deduplicates retrieved items, or deliberately treats this as harmless.
• If not, this may be worth keeping in mind as context-management sessions become much longer.

Concrete example: retrieval echo across windows

Suppose W1 contains a valuable, very large tool output:

W1
  tool_call analyze_repo()
  tool_result R              # ~50K tokens

Before rollover, notes preserve:

Important repository analysis:
  source = W1:item-48

W2 starts with a fresh active context and later needs the exact analysis:

W2
  history.read_item(W1:item-48)
  -> R

This is desirable: W2 pays the 50K-token cost only when the information is actually needed.

Now W3 later needs it again:

W3
  history.read_item(W1:item-48)
  -> R

Active-context behavior still looks excellent: W3 did not automatically inherit either earlier copy.

The question is what durable history now looks like conceptually:

W1:item-48     original R
W2:item-17     history.read_item(...) -> R
W3:item-09     history.read_item(...) -> R

If all three are independently searchable history items, a later query can potentially discover several manifestations of the same underlying evidence.

At sufficiently long horizons this can become a kind of "memory echo":

event
  -> retrieved representation of event
      -> retrieved representation of retrieved representation

Even if byte storage is deduplicated internally, the logical/indexing identity still matters.

Why stable window/item IDs seem important

The existing ID model may already provide an elegant partial answer.

If W1:item-48 is the canonical source, notes can preserve that exact reference across every rollover:

repo_analysis = W1:item-48

Then W2, W3, W8, etc. always dereference the original:

read_item(W1:item-48)

rather than notes gradually changing to:

W1:item-48
→ W2:item-17
→ W3:item-09
→ ...

That prevents a provenance chain from turning into a copy-of-a-copy chain at the model level.

This makes me wonder whether history search/read internally preserves an equivalent canonical-source relationship as well.

Event history vs artifact identity

A filesystem implementation of the same general context-management idea makes one distinction especially visible:

session/
  transcript.jsonl
  notes.md
  artifacts/
    repo-analysis.txt
    test-output.txt
    browser-dump.html

These have different semantics:

Event history

what happened, in what order

Notes/checkpoint

what the current agent believes matters

Artifacts

large immutable pieces of evidence produced or consumed during the work

A 50K-token compiler trace, repository survey, browser dump, subagent report, or generated analysis arguably behaves more like an artifact than like conversational prose.

One possible conceptual representation would therefore be:

W1:item-48
  tool_result:
    artifact = A

W2
  retrieve W1:item-48
    -> reference/read A

W3
  retrieve W1:item-48
    -> reference/read A

There is one artifact identity and many events referring to it.

This could be content-addressed, file-backed, database-backed, or implemented some completely different way. I'm not advocating a particular storage design here—the useful distinction to me is simply event identity vs content/artifact identity.

Questions I'm mainly curious about

1. Are outputs from history.read_item / history.search_contents themselves subsequently indexed as ordinary searchable history?

2. If the same original item is retrieved in multiple later windows, does the backend preserve a canonical source identity or treat each returned payload as an independent history item?

3. Does search deduplicate or down-rank such retrieval echoes?

4. Does a retrieved item retain provenance like:

canonical_source = W1:item-48

rather than only appearing as a new W2 tool result?

5. Is history conceptually intended to serve both as the event log and as storage for arbitrarily large tool-result artifacts, or is there another artifact/content layer involved internally?

6. If none of this is currently special-cased, is repeated retrieval expected to remain uncommon enough that the simpler model wins?

I'm mostly interested in clarifying the conceptual model rather than asking for a particular feature.

Related work / why I think this is slightly different

Some nearby Codex discussions/issues already explore pieces of this space:

• Thread-Scoped Session Memory Layer Over Compaction #15432 — thread-scoped session memory/retrieval across compaction.
• Feature proposal: cache-aware generational compaction with a dedicated low-cost summarizer agent #37448 — keeping archived context outside the active prompt and retrieving it when needed.
• RFC: Deterministic Session Checkpoint v1 (DSC) — Compaction Without Summarization #8573 — deterministic checkpoints, evidence references, and artifact records.
• core: reset context for token budget compaction #29743 — token-budget rollover using a fresh context rather than summarization.
• Add history and notes tools for token-budget sessions #39827 — history and notes tools for token-budget sessions.
• Include context window IDs in response metadata #39847 — context-window IDs in response metadata.

The distinction I'm trying to surface is specifically:

Once historical retrieval is first-class, retrieval itself becomes an event. What keeps repeated observations of one old piece of evidence from gradually becoming many logically independent pieces of history?

That feels like a secondary-order property of the new architecture rather than a request for another memory mechanism.

[jimy-r]

The echo risk exists only if history stores content. If the history layer stores a pointer to the tool result (the tool call id, the file path and range, a hash of the output) and re-materialises the content on demand, then retrieving the same result across three context windows produces three references to one object, not three copies. A pointer cannot duplicate itself into an echo. A copy always can.

The same rule that keeps this clean also keeps it honest. Memory and notes point at the source of truth instead of mirroring it, so when the underlying file changes, the pointer resolves to the new state instead of quietly serving a stale copy. Every write to the notes layer is one of four operations (add, update, delete, no-op), never a blind append, which is the other place echo entries come from.

Where a checkpoint genuinely needs content, because the source is ephemeral, store it once with a stable id and have later windows reference the id. A periodic consolidation pass then merges duplicates and prunes.

I maintain a reference architecture that documents this exact pattern: https://github.com/jimy-r/agent-workspace-architecture/blob/main/PATTERNS.md#5-memory-points-it-doesnt-mirror

[deephbz]

The echo risk exists only if history stores content. If the history layer stores a pointer to the tool result (the tool call id, the file path and range, a hash of the output) and re-materialises the content on demand, then retrieving the same result across three context windows produces three references to one object, not three copies. A pointer cannot duplicate itself into an echo. A copy always can.

The same rule that keeps this clean also keeps it honest. Memory and notes point at the source of truth instead of mirroring it, so when the underlying file changes, the pointer resolves to the new state instead of quietly serving a stale copy. Every write to the notes layer is one of four operations (add, update, delete, no-op), never a blind append, which is the other place echo entries come from.

Where a checkpoint genuinely needs content, because the source is ephemeral, store it once with a stable id and have later windows reference the id. A periodic consolidation pass then merges duplicates and prunes.

I maintain a reference architecture that documents this exact pattern: https://github.com/jimy-r/agent-workspace-architecture/blob/main/PATTERNS.md#5-memory-points-it-doesnt-mirror

Thanks for your inputs! I agree with the pattern "memory points but not mirrors". It should be able to solve duplication across context (rollover) windows. However, if agent session log requires a storage of what's sent to LLM then three retrievals of the same resourceID across three context windows will record the on-demand-materialized content three times. Is this solvable?