session_input_callback repeating a history item re-persists it as new session input

[hsusul]

Please read this first

• Have you read the docs? Yes — Sessions, Running agents, and .agents/references/session-persistence.md.
• Have you searched for related issues? Yes. Searched open/closed issues and PRs for session_input_callback duplicate, session_input_callback history re-appended, session history duplicated callback, prepare_input_with_session, session callback persists history, session grows duplicated history, and history appended twice session. Nearest prior work is Local Session loses the user's input turn when a model retry rewinds it (never re-persisted) #3852 (session loses a user turn when a retry rewinds it) and fix: load full history when compacting a limited session #3827 (compaction with a limited session); neither concerns callback-emitted duplicates.

Affected component

Client-managed session persistence — src/agents/run_internal/session_persistence.py::prepare_input_with_session, used by RunConfig.session_input_callback.

Describe the bug

If a session_input_callback emits an existing history item more than once, every extra occurrence is classified as new-turn input and written back into the session store. The stored conversation is corrupted, and because the re-persisted copy becomes part of the next turn's history, the corruption compounds on every subsequent turn.

prepare_input_with_session() classifies each item returned by the callback by first consuming an object-identity reference from history_refs/new_refs, then falling back to content-frequency maps. _consume_reference() pops the matched object out of the reference list, so the same history object appearing a second time no longer matches by identity; the identity match has also already decremented the content-frequency budget for that key, so the second occurrence falls through to appended.append(item) and is persisted as if it were fresh user input.

Debug information

• Agents SDK version: main @ c3f1781d56e8f1249a01674f18ba1f3e44a16dce (latest release tag v0.19.1)
• Python version: 3.13.5
• OS: macOS 15.7.3 (Darwin 24.6.0)
• No OpenAI API key, no network access, and no paid model call are needed — the repro uses tests/fake_model.py::FakeModel and a temporary SQLiteSession.

Repro steps

Save as tests/test_repro_session_callback.py and run uv run pytest tests/test_repro_session_callback.py -q:

from __future__ import annotations

import tempfile
from pathlib import Path

import pytest

from agents import Agent, RunConfig, Runner, SQLiteSession, TResponseInputItem

from .fake_model import FakeModel
from .test_responses import get_text_message


def repeat_first(
    history: list[TResponseInputItem], new: list[TResponseInputItem]
) -> list[TResponseInputItem]:
    """Re-emphasize the original request at the end of the model context."""
    if not history:
        return new
    return history + [history[0]] + new


@pytest.mark.asyncio
async def test_repeated_history_item_is_not_re_persisted() -> None:
    with tempfile.TemporaryDirectory() as tmp:
        session = SQLiteSession("s", Path(tmp) / "s.db")
        model = FakeModel()
        agent = Agent(name="A", model=model)
        config = RunConfig(session_input_callback=repeat_first)

        for turn in range(4):
            model.set_next_output([get_text_message(f"a{turn}")])
            await Runner.run(agent, input=f"u{turn}", session=session, run_config=config)

        stored = await session.get_items()
        user_inputs = [item.get("content") for item in stored if item.get("role") == "user"]
        # Each turn contributes exactly one user message; "u0" must not be stored again.
        assert user_inputs == ["u0", "u1", "u2", "u3"], user_inputs
        assert len(stored) == 8, len(stored)

Actual behavior

Fails deterministically (reproduced 3/3 runs):

FAILED tests/test_repro_session_callback.py::test_repeated_history_item_is_not_re_persisted
E       AssertionError: ['u0', 'u0', 'u1', 'u0', 'u2', 'u0', ...]
E       assert ['u0', 'u0', ...2', 'u0', ...] == ['u0', 'u1', 'u2', 'u3']

After 4 turns the session holds 11 items instead of 8:

['u:u0', 'a:a0', 'u:u0', 'u:u1', 'a:a1', 'u:u0', 'u:u2', 'a:a2', 'u:u0', 'u:u3', 'a:a3']

The u0 copies at indices 2, 5 and 8 were never new input — they are the same history[0] object the callback re-emitted. The identical behavior occurs with Runner.run_streamed().

Expected behavior

An item the callback took from the history argument must never be persisted as new-turn input, no matter how many times the callback emits it. Only genuinely new items belong in the append batch.

This is stated in .agents/references/session-persistence.md:

Existing history must not be re-appended as new input, even when session_input_callback deep-copies, reorders, filters, duplicates, or reconstructs items.

and in the prepare_input_with_session() docstring itself:

The callback may reorder, drop, or duplicate items. […] so retries and custom merge strategies do not accidentally re-persist old history as fresh input.

Reordering, deep-copying, filtering, and reconstructing all behave correctly today; only repetition of a history item is mishandled.

Root-cause hypothesis (hypothesis, not verified as the maintainers' intent)

_consume_reference() treats history object identity as a one-shot resource: it pops the matched object from the reference list. Object identity with an item in history_for_callback is, however, definitive proof that the item came from history and stays true for every occurrence. Because the identity match also decrements the content-frequency budget for that key, no fallback covers the repeat, and the classifier's final else branch persists it.

Proposed focused scope

Treat a history object-identity match as a durable fact rather than a consumable reference, so repeated occurrences of the same history object are pruned from the persistence batch while remaining in the model input. The content-frequency fallback for reconstructed history items should stay as-is, since it is what keeps a genuinely new item that coincidentally serializes identically to a history item from being dropped. Regression tests would cover repeat, reorder, deep-copy, filter, and reconstruct callbacks in both streaming and non-streaming modes, plus the case where new input duplicates history content.

User impact

Applications whose session_input_callback re-injects an earlier message — a common "restate the original goal at the end of the context" or RAG-style re-injection pattern — silently and permanently corrupt their stored conversation history, and the duplicate count grows every turn. The damage persists in the session store after the process exits.

I'd like to work on this and open a PR if that's welcome.