Skip to content

[Bug]: Context compaction summaries leak into chat as assistant responses when inserted with role="assistant" #42768

Open

Description

@gordan-bobic

Bug Description

Context compaction summaries are leaking into chat channels as visible assistant messages. In affected sessions, 29 of 30 assistant messages were compaction summaries — the model became trapped in a compaction loop, repeatedly echoing the summary back as its own response.

A part of this was fixed in #20169 in response to issues
#11475
#14521
but the fix seems to be incomplete because it doesn't cover the case where the ordering of messages by random alternating chance ends up landing in role=assistant slot.

The proposed fix is to always override to role=user for compaction summaries because of how some LLMs put more weight on user input than their own output, and from there the existing fix in PR 20169 will take care of the rest.

Steps to Reproduce

  1. Run a long-lived session that accumulates enough context to trigger compaction (observed with 1400+ messages, ~75K+ tokens)

  2. Use a compression model that is the same as the main model on a shared llama.cpp instance (so compression competes for GPU time)

  3. Compression request times out at the default 120s → falls back to deterministic summary

  4. Fallback summary is inserted with role="assistant" (see root cause below)

  5. Model sees the summary as its own prior output and reproduces it verbatim as its next response

  6. Response adds to context → triggers compaction again → loop

Expected Behavior

  1. Memory compaction should not have resulted in the summary being output back into the chat session.

  2. When both LLM summary and detereministic summary failed, compaction should probably give up as from that point on it will only end up pointlessly burning more GPU without ever compacting the context until some external intervention happens.

Actual Behavior

A large message starting with:

[CONTEXT COMPACTION — REFERENCE ONLY]

is emitted to the chat stream, as "assistant" role.

Affected Component

Agent Core (conversation loop, context compression, memory)

Messaging Platform (if gateway-related)

N/A (CLI only)

Debug Report

Report       https://paste.rs/X2Ldb
agent.log    https://paste.rs/IrqoN
gateway.log  https://paste.rs/7hg1H

Operating System

EL10

Python Version

3.12.13

Hermes Version

0.16.0

Additional Logs / Traceback (optional)

Root Cause Analysis (optional)

Two bugs in agent/context_compressor.py:

Bug 1 — assistant role summaries lack the end marker (primary cause of leak)

The role selection logic at line ~1997 picks a role to avoid consecutive same-role collisions with head and tail messages. When last_head_role="assistant" AND first_tail_role="user", both roles collide, so it falls through to role="assistant".

PR #20169 added the --- END OF CONTEXT SUMMARY --- end marker to prevent the model from treating the summary as fresh instructions — but guarded it with and summary_role == "user". Summaries inserted as role="assistant" never get the marker, so the model echoes them as its own prior output.

# Line ~2012 — only "user" role gets the end marker:
if not _merge_summary_into_tail and summary_role == "user":
    summary = summary + "\n\n--- END OF CONTEXT SUMMARY ---"

Bug 2 — Deterministic fallback doesn't count as ineffective (causes infinite loop)

When the LLM compression times out, the deterministic fallback produces minimal compression (e.g. 249→224 messages). The anti-thrashing logic at ~line 739 stops retrying after 2 ineffective compressions — but the fallback path never increments _ineffective_compression_count, so it keeps retrying indefinitely.

Proposed Fix (optional)

  1. Mark fallback as ineffective so the anti-thrashing logic won't retry compression on the next turn
    Possible part-mitigation for this is to increase auxiliary.compression.timeout from default 120 to something much larger when large contexts and/or slower hardware are used.

  2. Always use role="user" with end marker for memory compaction summary output

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Activity

  1. added
    type/bugSomething isn't working
    P2Medium — degraded but workaround exists
    comp/agentCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint
    on Jun 9, 2026
  2. GottZ commented on Jul 28, 2026

    @GottZ

    This was generated by AI during triage.

    Summary

    Four PRs address #42768 by preventing compaction summaries from appearing as assistant output; #42783, #42804, and #42897 cover variants of the user-role change, while #42830 most completely combines the role invariant, unconditional end marker, fallback anti-thrashing, and regression coverage. The relevant fixes were subsequently implemented on main through #45153 and #52160/#56197, with current role handling preserving provider alternation constraints.

    Related pull requests

    Duplicates

    #42783, #42804, #42830, and #42897 substantially overlap on preventing assistant-role compaction summaries; #42830 is the most complete historical implementation, while #42804 and #42897 are narrower duplicates and #42783 is its untested focused precursor.

    Suggested consolidation

    Do not reopen or merge #42830: although Verify selected it as the best fix in this cluster, the blocking contributor review explains from the evolved main diff that #45153 and #52160/#56197 already cover the leak and loop while preserving the current alternation invariant. Retain #42830 as the reference implementation, keep #42783, #42804, and #42897 closed as duplicates/superseded, and verify #42768 against current main before closing the issue as already fixed.

    Complex graph

    flowchart LR
        classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
        classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
        classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
        classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
        classDef best stroke-width:3px,stroke:#b45309
        classDef target stroke-width:3px,stroke:#4338ca
        I42768(["issue #42768 (open)"])
        subgraph Dup42783 ["PRs duplicating each other"]
            P42783["PR #42783 (closed)"]
            P42804["PR #42804 (closed)"]
            P42830["PR #42830 (closed)"]
            P42897["PR #42897 (closed)"]
        end
        P42783 -.->|partial| I42768
        P42804 -.->|partial| I42768
        P42830 -->|best fix| I42768
        P42897 -.->|partial| I42768
        class I42768 open
        class I42768 target
        class P42783 closed
        class P42804 closed
        class P42830 closed
        class P42897 closed
        class P42830 best
        click I42768 "https://github.com/NousResearch/hermes-agent/issues/42768"
        click P42783 "https://github.com/NousResearch/hermes-agent/pull/42783"
        click P42804 "https://github.com/NousResearch/hermes-agent/pull/42804"
        click P42830 "https://github.com/NousResearch/hermes-agent/pull/42830"
        click P42897 "https://github.com/NousResearch/hermes-agent/pull/42897"
    
    Loading

    Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed or no verify verdict yet (state tag in the node label).

    Cross-PR triage: Reviewed 4 pull requests and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 52 kB of PR diffs, 16 kB of issue/PR text, 5 kB of discussion (9 comments), 8 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

  3. pastyGRB commented on Aug 8, 2026

    @pastyGRB

    Related bug: compaction summary spliced into an existing role="user" message (not just injected as a new role="assistant" message)

    Filing as a comment on this issue since it's the same underlying subsystem (agent/context_compressor.py) and the same class of failure (compaction summary text bleeding into the wrong conversational role), but the trigger and the concrete symptom are different from the original report, so I want to make sure the reproduction case doesn't get lost if a fix lands here that only addresses the role="assistant" injection path.

    What we were trying to do

    User uploaded an image into an active session and asked for analysis of it (a vision request). This was a normal, single-turn "here's an image, please look at it" interaction — nothing unusual about the session up to that point.

    What happened

    The vision/analysis call on that image did not complete successfully (appears to have failed or timed out — we did not get a clean assistant reply attached to that turn). This left the session's tail in an unusual state: the last real message in the transcript was a lone, unanswered role="user" message carrying the uploaded image, with no assistant response after it.

    Some time later, context compaction ran on this session (transcript was otherwise short). Because the tail was just system prompt → single unanswered user (image) turn, the compactor's merge-into-tail logic decided it needed to merge the LLM-generated compaction summary into that trailing message rather than insert it as a new one.

    The result: the assistant-voice, third-person compaction summary ("Historical Task Snapshot / Completed Actions / Remaining Work…") got spliced directly into the content of the user's own image message, but the message's role stayed "user". From that point on, replaying/reading the session, it looks exactly like the user typed a paragraph describing "what needs to be done next" — i.e., the agent's own planning output got attributed to the human. This is worse than simple visible pollution: it silently rewrites the historical record of who said what.

    Because the session transcript remained small after this, compaction re-triggered on subsequent turns and stacked additional summary text onto the same user message across multiple passes — we saw this happen at least twice more in the same session (three affected messages total).

    Why this is distinct from the original issue on this thread

    • The original report here concerns summaries injected as new messages with role="assistant", which then get echoed back by the model in a loop. The suggested fix in this thread is "always force role="user" for injected compaction summaries."
    • Our case is closer to the mirror image: the merge path spliced assistant-authored text into an already-existing, genuine role="user" message. If the fix for this issue is "always coerce injected-summary role to user," that would not catch our case — arguably it makes the failure mode structurally more likely, since it removes the one signal (role="assistant") that could otherwise distinguish "text the agent generated" from "text the user actually typed."

    Root cause (our analysis)

    In agent/context_compressor.py, the compaction routine has a merge path — referred to here as _merge_summary_into_tail (name from our read of the logic; please forgive if the current symbol name differs slightly) — that is taken when the compactor determines the transcript tail is too sparse/short to insert the summary as its own standalone message (e.g., only one trailing message present, or roles that would otherwise "collide" if a new message were inserted). This path:

    1. Takes the LLM-generated compaction summary (assistant-voice, describing completed actions / remaining work).
    2. Splices that summary text directly into the existing tail message's content field.
    3. Never inspects or updates that message's role.

    When the tail message being merged into happens to be a genuine, unanswered role="user" message (as in our case — an image upload with no assistant reply yet), the merged result is a message that is 100% agent-authored content permanently stored under role="user". There is no marker anywhere in the message distinguishing "original user content" from "injected compaction summary" — so any downstream consumer (including us, and presumably other users) reading the transcript has no way to tell this apart from something the human actually typed.

    Steps to reproduce

    1. Start a session and upload an image, asking for analysis/description of it.
    2. Arrange for (or simulate) the vision/analysis call on that turn to fail or time out, such that no assistant response gets attached — the session's last message remains a role="user" message containing only the image + request, with nothing after it.
    3. Trigger context compaction on that session (e.g., via normal token-budget compaction, or by forcing it if there's a debug/manual trigger) while the transcript tail is still just system → lone unanswered user/image message.
    4. Observe: the compactor takes the merge-into-tail branch (because the tail is too sparse to insert a standalone summary message), and splices the LLM's compaction summary text into that user message's content.
    5. Inspect the resulting message: role is still "user", but content now contains an assistant-voice summary (typically structured like "Historical Task Snapshot," "Completed Actions," "Remaining Work," etc.) glued onto/after the original image request.
    6. If the transcript remains short, keep interacting — compaction re-triggers on subsequent turns and repeats step 4 against the same message, stacking multiple summary blocks into that one role="user" entry.

    Concrete instance we observed

    • Session ID: 20260808_155141_936d6fdf
    • Primary affected message ID: 10961 — assistant-voice compaction summary text stored under role="user", attached to the user's uploaded-image turn.
    • Recurrence on the same session after repeated compaction passes: message IDs 10985 and 10990 — same mechanism, more summary boilerplate appended to the same image-upload user message.
    • The image that was originally uploaded (triggering the failed vision call that left the turn unanswered) is attached to this write-up / will be attached to the GitHub issue for reference.

    Suggested fix direction

    Whatever change lands for the role="assistant" injection case in this issue should also cover this merge-into-tail path:

    1. Never splice assistant-authored summary content into a message whose role is "user". If the compactor decides a standalone summary message can't be inserted because the tail is too sparse, it should still emit the summary as its own message (with clearly agent/system-attributed role or a dedicated metadata flag), rather than merging it into whatever message happens to be sitting there.
    2. Specifically guard against selecting a solitary, unanswered image-bearing (or any unanswered) role="user" message as a merge/attachment target — this is exactly the scenario where merging is most likely to corrupt the historical record, since there's no assistant turn to "absorb" the summary into safely.
    3. Consider a distinct role/marker for injected compaction summaries (e.g., role="system" or a metadata.injected_by="compactor" flag) so they are provably distinguishable from both genuine user and genuine assistant turns, regardless of which merge path is taken. This would resolve both the original role="assistant" echo-loop issue and our role="user" corruption case with one consistent mechanism.

    Happy to provide further transcript excerpts or logs if useful.

    Setup / environment details

    • hermes-agent version: 0.18.2 (from hermes_agent.egg-info/PKG-INFO)
    • Deployment: container built by pulling the latest hermes-agent repo at build time (no local modifications to delivered agent code); custom config/env/skills are supplied via mounted volumes only.
    • Python: 3.13.5
    • Key pinned deps (from package metadata): openai==2.24.0, httpx==0.28.1, pyyaml==6.0.3, tenacity==9.1.4, jinja2==3.1.6
    • Model in use on the affected session:
      • Primary: github_copilot/claude-sonnet-5 via a custom OpenAI-compatible router provider (custom:router), supports_vision: true
      • Fallback: github_copilot/gpt-5.4 (same router)
      • Configured context length for the primary model: 200,000 tokens (context_length: 200000 in custom_providers model map); fallback gpt-5.4 is configured at 256,000
    • Relevant compaction config (compression: block in config.yaml):
      • enabled: true
      • threshold: 0.5 (compaction triggers at 50% of context budget)
      • target_ratio: 0.2 (compress down to 20% of budget)
      • protect_last_n: 20, protect_first_n: 3
      • This is relevant to reproduction: with a small/short session (as in our case — essentially system prompt + one unanswered image turn), the absolute token count is tiny, but if the failed vision call or oversized image payload pushed estimated usage past the 50% threshold, compaction can still trigger on an otherwise "short" transcript, hitting the sparse-tail merge path described above.
    • Agent config: max_turns: 60, reasoning_effort: medium, verify_on_stop: false
    • Multi-agent context: this session's environment also has MoA (mixture-of-agents) and delegation enabled, but the affected session itself was a direct single-agent chat to a specialized profile, not a MoA/delegated run — noted in case that's relevant to whichever code path handles compaction across different execution modes.
    • We did not modify agent/context_compressor.py or any other delivered file — this is 100% stock behavior on a clean pull of the repo.

    Image being analyzed is provided below for context.

    Image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/agentCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpointtype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions