You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Content appended by a transform_llm_output plugin is passed to the external memory provider (Honcho) via _sync_external_memory_for_turn. Because the memory sync fires aftertransform_llm_output, it receives the fully-transformed final_response — including any content the plugin appended that was intended only for display. On future turns, Honcho retrieves that content as part of the conversation context, so the model sees the appended text as if it were part of the original assistant response.
Steps to Reproduce
Configure Hermes with a Honcho memory provider (hermes memory set honcho).
Install a plugin that implements transform_llm_output and appends a display-only suffix — for example, a citation, disclaimer, or any content that should not become part of the agent's memory.
Send a message that triggers the plugin to append content.
On the next turn, ask the agent to recall or summarize the previous exchange.
Observe that the agent references the appended display-only content as if it were part of its own prior response.
Root Cause Analysis
In agent/conversation_loop.py, the execution order is:
# Line ~4375agent._persist_session(messages, conversation_history)
# → SQLite session DB gets the raw LLM response ✓# Line ~4448transform_llm_outputfires# → final_response is now raw response + plugin-appended content# Line ~4565agent._sync_external_memory_for_turn(
final_response=final_response, # ← transformed value, includes appended content
...
)
# → Honcho receives the appended content and stores it as the assistant turn ✗
The SQLite session DB and the OpenAI messages list (used for in-context history) are both written before the transform fires, so they correctly contain only the raw LLM response. But the external memory sync fires after, so Honcho gets the polluted value.
Why this is hard to fix without a new API
There are two legitimate plugin archetypes with opposite requirements:
Plugin type
What it does in transform_llm_output
What memory should receive
Display-only append (e.g. citation, shout)
Appends a suffix after the response
Raw LLM response (no suffix)
Content restoration (e.g. PII de-redaction)
Replaces placeholder tokens with real values
Restored content (not placeholders)
Simply passing pre_transform_response to _sync_external_memory_for_turn instead of final_response would fix the append case but break the restoration case. There is no single mechanical fix that works for both without some form of plugin-declared intent.
Potential approaches
Display-only flag in hook registration — when registering a transform_llm_output hook, plugins declare whether their transform is display_only=True (memory gets pre-transform) or display_only=False (memory gets post-transform, current behavior). The runtime passes the appropriate value to _sync_external_memory_for_turn.
Structured return from transform_llm_output — instead of returning a plain string, a plugin can optionally return {"display": "...", "memory": "..."}, allowing different values for display and memory. Plain string return preserves current behavior.
Bug Description
Content appended by a
transform_llm_outputplugin is passed to the external memory provider (Honcho) via_sync_external_memory_for_turn. Because the memory sync fires aftertransform_llm_output, it receives the fully-transformedfinal_response— including any content the plugin appended that was intended only for display. On future turns, Honcho retrieves that content as part of the conversation context, so the model sees the appended text as if it were part of the original assistant response.Steps to Reproduce
hermes memory set honcho).transform_llm_outputand appends a display-only suffix — for example, a citation, disclaimer, or any content that should not become part of the agent's memory.Root Cause Analysis
In
agent/conversation_loop.py, the execution order is:The SQLite session DB and the OpenAI
messageslist (used for in-context history) are both written before the transform fires, so they correctly contain only the raw LLM response. But the external memory sync fires after, so Honcho gets the polluted value.Why this is hard to fix without a new API
There are two legitimate plugin archetypes with opposite requirements:
transform_llm_outputSimply passing
pre_transform_responseto_sync_external_memory_for_turninstead offinal_responsewould fix the append case but break the restoration case. There is no single mechanical fix that works for both without some form of plugin-declared intent.Potential approaches
Display-only flag in hook registration — when registering a
transform_llm_outputhook, plugins declare whether their transform isdisplay_only=True(memory gets pre-transform) ordisplay_only=False(memory gets post-transform, current behavior). The runtime passes the appropriate value to_sync_external_memory_for_turn.Structured return from
transform_llm_output— instead of returning a plain string, a plugin can optionally return{"display": "...", "memory": "..."}, allowing different values for display and memory. Plain string return preserves current behavior.New pre-memory-sync hook — a
transform_memory_assistanthook (analogous to the proposedtransform_persisted_assistantin [Feature]: Add plugin hook for transforming assistant messages before session DB persistence (transform_persisted_assistant) #46574) fires immediately before_sync_external_memory_for_turnand lets plugins strip or modify what the memory provider receives, independently of what the display layer shows.Option 3 is most flexible and avoids a breaking change to the
transform_llm_outputcontract. It also composes naturally with #46574.Related
transform_persisted_assistanthook for controlling what the session DB receives (analogous problem, DB layer instead of memory layer)transform_llm_outputappended content silently dropped in CLI streaming (relatedtransform_llm_outputlimitation)Operating System
Ubuntu 24.04.4 LTS
Python Version
3.11.15
Hermes Version
0.18.0
Are you willing to submit a PR for this?
This requires a design decision on the right API shape before implementation — happy to contribute a PR once there's consensus on approach.