[Bug] Gateway does not call _post_turn_goal_continuation after each turn, preventing goal evaluation

[Avakiano]

The gateway (gateway/run.py) does not invoke _post_turn_goal_continuation() after each assistant turn. As a result, the goal judge (evaluate_after_turn) is never called, turns_used never increments past 0, and the /goal loop is effectively dead despite all configuration being correct.

Reproduction Steps

1. Set a goal: /goal Respond "hello" and stop.
2. Wait for the assistant to respond with "hello"
3. Check status: /goal status → shows 0/20 turns
4. The goal judge never evaluates the response, turns never increment

Expected Behavior

Per the documentation, after each assistant response the goal judge should fire, evaluate the response against the goal, increment turns_used, and either continue or mark the goal as done.

Root Cause

In gateway/run.py, after the agent produces a response:

_agent_result = await self._handle_message_with_agent(event, source, _quick_key, _run_generation)

The code immediately after (lines ~10272-10288) checks if _final_text.strip(): and then calls self.session_store.get_or_create_session(source) → _post_turn_goal_continuation(...). However, this section is never reached in actual message processing — the function returns before executing the goal continuation hook.

Evidence

The goal system itself works perfectly. Calling evaluate_after_turn() directly from a Python script:

mgr = GoalManager(session_id="<active_session_id>")
result = mgr.evaluate_after_turn(
last_response="Test completed.",
user_initiated=True,
)

→ verdict="done", turns_used=1, status="done", persisted to DB correctly

The auxiliary client resolves correctly:
get_text_auxiliary_client("goal_judge") → OpenAI client
Model: nvidia/nemotron-3-super-120b-a12b
Chat completions: available

The goal judge model responds with valid JSON:
{"done": true, "reason": "Goal completed."}

DB evidence:

• Querying state_meta for goal:<session_id> shows turns_used=0, last_verdict=None for ALL goals ever created via the gateway
• After direct evaluate_after_turn() call: turns_used=2, last_verdict="done"

Config Used

goals:
enabled: true
max_turns: 20
allow_early_stop: true
send_result_reply: true

auxiliary:
goal_judge:
provider: nvidia
model: nvidia/nemotron-3-super-120b-a12b
max_tokens: 1024
timeout: 120

Workaround

Until this is fixed, a cron-based watchdog can replicate the goal behavior:

Runs every 60s, detects new assistant messages, calls evaluate_after_turn

Only notifies on goal completion

See: /data/.hermes/scripts/goal_watchdog.py

Install: hermes cron create --no-agent --script goal_watchdog.py --schedule "every 1m"

Additional Context

• Hermes version: 0.18.2 (confirmed in pyproject.toml)
• Previous related bugs: [Bug]: curator.auxiliary config is dead — curator always runs on main model #17572 (curator auxiliary config ignored — different subsystem, same class of orchestration bug)
• Gateway log: No "goal judge:" entries appear in the log at any log level, confirming the judge is never invoked during normal operation

[GottZ]

This was generated by AI during triage.

Summary

Six PRs address or reference #62202, but only #23179, #58828, #62235, and #62273 target the reported streaming return-None boundary; #21591 changes goal kickoff architecture, while #45740 handles erroneous continuation after explicit completion. Among the direct fixes, #62273 places a best-effort continuation call where the streamed response remains available and preserves normal delivery on judge failure.

Related pull requests

• fix(gateway): run slash goals as background missions #21591 partial — (+382/-25) — partial fix: Replaces foreground FIFO goal kickoff with a dedicated background mission and adds lifecycle guards, addressing a different kickoff/replay path rather than the streamed-response hook missing in [Bug] Gateway does not call _post_turn_goal_continuation after each turn, preventing goal evaluation #62202. Despite the keep_open review, it should not be the consolidation target because the diff changes the documented same-session contract and its pause/resume ownership token can permit competing missions.
• fix(gateway): keep /goal running after streamed replies #23179 fixes — (+215/-11) — fixes: Preserves the final streamed result in a transient handoff so the outer post-turn hook can evaluate it while still returning None to prevent duplicate delivery. Despite the keep_open review, it is not preferred as-is because its compression-migration hunk is documented as superseded and the branch conflicts with current main; only the streamed-result handoff is relevant to [Bug] Gateway does not call _post_turn_goal_continuation after each turn, preventing goal evaluation #62202.
• fix(goal): stop loop on explicit completion #45740 unrelated — (+121/-5) — unrelated: Short-circuits the judge when completion wording is detected and changes deferred callback scheduling, addressing judge-failure looping after evaluation rather than the missing evaluation reported in [Bug] Gateway does not call _post_turn_goal_continuation after each turn, preventing goal evaluation #62202. Despite the keep_open review, this should remain separate because the shortcut can bypass contract and subgoal verification, while the callback portion is documented as superseded.
• fix: /goal turn counter stuck at 0 when streaming is enabled #58828 fixes — (+103/-0) — fixes: Adds a session-keyed, one-shot stash so the outer hook can recover text after the streaming path returns None, directly addressing the stuck counter and absent judge call. Despite the keep_open review, fix(gateway): run goal continuation for streaming already-sent turns #62273 is the better consolidation target: fix: /goal turn counter stuck at 0 when streaming is enabled #58828 adds mutable handoff state, and its added test manually pops the stash instead of exercising the patched streamed return-None boundary.
• fix(gateway): call post-turn goal continuation on streaming path to prevent dead /goal loop (#62202) #62235 [closed] partial — (+15/-0) — partial fix: The closed PR calls goal continuation directly before the streaming return None and remains relevant as an early reference for the correct code site. Despite its keep_open review, its closure as a duplicate is appropriate because the diff lacks the required best-effort judge error boundary, uses the synchronous session store on the event loop, and adds no regression test.
• fix(gateway): run goal continuation for streaming already-sent turns #62273 [closed] best fix — (+47/-0) — best fix: The closed PR invokes goal continuation inside the already-sent streaming branch while the actual response and session entry remain available, and catches judge failures so a delivered response is not converted into an agent error. It remains relevant as the verify-selected reference implementation, although its test exercises the helper directly rather than proving the complete handler branch.

Duplicates

#23179, #58828, #62235, and #62273 substantially duplicate the same streamed-response continuation fix; #62235 and #62273 use a direct in-branch call, while #23179 and #58828 hand the streamed result back to the outer hook. #21591 and #45740 address distinct failure modes and are not strict duplicates.

Suggested consolidation

Merge #62273 after reopening/rebasing and strengthening its regression test to exercise the already-sent handler branch — it is the verify-selected best fix and has the narrowest direct implementation with the necessary error boundary. This recommendation explicitly differs from contributor triage marking #62273 duplicate of #54222: #54222 is outside the supplied candidate set, while the provided #62273 diff contains the response-in-scope call and failure isolation needed for #62202. Close #23179 and #58828 as duplicate implementations despite their keep_open reviews; keep #62235 closed as a superseded duplicate, and handle the distinct #21591 and #45740 concerns separately rather than merging them for #62202.

Complex graph

flowchart TD
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I62202(["issue #62202 (open)"])
    P21591["PR #21591 (open)"]
    subgraph Dup23179 ["PRs duplicating each other"]
        P23179["PR #23179 (open)"]
        P58828["PR #58828 (open)"]
        P62235["PR #62235 (closed)"]
        P62273["PR #62273 (closed)"]
    end
    P21591 -.->|partial| I62202
    P23179 -->|fixes| I62202
    P58828 -->|fixes| I62202
    P62235 -.->|partial| I62202
    P62273 -->|best fix| I62202
    class I62202 open
    class I62202 target
    class P21591 open
    class P23179 open
    class P58828 open
    class P62235 closed
    class P62273 closed
    class P62273 best
    click I62202 "https://github.com/NousResearch/hermes-agent/issues/62202"
    click P21591 "https://github.com/NousResearch/hermes-agent/pull/21591"
    click P23179 "https://github.com/NousResearch/hermes-agent/pull/23179"
    click P58828 "https://github.com/NousResearch/hermes-agent/pull/58828"
    click P62235 "https://github.com/NousResearch/hermes-agent/pull/62235"
    click P62273 "https://github.com/NousResearch/hermes-agent/pull/62273"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed or no verify verdict yet (state tag in the node label).

Cross-PR triage: Reviewed 6 pull requests and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 60 kB of PR diffs, 13 kB of issue/PR text, 13 kB of discussion (16 comments), 12 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[GottZ]

This was generated by AI during triage.

Delta since our previous triage comment

Since the previous comment, #21591 has closed, while @pgjh rebased #58828 onto current main and reported six targeted tests passing. This strengthens #58828’s currency, but the evidence remains author-reported because CI is still awaiting approval, and its regression test still manually performs the stash handoff rather than exercising the streamed return-None handler path.

Changed pull requests

• fix(gateway): run slash goals as background missions #21591 [closed] partial — (+382/-25) — partial fix, now closed: Despite the keep_open review, its closure is consistent with the unresolved same-session contract and resume-ownership concerns; it remains relevant only as a reference for the distinct background-mission approach, not as the [Bug] Gateway does not call _post_turn_goal_continuation after each turn, preventing goal evaluation #62202 streaming fix.
• fix: /goal turn counter stuck at 0 when streaming is enabled #58828 fixes — (+142/-0) — fixes, newly rebased: @pgjh reports the branch is now 0 commits behind with six targeted tests passing, addressing the earlier staleness concern. Despite the keep_open review, this does not change consolidation because CI has not run independently and the diff’s test manually pops the stash instead of driving the patched streamed return-None boundary.

Suggested consolidation

The recommendation changes because no candidate is currently merge-lane eligible: retain closed #62273 as the verify-selected best existing fix and reference implementation, but salvage its guarded direct-call approach into the canonical current-main fix rather than reopening it; #23179 and #58828 remain duplicate implementations.

Complex graph

flowchart TD
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I62202(["issue #62202 (open)"])
    P21591["PR #21591 (closed)"]
    subgraph Dup23179 ["PRs duplicating each other"]
        P23179["PR #23179 (open)"]
        P58828["PR #58828 (open)"]
        P62235["PR #62235 (closed)"]
        P62273["PR #62273 (closed)"]
    end
    P21591 -.->|partial| I62202
    P23179 -->|fixes| I62202
    P58828 -->|fixes| I62202
    P62235 -.->|partial| I62202
    P62273 -->|best fix| I62202
    class I62202 open
    class I62202 target
    class P21591 closed
    class P23179 open
    class P58828 open
    class P62235 closed
    class P62273 closed
    class P62273 best
    click I62202 "https://github.com/NousResearch/hermes-agent/issues/62202"
    click P21591 "https://github.com/NousResearch/hermes-agent/pull/21591"
    click P23179 "https://github.com/NousResearch/hermes-agent/pull/23179"
    click P58828 "https://github.com/NousResearch/hermes-agent/pull/58828"
    click P62235 "https://github.com/NousResearch/hermes-agent/pull/62235"
    click P62273 "https://github.com/NousResearch/hermes-agent/pull/62273"
    classDef changedSince stroke:#dc2626,stroke-width:3px,stroke-dasharray: 6 3
    class P21591 changedSince

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).
Dashed red border = changed since our previous triage comment (1 added/updated, 1 removed graph elements).

Cross-PR triage: Reviewed 6 pull requests and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 61 kB of PR diffs, 14 kB of issue/PR text, 10 kB of discussion (14 comments), 6 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[dnazhang-max]

Additional root-cause evidence: streaming platforms are specifically broken

Confirming this bug on v0.20.0 (Windows gateway, Telegram DM). Adding a more precise root cause than "the function returns before the hook" — the hook IS reached, but it receives None on every streamed turn.

The exact short-circuit path (current main, 9d4ef04ed):

1. _handle_message calls _handle_message_with_agent, then reads the result:

   _agent_result = await self._handle_message_with_agent(...)
   _final_text = ""
   if isinstance(_agent_result, dict):
       _final_text = str(_agent_result.get("final_response") or "")
   elif isinstance(_agent_result, str):
       _final_text = _agent_result
   if _final_text.strip():   # <-- gate for the judge hook
       ... _post_turn_goal_continuation(...)

2. But on the streaming-delivery path, _handle_message_with_agent ends with:

   if agent_result.get("already_sent") and not agent_result.get("failed"):
       ...
       return None          # <-- streamed turns return None
   return response

3. So _agent_result is None → neither dict nor str → _final_text == "" → the hook is silently skipped. The except around the hook only fires on exceptions, so there is zero logging — the loop just never runs.

Consequence: every platform that streams final replies (Telegram in my case; presumably Desktop streaming too) can never trigger the judge. Non-streamed platforms work. That matches the report here — the hook "is never reached" because it is starved of input on streamed turns.

Control experiments (all on the same install):

• judge_goal() called directly: returns verdict=done correctly in ~5s.
• GoalManager(session_id=...).evaluate_after_turn(...) in a fresh process: state machine works, persists to state_meta correctly.
• Goal registration via /goal: state row created, status=active, correct session_id matching the live session.
• DB evidence: turns_used=0, last_verdict=None after a full multi-turn goal run where the completion criteria were verifiably met.

Local workaround I'm running (minimal, does not change the return None contract that callers like the CLI handoff path depend on): stash response.get("final_response") into a per-session-key dict in the streamed branch, and have the goal hook pop it when _agent_result yields no text. Happy to open a PR if maintainers want it upstream — but the cleaner fix is probably moving the hook invocation inside _handle_message_with_agent before the return None, where response is still in scope.