Skip to content

[Bug] active-memory circuit breaker too aggressive; fallback prompt pollutes main session (v2026.6.1) #90082

Closed

Description

@ryswork1993

Summary

The active-memory plugin's circuit breaker is far too aggressive by default, and when it opens, OpenClaw silently injects a fallback prompt string ("Agent couldn't generate a response. Please try again.") into the main session's prompt context. This causes two user-visible problems:

  1. The circuit breaker opens permanently after only a handful of transient failures, with no visible status indicator anywhere in the user-facing UI.
  2. The fallback string gets injected into the main LLM prompt, polluting context and misleading the main model about what just happened.

Reproduction

  1. Enable plugins.entries.active-memory (set enabled: true).
  2. Configure a model that occasionally returns an empty response (e.g. minimax/MiniMax-M3).
  3. Send 3–5 user messages. The model returns empty 1–2 times → empty response detected → retrying 1/1 fires.
  4. After ~5–7 cumulative failures, the circuit breaker opens permanently. Subsequent recalls return status=unavailable immediately without calling the model.
  5. The string Agent couldn't generate a response. Please try again. now appears in the injected untrusted context (<active_memory_plugin> block) of every main-session prompt, indefinitely.
  6. Even if the underlying model recovers, the circuit breaker does not reset within a reasonable window — there is no way for the user to know it is open except by grepping gateway logs.

Expected

  • The default circuitBreakerMaxTimeouts should be high enough (e.g. ≥ 10) that transient flakiness does not permanently disable the plugin.
  • circuitBreakerCooldownMs should reset the breaker on a reasonable interval (e.g. ≤ 60 s) so the plugin self-heals.
  • When the circuit breaker is open, active-memory should either:
    • silently skip (no injected fallback string), or
    • emit a visible status indicator (log line, tool output, or heartbeat entry) so users can diagnose.
  • The empty response detected → retrying 1/1 path should not blindly retry the same prompt on the same model — consider degrading to a fallback model on the second retry.
  • The two config keys circuitBreakerMaxTimeouts and circuitBreakerCooldownMs are accepted at runtime but are not documented in the plugin schema and not surfaced in the gateway health report. They should be documented, validated, and visible in openclaw doctor / openclaw plugins describe active-memory.

Actual

  • After ~5–7 transient failures, circuit breaker opens.
  • The fallback string Agent couldn't generate a response. Please try again. is injected into the main session on every recall attempt.
  • No visible way to know the breaker is open, except reading gateway logs and counting active-memory done status=unavailable lines.
  • Default values appear to be roughly maxTimeouts ≤ 5 and cooldownMs ≥ 5 min, which is too strict for production traffic.

Environment

  • OpenClaw v2026.6.1 (commit 2e08f0f)
  • Platform: Linux container, Node v22.22.2
  • Model: minimax/MiniMax-M3 via https://api.minimaxi.com/anthropic/v1/messages (MiniMax platform)
  • Reproduced after fresh gateway restart with MINIMAX_API_KEY correctly injected in env

Workaround (current)

Set in plugins.entries.active-memory.config:

{
  "circuitBreakerMaxTimeouts": 20,
  "circuitBreakerCooldownMs": 60000
}

This reduces the failure rate but does not eliminate the fallback-string pollution.

Suggested fix

  1. Raise the default circuitBreakerMaxTimeouts to 10 and circuitBreakerCooldownMs to 60_000.
  2. When the breaker is open, do not inject the fallback string into the main session — return a silent no-op.
  3. Document the two config keys in the plugin schema and surface them in openclaw plugins describe active-memory.
  4. Add a circuitBreakerOpen flag to openclaw doctor so users can see at a glance when active-memory is degraded.
  5. On empty response detected, skip the same-model retry and go directly to a configured fallback model (e.g. plugin.config.fallbackModel).

Activity

  1. clawsweeper commented on Jun 4, 2026

    @clawsweeper
    Contributor

    Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

    Close as implemented: current Active Memory no longer turns an open circuit breaker into the reported fallback prompt, it resets after the documented one-minute cooldown, and the configuration is now validated and documented. Requests to raise the intentionally documented threshold further or add runtime model failover are separate product-policy work, not evidence that the reported current behavior remains broken.

    Root-cause cluster
    Relationship: fixed_by_candidate
    Canonical: #120658
    Summary: The merged repair directly addresses the failed-recall prompt-contamination portion of this report; current source also shows bounded breaker recovery.

    Members:

    Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

    So I’m closing this as already implemented rather than keeping a duplicate issue open.

    Review details

    Best possible solution:

    Keep the shipped error-exclusion and bounded circuit breaker; discuss any higher default threshold, Doctor state, or runtime model failover as a separate Active Memory product decision.

    Do we have a high-confidence way to reproduce the issue?

    No live current-main run was performed; source inspection shows the reported breaker path now returns a timeout with no summary and therefore cannot prepend the historical fallback text.

    Is this the best way to solve the issue?

    Yes for the reported defect: the merged owner-boundary repair excludes error payloads before context construction, while the breaker is bounded, resettable, documented, and schema-validated.

    Security review:

    Security review: This is issue triage with no proposed patch to assess.

    AGENTS.md: found and applied where relevant.

    What I checked:

    • Open breaker does not create the reported prompt context: The circuit-breaker branch returns a timeout result with a null summary and persists a session diagnostic; the prompt hook only prepends a summary or an unavailable-memory outcome, so this branch cannot inject the historical fallback response. (extensions/active-memory/recall.ts:246, bc82742f7b04)
    • Failed payloads are excluded from recall output: The recall runner removes error payloads before composing raw recall text, and the current regression table verifies a terminal incomplete-turn error yields no prepended context while retaining a failed status line. (extensions/active-memory/recall-run.ts:337, bc82742f7b04)
    • Recovery settings are now documented and schema-validated: Current source defines a three-timeout threshold and 60-second cooldown; the plugin manifest validates both keys and the public Active Memory documentation states their ranges, defaults, and reset behavior. (extensions/active-memory/types.ts:28, bc82742f7b04)
    • Merged repair provenance: The merged repair explicitly targets recall errors entering prompt context, and its commit is an ancestor of the checked-out current main. (c2a0e706e7a4)
    • Shipped-release provenance: The repair commit is contained in stable release v2026.9.1. (c2a0e706e7a4)

    Likely related people:

    • soltonigiri: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
    • ZOOWH: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
    • Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

    Codex review notes: model internal, reasoning high; reviewed against bc82742f7b04; fix evidence: release v2026.9.1, commit c2a0e706e7a4.

  2. added
    P1High-priority user-facing bug, regression, or broken workflow.
    impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.
    impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.
    clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.
    on Jun 4, 2026
  3. Hansen1018 commented on Jun 4, 2026

    @Hansen1018

    +1 Repro: same bug, two trigger modes (consolidated)

    I previously left two separate +1 comments with overlapping evidence. Consolidating into one clean repro now per maintainer workflow preference. Both comments deleted; this replaces them.

    Environment: OpenClaw v2026.6.1 (commit 2e08f0f), Linux container, Node v22, primary model minimax-portal/MiniMax-M3.


    Trigger A — OAuth 24h quota exhaustion

    minimax-portal runs on an OAuth token with a 24-hour usage cap separate from the token's expires_in. We hit usage: 24h 0% left while the token was still ok expires in 365d. The provider actively aborts every fetch with name=AbortError and elapsedMs of 11ms–3.3s — clearly not a network issue.

    Three raw model-fetch errors during one stuck window:

    19:37:29 [model-fetch] error provider=minimax-portal api=anthropic-messages model=MiniMax-M3 elapsedMs=3381 name=AbortError
    19:41:00 [model-fetch] error provider=minimax-portal api=anthropic-messages model=MiniMax-M3 elapsedMs=3381 name=AbortError
    19:44:53 [model-fetch] error provider=minimax-portal api=anthropic-messages model=MiniMax-M3 elapsedMs=11   name=AbortError
    

    active-memory sees the embedded fetch fail → empty response → emptyRetries=1/1 exhausted → incomplete turn ... surfacing error to user.

    Trigger B — inbound fallback injection (not just outbound)

    Once Trigger A (or transient flakiness) trips the breaker, the fallback string is injected into the inbound prompt envelope, not just the outbound reply. A user in a Telegram group sent a normal reply, and the main agent received this as the user-role block:

    <active_memory_plugin>
    ⚠️ Agent couldn't generate a response. Please try again.
    </active_memory_plugin>
    <actual user content>
    

    From the main agent's context window this is indistinguishable from the user typing that string verbatim. The <active_memory_plugin> framing is supposed to mark it as untrusted, but the surrounding <actual user content> arrives as a separate user message right after, so the main model often tries to answer the injected string instead of ignoring it.

    This means a non-trivial fraction of "weird bot replies" users observe may actually be the main LLM attempting to address the injected fallback block, not a real reply to the user's content.

    Repro determinism

    While OAuth 24h quota is at 0%, every inbound to any agent using minimax-portal as primary will arrive with this <active_memory_plugin> block prepended by before_prompt_build. We can keep producing timestamps as long as it lasts.

    Three asks for maintainers

    1. Wire circuitBreakerMaxTimeouts to a sane default (e.g. ≥ 10) and document it. Grepping the dist for the constant in 2e08f0f doesn't surface a clear binding — the schema / openclaw plugins describe active-memory should expose it.
    2. Treat quota-exhaustion as a permanent backoff, not a transient failure. It has a known reset clock (e.g. 24h OAuth cap), so the circuit breaker holding it open for 5+ minutes serves no purpose — the underlying cause is not flapping.
    3. Stop injecting the fallback string as a user-role block. When the breaker is open, active-memory should either silently skip (no injected content) or emit a clearly-machine-tagged side channel (log / heartbeat / tool result) the main model is not asked to address. The current <active_memory_plugin> framing is not strong enough to prevent the main LLM from answering the string.

    Happy to provide more logs, gateway config dump, or a config-side reproduction that doesn't burn OAuth quota.

  4. added
    P2Normal backlog priority with limited blast radius.
    and removed
    P1High-priority user-facing bug, regression, or broken workflow.
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    on Jul 30, 2026
  5. 13 remaining items

  6. added
    no-staleExclude from stale automation
    clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    and removed
    no-staleExclude from stale automation
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.
    on Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions