Skip to content

fix: gateway boots when a configured plugin payload is broken#110239

Merged
steipete merged 6 commits into
openclaw:mainfrom
VACInc:feat/plugin-payload-degrade-and-boot
Jul 18, 2026
Merged

fix: gateway boots when a configured plugin payload is broken#110239
steipete merged 6 commits into
openclaw:mainfrom
VACInc:feat/plugin-payload-degrade-and-boot

Conversation

@VACInc

@VACInc VACInc commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Refs #109833

PR #110200 supplied typed package-verification reasons on main; this PR adds the bounded Gateway readiness policy for failures with a known plugin owner.

What Problem This Solves

Gateway startup was blocked when a configured and enabled plugin package failed payload verification, even when the failure belonged to one known plugin surface that could be isolated safely.

Why This Change Was Made

This implements the maintainer decision on #109833: degrade and boot. Startup quarantines the exact broken plugin install, emits a typed verification diagnostic, and prevents that payload from importing. Doctor, status, and health expose the owner as configured-unavailable.

The explicit openclaw update path still owns repair/install convergence. A current startup checkpoint performs static verification only, so Gateway boot does not repeat install or network repair work.

User Impact

Operators can keep the Gateway ready when one configured plugin package is broken. The broken plugin stays unavailable for that boot and its typed reason remains visible. State-migration failures, structurally invalid config, missing ownership, and ambiguous verification failures still block readiness.

Summary

  • Verifies active configured plugin payloads before import and records an exact-root, boot-stable quarantine snapshot.
  • Prevents quarantined payloads from reaching artifact resolution, import, or registration.
  • Keeps ownerless and pathless active failures fail-closed.
  • Allows a healthy explicit plugins.load.paths override with the same plugin id; stale install quarantine clears only after the override registers successfully.
  • Projects degraded state through doctor, status, and health without duplicating the synthetic loader error.
  • Redacts install roots and host peer-link targets from public diagnostics.
  • Sorts package-read diagnostics deterministically and documents recovery behavior.

Root Cause and Best-Fix Boundary

src/commands/doctor-config-preflight.ts treated every active plugin payload warning as a fatal startup diagnostic. That policy made one known plugin-owned package failure take down the entire Gateway.

Pre-import verification plus process-local exact-root quarantine is the narrow owner boundary. It preserves readiness without adding config, persistent fallback state, dual readers, or repeated convergence. Loader, doctor, health, status, and docs consume the same degraded-state contract.

Real Behavior Proof

The contributor exercised a packed npm plugin with activation.onStartup: true in an isolated state directory. Its installed package.json was changed to mode 000; a same-user read returned EACCES.

Baseline at merge base 03d5d88a14e4:

gateway_rc=1
readyz_rc=7
plugin_register_marker=absent
OpenClaw plugin verification failed; refusing to report the gateway ready.

Contributor head 9ac87a5d7ba, broken payload:

ready=yes status_rc=0 health_rc=0 doctor_rc=0
plugin_register_marker=absent
typed_startup_diagnostic_count=1
GET /readyz -> {"ready":true,"failing":[]}
status.degradedPlugins -> proof-degrade-plugin / configured-unavailable / unreadable-package-json

Second startup with the same broken payload remained ready and kept the plugin quarantined. Restoring mode 0644 removed the degraded state, loaded the plugin, and made its RPC method callable. A structurally invalid Gateway config still exited with code 78 and never reached readiness.

All temporary processes were stopped and the isolated proof state was removed.

Maintainer Review Improvements

The reviewed head b58bf3b37e78 also fixes issues found during maintainer review:

  • splits static startup verification from one-time repair convergence;
  • binds quarantine and health deduplication to canonical plugin roots, including symlink aliases;
  • preserves the original degraded state when a same-id override fails to import or register;
  • removes duplicate health reporting and closes peer-link path disclosure;
  • adds override success/failure, ownerless failure, second-boot, health redaction, and readiness regressions.

Evidence

Rebased-head Blacksmith Testbox run 29626869038:

Test files  15 passed
Tests       526 passed
Shards      unit-fast, gateway, CLI, commands

Additional closeout:

  • focused Gateway quarantine: 3 passed;
  • focused health snapshot: 18 passed;
  • CI-fix Testbox run 29627441509: 44 focused tests and pnpm deadcode:full passed;
  • targeted format and oxlint checks passed after splitting the health regression into its own focused test file;
  • git diff --check: passed;
  • fresh branch-wide autoreview after resolving the current-main rebase: no accepted or actionable findings; patch correct, confidence 0.86;
  • fresh autoreview of the CI-fix delta: no accepted or actionable findings; patch correct, confidence 0.96;
  • docs/tools/plugin.md: updated with startup quarantine, diagnostics, and recovery behavior.

What Was Not Tested

  • No live channel/provider traffic was needed; the exercised surfaces were package verification, startup/readiness, plugin import/registration, status, health, doctor, recovery, and invalid-config blocking.
  • Exact-head hosted prepare gates remain the merge authority.

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime cli CLI command changes commands Command implementations size: L labels Jul 17, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jul 17, 2026
@clawsweeper

clawsweeper Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 17, 2026, 10:45 PM ET / July 18, 2026, 02:45 UTC.

Summary
The branch quarantines a configured plugin whose installed payload fails static verification for the current boot, keeps the Gateway ready when ownership is unambiguous, and reports the plugin as configured-unavailable through loader, doctor, status, health, and documentation.

PR surface: Source +593, Tests +993, Docs +7. Total +1593 across 29 files.

Reproducibility: yes. at source level and in the supplied real-run evidence: configure an active managed plugin, make its installed payload unreadable or otherwise fail static verification, then start the Gateway and inspect readiness plus plugin status. I could not independently execute that path in this read-only review environment.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: migration/backfill/repair: docs/tools/plugin.md, migration/backfill/repair: src/cli/update-cli/post-core-plugin-convergence.test.ts, migration/backfill/repair: src/commands/doctor-config-preflight.state-migration.test.ts, migration/backfill/repair: src/commands/doctor-config-preflight.ts, migration/backfill/repair: src/commands/doctor-gateway-health.test.ts, migration/backfill/repair: src/commands/doctor-plugin-registry.test.ts, and 6 more. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #109833
Summary: This PR is the candidate implementation of the remaining Gateway-readiness policy requested by the canonical unreadable-plugin-payload issue; the earlier merged classification fix is adjacent but did not change readiness.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Record the plugin/startup owner’s decision on the readiness contract before merge.

Risk before merge

  • [P1] Merging changes the upgrade-time behavior for an existing configured plugin from Gateway startup failure to a ready Gateway with that plugin unavailable; operators must be able to notice and repair the degraded plugin through the exposed doctor, status, and health surfaces.
  • [P1] The boot-local quarantine must remain limited to the verified plugin root and must not conceal ambiguous ownership, invalid configuration, migration failures, or a failed same-id override registration.

Maintainer options:

  1. Approve the bounded readiness policy (recommended)
    Accept that a known broken configured plugin becomes unavailable for the boot while the rest of the Gateway remains ready and visibly degraded.
  2. Keep strict startup validation
    Do not merge the readiness-policy change if maintaining fail-closed startup for every enabled configured plugin is the intended upgrade contract.

Next step before merge

  • [P2] No narrow mechanical repair is identified; the next action is the assigned owner’s explicit decision on the startup compatibility policy before merge.

Maintainer decision needed

  • Question: Should Gateway startup remain ready while a single configured plugin with a known exact-root payload-verification failure is quarantined and reported as configured-unavailable?
  • Rationale: This is an intentional compatibility and availability policy change, not a defect that code review can settle: existing operators may experience a ready Gateway with one configured capability absent instead of a startup failure.
  • Likely owner: steipete — They are assigned to the PR and authored the latest hardening commits implementing the policy boundary.
  • Options:
    • Accept bounded quarantine (recommended): Merge the exact-root, boot-local quarantine policy, relying on doctor, status, and health to make the unavailable configured plugin visible.
    • Keep fail-closed startup: Retain the merged typed diagnostics from the prerequisite fix but reject readiness when any enabled configured plugin payload cannot be verified.
    • Require an explicit policy surface: Defer merge until maintainers choose a documented opt-in or other product policy for availability versus strict startup validation.

Security
Cleared: The inspected diff adds no dependency source, workflow permission, secret, package-resolution, or new code-execution surface; it also adds public-diagnostic redaction for install-root and peer-link details.

Review details

Best possible solution:

Land only after the plugin/startup owner explicitly accepts degrade-and-boot as the supported availability policy for known, exact-root plugin payload verification failures, with fail-closed handling retained for ambiguous or global startup failures.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level and in the supplied real-run evidence: configure an active managed plugin, make its installed payload unreadable or otherwise fail static verification, then start the Gateway and inspect readiness plus plugin status. I could not independently execute that path in this read-only review environment.

Is this the best way to solve the issue?

Unclear pending owner intent: the exact-root, boot-local quarantine is a coherent bounded implementation, but whether a ready Gateway with one configured plugin absent is preferable to the current fail-closed contract is a maintainer product and upgrade-policy choice.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against bd8c5399d86d.

Label changes

Label justifications:

  • P1: The PR addresses a real Gateway availability failure for operators with one broken configured plugin payload.
  • merge-risk: 🚨 compatibility: Existing installations change from startup failure to a ready process with one configured plugin unavailable.
  • merge-risk: 🚨 availability: The diff deliberately changes the Gateway readiness decision for plugin payload verification failures.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body provides a concrete isolated before/after Gateway run with a real unreadable plugin payload, readiness and status observations, a second-boot check, recovery after permission restoration, and a negative invalid-config control; redact private paths and identifiers in any follow-up evidence.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides a concrete isolated before/after Gateway run with a real unreadable plugin payload, readiness and status observations, a second-boot check, recovery after permission restoration, and a negative invalid-config control; redact private paths and identifiers in any follow-up evidence.
Evidence reviewed

PR surface:

Source +593, Tests +993, Docs +7. Total +1593 across 29 files.

View PR surface stats
Area Files Added Removed Net
Source 17 709 116 +593
Tests 11 1008 15 +993
Docs 1 7 0 +7
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 29 1724 131 +1593

What I checked:

  • Real behavior evidence: The PR body documents an isolated Linux before/after run: the baseline refused readiness for an unreadable active plugin payload, while the branch reported readiness, skipped plugin import, exposed configured-unavailable, remained quarantined on a second boot, and recovered after permissions were restored. (src/gateway/server-startup-plugin-quarantine.test.ts:1, b58bf3b37e78)
  • Bounded startup policy: The patch separates active-payload static verification from repair convergence and converts only known-owner verification failures into boot-local degraded-plugin records; ownerless, pathless, structurally invalid-config, and migration failures remain blocking according to the PR description and changed preflight surface. (src/commands/doctor-config-preflight.ts:155, b58bf3b37e78)
  • No payload import under quarantine: The loader checks the active degraded state before runtime artifact resolution, records the configured-unavailable registry result, and returns before activation for the exact quarantined plugin root. (src/plugins/loader-runtime-candidate.ts:151, b58bf3b37e78)
  • Current PR provenance: The latest two PR commits are authored by steipete and explicitly harden plugin payload quarantine and satisfy the related checks, which is strong evidence that the current branch reflects active owner review rather than an abandoned proposal. (src/plugins/runtime-degraded-state.ts:1, b58bf3b37e78)
  • Adjacent merged prerequisite: The related merged PR supplied the distinct typed unreadable-package-json verification classification while deliberately preserving fail-closed startup; this PR owns the subsequent readiness-policy decision rather than duplicating that diagnostic fix. (src/cli/update-cli/plugin-payload-validation.ts:1, 748fc044192a)
  • Current-main status: The supplied current-main SHA is bd8c5399d86d7d244b5cfa3295d19a2698aa2275, while this PR remains open at b58bf3b37e7824b648e5f36e7c108c66a1ffd20a; the degraded-and-ready policy is not established as merged current-main behavior. (bd8c5399d86d)

Likely related people:

  • steipete: They are assigned to this PR and authored the two latest hardening commits that establish the current quarantine behavior and CI follow-up. (role: recent area contributor and reviewer; confidence: high; commits: 07c06142a84f, b58bf3b37e78; files: src/commands/doctor-config-preflight.ts, src/plugins/runtime-degraded-state.ts, src/gateway/server-startup-plugin-quarantine.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (6 earlier review cycles)
  • reviewed 2026-07-17T23:35:50.040Z sha 6733485 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-17T23:43:09.202Z sha f604236 :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-18T00:38:19.472Z sha 9ac87a5 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-18T01:20:42.138Z sha 9ac87a5 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-18T02:14:47.234Z sha b8246de :: needs maintainer review before merge. :: none
  • reviewed 2026-07-18T02:29:01.516Z sha 07c0614 :: needs maintainer review before merge. :: none

@VACInc
VACInc force-pushed the feat/plugin-payload-degrade-and-boot branch from 6733485 to f604236 Compare July 17, 2026 23:39
@VACInc

VACInc commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 18, 2026
@steipete steipete self-assigned this Jul 18, 2026
@openclaw-barnacle openclaw-barnacle Bot added the docs Improvements or additions to documentation label Jul 18, 2026
@steipete
steipete force-pushed the feat/plugin-payload-degrade-and-boot branch from b8246de to 07c0614 Compare July 18, 2026 02:26
Co-authored-by: VACInc <3279061+VACInc@users.noreply.github.com>
@steipete
steipete merged commit cd1ab40 into openclaw:main Jul 18, 2026
119 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 18, 2026
…aw#110239)

* fix: quarantine broken plugins during gateway startup

* fix(plugins): preserve degraded boot on package read errors

* fix(gateway): emit quarantine diagnostic once

* fix(gateway): refresh plugin quarantine every boot

* fix(gateway): harden plugin payload quarantine

Co-authored-by: VACInc <3279061+VACInc@users.noreply.github.com>

* fix(ci): satisfy plugin quarantine checks

Co-authored-by: VACInc <3279061+VACInc@users.noreply.github.com>

---------

Co-authored-by: VACInc <3279061+VACInc@users.noreply.github.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli CLI command changes commands Command implementations docs Improvements or additions to documentation gateway Gateway runtime merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: XL status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants