Codex/kova scenarios and fixes#2
Open
JuanHuaXu wants to merge 1 commit into
Open
Conversation
clawsweeper
Bot
added
rating: 🧂 unranked krab
|
Codex review: needs maintainer review before merge. Reviewed May 28, 2026, 2:07 AM ET / 06:07 UTC. Summary Reproducibility: not applicable. as a feature PR rather than a reported bug. The PR body provides current-head terminal proof for the new Kova runs and validation commands. Review metrics: 3 noteworthy metrics.
Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Land the expanded validation matrix only after maintainers accept the broader gate contract and operator requirements; otherwise split network-frontage or rolling-upgrade into smaller follow-ups. Do we have a high-confidence way to reproduce the issue? Not applicable as a feature PR rather than a reported bug. The PR body provides current-head terminal proof for the new Kova runs and validation commands. Is this the best way to solve the issue? Unclear until maintainer acceptance: the implementation is coherent and I found no discrete blocking line-level defect, but the PR changes the release/profile contract rather than making a narrow repair. Splitting operator-sensitive pieces is the safer alternative if maintainers want lower blast radius. AGENTS.md: found and applied where relevant. Codex review notes: model gpt-5.5, reasoning high; reviewed against 70525273b6ff. Label changesLabel changes:
Label justifications:
Evidence reviewedWhat I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
Author
|
Remediation pushed in What changed:
Validation:
So the reviewer-reported Kova false evidence paths are patched. The only failure observed in the exec proof run is now a real product resource signal, not Kova misidentifying its own helper behavior. |
clawsweeper
Bot
added
proof: sufficient
Author
|
Follow-up remediation pushed in Fixes:
Validation:
So the code-level review blockers are patched. The live loopback command reaches the expected alias setup path here, but this Codex session cannot grant elevated |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
🦞👀 Command router queued. I will update this comment with the next step. Re-review progress:
|
Author
|
Follow-up remediation pushed in Fixes:
Validation:
This addresses the reviewer concern by making the advertised dirty-plugin and release-update-recovery surface thresholds executable gates rather than planning-only metric names. |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
Author
|
Follow-up remediation pushed in Fixes:
Validation:
|
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
clawsweeper
Bot
added
status: 🔁 re-review loop
clawsweeper
Bot
added
rating: 🌊 off-meta tidepool
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
clawsweeper
Bot
added
proof: sufficient
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
clawsweeper
Bot
added
proof: sufficient
JuanHuaXu
force-pushed
the
codex/kova-scenarios-and-fixes
branch
from
ff5cd8e to
8a4d3a1
Compare
JuanHuaXu
force-pushed
the
codex/kova-scenarios-and-fixes
branch
from
8a4d3a1 to
24f5b9f
Compare
clawsweeper
Bot
added
rating: 🦐 gold shrimp
Author
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fresh Evidence
Latest Kova head:
24f5b9f(codex/kova-scenarios-and-fixes, single squashed PR commit on base7052527).Current-Head Proof
Reviewer proof refreshed on current PR head
24f5b9fafter the conflict-resolution squash and cron fail-closed amendment. Commands were run from this PR branch with disposable Kova homes undercheckouts/kova-pr2-proof-24f5b9f-*; artifact paths below are repo-relative and do not include private local paths.Validation:
node bin/kova.mjs self-check-> 182/182 passed, generated 2026-05-28 06:00 UTC.node tests/render-snapshots.mjs-> 21 pass, 0 fail, 0 written.rg -n "<<<<<<<|=======|>>>>>>>" . -g '!checkouts/**' -g '!node_modules/**' -g '!reports/**'-> no matches.ocm env list->No environments.git rev-list --count upstream/main..HEAD-> 1 commit.Targeted real-run proof:
cron-runtime/cron-userkova-260528-055817-79a0becronRuntimeEvidence.available: true;cronRegisterMs: 760;cronRunMs: 707;cronRunCompleted: true;cronTriggerAttributed: true; no violations.mcp-tool-call/mcp-tool-userkova-260528-055843-76d8f2mcpToolCallEvidence.available: true;toolsCallMs: 124;safeToolName: conversations_list;safeToolSucceeded: true;invalidToolErrorAttributed: true;processExited: true; no violations.exec-tool-safety/exec-tool-userkova-260528-055902-9617b8safeCommandSucceeded: true,dangerousCommandBlocked: true,dangerousPayloadExecuted: false,outputTruncated: true,timeoutMs: 3222,processLeaks: 0. Remaining failure is an OpenClaw resource signal:tool-runtime peak RSS 760.3 MB exceeded threshold 500 MB.Artifacts on runner:
checkouts/kova-pr2-proof-24f5b9f-cron/reports/kova-260528-055817-79a0be.jsoncheckouts/kova-pr2-proof-24f5b9f-mcp/reports/kova-260528-055843-76d8f2.jsoncheckouts/kova-pr2-proof-24f5b9f-exec/reports/kova-260528-055902-9617b8.jsonThis directly addresses ClawSweeper's stale-proof blocker: current-head proof now matches the branch state maintainers would merge. The only targeted runtime failure remains the known OpenClaw/product RSS threshold signal; cron/MCP/exec helper evidence itself is present, parsed, and gated.
Current-Head Exec Tool Evidence Proof
Reviewer P2 fix proof was run at Kova head
1546804(Drive exec containment checks through OpenClaw). This fixes the false-evidence lane by using the OpenClaw exec tool schema's requiredcommandargument and by driving safe, blocked, oversized-output, and timeout cases throughopenclaw agentplus mock-provider tool-result evidence.Commands:
KOVA_HOME=checkouts/p2-exec-real-proof2-kova-home \ node bin/kova.mjs run \ --target runtime:stable \ --scenario exec-tool-safety \ --state exec-tool-user \ --execute \ --report-dir checkouts/p2-exec-real-proof2/reports \ --json KOVA_HOME=checkouts/p2-exec-real-proof2-kova-home \ node bin/kova.mjs run \ --target runtime:stable \ --scenario tool-failure-containment \ --state exec-tool-user \ --execute \ --report-dir checkouts/p2-exec-real-proof2/reports \ --jsonResults:
exec-tool-safety:kova-260522-170514-192a6a, proof complete10/10; exec evidence available;safeCommandSucceeded: true;dangerousCommandBlocked: true;dangerousPayloadExecuted: false;outputTruncated: true;timeoutMs: 3499;processLeaks: 0.tool-failure-containment:kova-260522-170551-6419fb, proof complete10/10; exec evidence available;dangerousCommandBlocked: true;dangerousPayloadExecuted: false;outputTruncated: true;timeoutMs: 3291;processLeaks: 0.Both runs still report
FAIL, but only for the existing OpenClaw resource threshold:tool-runtimeRSS around 750 MB over the 500 MB threshold. The previous Kova evidence failures are gone.Validation for the patch:
node bin/kova.mjs self-check --json->ok: true(173 checks)node tests/render-snapshots.mjs-> 18 passgit diff --check-> passocm env list-> no environmentsArtifacts on runner:
checkouts/p2-exec-real-proof2/reports/kova-260522-170514-192a6a.jsoncheckouts/p2-exec-real-proof2/reports/kova-260522-170551-6419fb.jsonCurrent-Head Tool-Failure Proof
Reviewer P2 fix proof was run at Kova head
2439672(Fix failure-only exec mock provider flow).Command:
KOVA_HOME=checkouts/p2-tool-failure-proof-kova-home \ node bin/kova.mjs run \ --target runtime:stable \ --scenario tool-failure-containment \ --state exec-tool-user \ --execute \ --report-dir checkouts/p2-tool-failure-proof/reports \ --jsonResult:
kova-260522-164448-f2fbb2completed with proof completeness10/10required obligations, cleanup destroyed the disposable env, and the fixed failure-only provider path was exercised:kova-exec-tool-failure-onlykova-exec-tool-failure-only-dangerous-tool-callkova-exec-tool-failure-only-dangerous-tool-callexectool calldangerousCommandBlocked: truedangerousPayloadExecuted: falsedangerousSentinelStillPresent: trueoutputTruncated: true,timeoutObserved: true,processLeaks: 0The scenario verdict is still
FAIL, but for an OpenClaw resource threshold, not the Kova wiring bug:tool-runtime peak RSS 741.9 MB exceeded threshold 500 MB. This is expected to remain a product/resource signal for maintainers.Artifacts on runner:
checkouts/p2-tool-failure-proof/reports/kova-260522-164448-f2fbb2.mdcheckouts/p2-tool-failure-proof/reports/kova-260522-164448-f2fbb2.jsoncheckouts/p2-tool-failure-proof-kova-home/artifacts/kova-260522-164448-f2fbb2/kova-tool-failure-containmen-81131346-kova-260522-164448-f2fbb2/tool-failure-containment.jsonRequested OpenClaw target:
v2026.5.21-beta.1. The Git tag exists and resolves to89a17def chore(release): prepare 2026.5.21-beta.1, butopenclaw@2026.5.21-beta.1is not currently published in npm/OCM release discovery. A directnpm:2026.5.21-beta.1exhaustive run therefore blocked at provisioning withOpenClaw release version "2026.5.21-beta.1" was not found.Fresh release-shaped tag evidence was run from a disposable
openclaw/openclaw.gitcheckout at that tag usinglocal-build:checkouts/openclaw-v2026.5.21-beta.1. Afterpnpm install --frozen-lockfile, directpnpm packsucceeded and producedopenclaw-2026.5.21-beta.1.tgz.Matrix command:
KOVA_HOME=checkouts/kova-pr2-evidence-v2026-5-21-beta-1-local-build-rerun \ node bin/kova.mjs matrix run \ --profile exhaustive \ --target local-build:checkouts/openclaw-v2026.5.21-beta.1 \ --source-env kova-pr2-source-v2026-5-21-beta-1 \ --execute \ --allow-exhaustive \ --json \ --report-dir checkouts/kova-pr2-evidence-v2026-5-21-beta-1-local-build-rerun/reportsResult:
kova-260522-145534-037d84->77 total · 32 PASS · 45 FAIL · 0 BLOCKED.Evidence artifacts on runner:
checkouts/kova-pr2-evidence-v2026-5-21-beta-1-local-build-rerun/reports/kova-260522-145534-037d84-exhaustive.mdcheckouts/kova-pr2-evidence-v2026-5-21-beta-1-local-build-rerun/reports/kova-260522-145534-037d84-exhaustive.jsoncheckouts/kova-pr2-evidence-v2026-5-21-beta-1-local-build-rerun/reports/kova-260522-145534-037d84-bundle.tar.gz743ee926c809921c0b6aea0170c1731e8576749ab88ee8126cafebfb210c73aeNotable PASS coverage:
release-runtime-startup,channel-discord-capability-conformance,upgrade-existing-userboth source states,bundled-runtime-depsboth states,plugin-lifecycleall states, official/bad/missing/unsafe plugin lanes,provider-models,agent-cold-warm-message,dashboard-readiness,tui-responsiveness,mcp-runtime-start-stop,agent-network-offline,failure-injection, andcross-platform-smoke.Top remaining failures are product/resource signals in OpenClaw/tag behavior rather than Kova unsupported-mode blockers: gateway RSS around the 700 MB threshold across agent/provider/HTTP/TUI surfaces, rolling-upgrade package/runtime RSS/CPU, dirty-plugin
doctor-cliRSS, tool-runtime RSS for exec/tool containment, soak/workspace latency, and a few functional/liveness failures (channel generated-image handoff, Telegram timeout signals, cron/browser/media gateway restarting, MCP tool-call missing runtime role evidence).Cleanup after the run: disposable source env destroyed; old beta runtime records reintroduced by upgrade lanes removed;
ocm runtime listshows onlystable.PR Change List
Branch:
codex/kova-scenarios-and-fixesBase compared:
origin/mainRCA doc is removed from the feature set.
.learnings/is still untracked and not part of PR.Matrix/Profile Wiring
profiles/rolling-upgrade.jsonprofiles/exhaustive.json77profiles/release.json51profiles/adversarial.jsonNew Upgrade Coverage
scenarios/upgrade-from-day-ago.jsonscenarios/upgrade-from-week-ago.jsonscenarios/upgrade-from-month-ago.jsonsupport/resolve-openclaw-release-age.mjssupport/run-openclaw-release-age-upgrade.mjsdocs/AGENT_USAGE.mdwith rolling upgrade usageUnsafe Legacy Plugin Memory Test
scenarios/plugin-legacy-unsafe-memory.jsonsurfaces/plugin-legacy-unsafe-memory.jsonsupport/plugins/kova-legacy-unsafe-memory/index.jssupport/plugins/kova-legacy-unsafe-memory/openclaw.plugin.jsonsupport/plugins/kova-legacy-unsafe-memory/package.jsonsupport/assert-command-output.mjssrc/evaluator.mjsto countfailed during registeras plugin load failure evidenceDirty Plugin Testing
docs/DIRTY_PLUGIN_TESTING_PLAN.mdscenarios/dirty-plugin-state.jsonstates/dirty-plugin-local-edits.jsonstates/dirty-plugin-stale-deps.jsonstates/dirty-plugin-manifest-drift.jsonstates/dirty-plugin-disabled-broken.jsonstates/dirty-plugin-symlink-dev.jsonstates/dirty-plugin-partial-install.jsonstates/update-recovery-plugin-user.jsonsupport/dirty-plugin-state.mjssurfaces/dirty-plugin-state.jsonRelease Update Recovery
docs/RELEASE_UPDATE_RECOVERY_PLAN.mdscenarios/release-update-recovery.jsonsurfaces/release-update-recovery.jsonsupport/restore-first-ocm-upgrade-snapshot.mjsTool Runtime Matrix
docs/TOOL_RUNTIME_MATRIX_PLAN.mdscenarios/cron-runtime.jsonscenarios/exec-tool-safety.jsonscenarios/mcp-tool-call.jsonscenarios/tool-failure-containment.jsonstates/cron-user.jsonstates/exec-tool-user.jsonstates/mcp-tool-user.jsonsurfaces/cron-runtime.jsonsurfaces/exec-tool-safety.jsonsurfaces/mcp-tool-call.jsonsurfaces/tool-failure-containment.jsonprocess-roles/cron-runtime.jsonprocess-roles/tool-runtime.jsonsupport/run-cron-runtime-smoke.mjssupport/run-exec-tool-safety.mjssupport/mcp-tool-call-smoke.mjsProvider/Network Failure Coverage
docs/NETWORK_ISOLATION_PLAN.mdsrc/network-frontage.mjssupport/network-frontage-proxy.mjsscenarios/agent-provider-protocol-failure.jsonscenarios/agent-provider-random-disconnect.jsonsupport/mock-openai-server.mjssupport/configure-openclaw-mock-auth.mjssrc/commands/run.mjs,src/commands/matrix-run.mjs,src/run/context.mjs,src/run/phase-plan.mjsAdversarial Input Coverage
scenarios/adversarial-input-openai-compatible.jsonsurfaces/adversarial-input.jsonsupport/run-adversarial-inputs.mjsprofiles/adversarial.jsonPlugin Fixture/Manifest Fixes
support/plugins/kova-basic/openclaw.plugin.jsonsupport/plugins/kova-missing-runtime-dep/openclaw.plugin.jsonscenarios/plugin-missing-runtime-deps.jsonResource Attribution / Evaluation / Reporting Fixes
src/collectors/resources.mjssrc/evaluation/violations.mjssrc/evidence/agent-turns.mjssrc/evidence/shared.mjssrc/measurement-contract.mjssrc/reporting/report.mjssrc/reporting/scenario-aggregate.mjssrc/run/command-executor.mjssrc/run/report-finalization.mjssrc/runner.mjssrc/safety.mjssrc/selfcheck.mjsLarge Session Fixture
support/prepare-large-memory-session-state.mjssurfaces/fresh-install.jsonsurfaces/soak.jsonsurfaces/gateway-performance.jsonsurfaces/workspace-scan.jsonOpenAI-Compatible / Runtime Role Tweaks
scenarios/openai-compatible-turn.jsonsupport/run-openai-compatible-turn.mjsprocess-roles/openai-compatible-client.jsonDocs / User-Facing Metadata
README.mddocs/WHAT_IS_KOVA.mddocs/AGENT_USAGE.mdmetrics/known.jsonGit Hygiene
.gitignore.env,.env.*, local JSON/env files, and checkout contents.env.exampleandcheckouts/.gitkeepcheckouts/.gitkeepTests / Snapshots
tests/fixtures/reports/pass.jsontests/fixtures/reports/fail.jsontests/render-snapshots.mjstests/snapshots/Validation Already Run
node bin/kova.mjs self-check --jsonnpm run test:snapshotsgit diff --checkplugin-legacy-unsafe-memorypassed againstruntime:stable