Skip to content

test: tolerate OpenClaw JSON envelope changes#4038

Merged
jyaunches merged 2 commits into
NVIDIA:mainfrom
jyaunches:fix/openclaw-json-e2e-parser
May 22, 2026
Merged

test: tolerate OpenClaw JSON envelope changes#4038
jyaunches merged 2 commits into
NVIDIA:mainfrom
jyaunches:fix/openclaw-json-e2e-parser

Conversation

@jyaunches
Copy link
Copy Markdown
Contributor

@jyaunches jyaunches commented May 22, 2026

Summary

  • add a shared E2E helper for extracting assistant reply text from openclaw agent --json
  • tolerate both legacy result.payloads[] and current top-level payloads[] envelopes, plus common nested content shapes
  • update OpenClaw-agent E2E assertions to use the helper instead of brittle inline parsers

Validation

  • bash -n test/e2e/lib/openclaw-json.sh test/e2e/test-messaging-compatible-endpoint.sh test/e2e/test-brave-search-e2e.sh test/e2e/test-launchable-smoke.sh test/e2e/test-sandbox-operations.sh test/e2e/test-openclaw-inference-switch.sh test/e2e/test-bedrock-runtime-compatible-anthropic.sh test/e2e/test-full-e2e.sh
  • manual helper checks for top-level payloads[], legacy result.payloads[], prefixed JSON, and OpenAI-style choices[].message.content
  • git diff --check

Fixes the false negative seen in messaging-compatible-endpoint C8 where the agent returned PONG in top-level payloads[] but the test only parsed result.payloads[].

Summary by CodeRabbit

  • Tests
    • Centralized JSON response parsing into a shared test utility for reuse.
    • Updated end-to-end tests to call the shared parser instead of repeating inline parsing logic.
    • Improved robustness of test parsing so assistant reply text is extracted reliably from diverse JSON envelopes.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 22, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e2e62f78-19a4-4d81-b057-248bcdc4df7a

📥 Commits

Reviewing files that changed from the base of the PR and between f177ded and 8121991.

📒 Files selected for processing (1)
  • test/e2e/lib/openclaw-json.sh
💤 Files with no reviewable changes (1)
  • test/e2e/lib/openclaw-json.sh

📝 Walkthrough

Walkthrough

Adds a shared parse_openclaw_agent_text() helper (shell script with embedded Python) to extract assistant-visible text from openclaw agent --json output, and updates multiple E2E tests to source and use this parser instead of inline JSON-extraction one-liners.

Changes

OpenClaw JSON Parsing Consolidation

Layer / File(s) Summary
Robust OpenClaw JSON text parser
test/e2e/lib/openclaw-json.sh
Introduces parse_openclaw_agent_text() shell function with embedded Python that extracts assistant-visible text from openclaw agent --json output by traversing multiple envelope shapes, aggregating likely text fields, deduplicating visited objects by identity, and providing a partial-JSON fallback when full parsing fails.
Migrate E2E test suites to shared parser
test/e2e/test-bedrock-runtime-compatible-anthropic.sh, test/e2e/test-brave-search-e2e.sh, test/e2e/test-full-e2e.sh, test/e2e/test-launchable-smoke.sh, test/e2e/test-messaging-compatible-endpoint.sh, test/e2e/test-openclaw-inference-switch.sh, test/e2e/test-sandbox-operations.sh
Seven E2E test scripts source the new helper and replace inline Python JSON parsing with calls to parse_openclaw_agent_text, updating reply extraction and related comments while preserving existing assertion and error-tolerance behavior.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

Integration: OpenClaw

Suggested reviewers

  • cv

Poem

🐰 Hopped through JSON, keys in tow,
I gathered text where it would show.
One helper now replaces seven,
Tests sing softly — parse it, heaven! 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main change: introducing tolerance for OpenClaw JSON envelope variations by extracting text via a shared helper function instead of brittle inline parsers.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (4)
test/e2e/test-messaging-compatible-endpoint.sh (1)

40-41: Run the messaging-compatible endpoint nightly lane for this change.

Please run:
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=messaging-compatible-endpoint-e2e

As per coding guidelines, test/e2e/test-messaging-compatible-endpoint.sh recommends messaging-compatible-endpoint-e2e.

Also applies to: 530-530

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/test-messaging-compatible-endpoint.sh` around lines 40 - 41, The PR
needs the nightly messaging-compatible-endpoint E2E lane run; trigger the GitHub
Actions workflow for this branch by running: gh workflow run nightly-e2e.yaml
--ref <branch> -f jobs=messaging-compatible-endpoint-e2e, ensuring you use the
branch name for this PR and confirm the job name matches the
test/e2e/test-messaging-compatible-endpoint.sh recommendation
(messaging-compatible-endpoint-e2e); if the workflow is missing, add or update
the nightly-e2e.yaml to include that job before triggering.
test/e2e/test-openclaw-inference-switch.sh (1)

296-297: Run the inference-switch nightly lane after migrating to the shared parser.

Please run:
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=openclaw-inference-switch-e2e

As per coding guidelines, test/e2e/test-openclaw-inference-switch.sh recommends openclaw-inference-switch-e2e.

Also applies to: 277-277

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/test-openclaw-inference-switch.sh` around lines 296 - 297, The test
suite file test-openclaw-inference-switch.sh should trigger the new nightly job
after migrating to the shared parser; run the GitHub Actions workflow with the
command gh workflow run nightly-e2e.yaml --ref <branch> -f
jobs=openclaw-inference-switch-e2e (replace <branch> with your branch) so the
openclaw-inference-switch-e2e lane is executed; confirm the job runs and update
any CI docs or PR description to note the nightly invocation for
test-openclaw-inference-switch.sh.
test/e2e/test-bedrock-runtime-compatible-anthropic.sh (1)

28-29: Run the dedicated Bedrock-compatible E2E lane for this parser migration.

Please run:
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=bedrock-runtime-compatible-anthropic-e2e
to validate this path end-to-end after switching to the shared JSON parser.

As per coding guidelines, test/e2e/test-bedrock-runtime-compatible-anthropic.sh recommends running bedrock-runtime-compatible-anthropic-e2e.

Also applies to: 703-703

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/test-bedrock-runtime-compatible-anthropic.sh` around lines 28 - 29,
The test script test/e2e/test-bedrock-runtime-compatible-anthropic.sh recommends
running the dedicated Bedrock-compatible E2E lane but the PR hasn't validated
it; run the nightly workflow for that lane to validate the shared JSON parser
path by executing: gh workflow run nightly-e2e.yaml --ref <branch> -f
jobs=bedrock-runtime-compatible-anthropic-e2e (replace <branch> with your
branch), then confirm the job completed successfully and attach the run link or
results to the PR so the bedrock-runtime-compatible-anthropic-e2e path is
verified.
test/e2e/test-full-e2e.sh (1)

72-73: Run the two nightly lanes tied to this full onboard path.

Please run:
gh workflow run nightly-e2e.yaml --ref <branch> -f jobs=cloud-e2e,openclaw-onboard-security-posture-e2e

As per coding guidelines, test/e2e/test-full-e2e.sh recommends cloud-e2e and openclaw-onboard-security-posture-e2e.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/test-full-e2e.sh` around lines 72 - 73, Update the E2E run to
trigger the two nightly lanes for the full onboard path: from the test script
(test-full-e2e.sh) invoke the GitHub workflow "nightly-e2e.yaml" for your branch
using the GitHub CLI and specify jobs=cloud-e2e and
jobs=openclaw-onboard-security-posture-e2e (i.e., run the workflow with gh
workflow run nightly-e2e.yaml --ref <branch> -f
jobs=cloud-e2e,openclaw-onboard-security-posture-e2e) so the cloud-e2e and
openclaw-onboard-security-posture-e2e lanes are executed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/e2e/test-bedrock-runtime-compatible-anthropic.sh`:
- Around line 28-29: The test script
test/e2e/test-bedrock-runtime-compatible-anthropic.sh recommends running the
dedicated Bedrock-compatible E2E lane but the PR hasn't validated it; run the
nightly workflow for that lane to validate the shared JSON parser path by
executing: gh workflow run nightly-e2e.yaml --ref <branch> -f
jobs=bedrock-runtime-compatible-anthropic-e2e (replace <branch> with your
branch), then confirm the job completed successfully and attach the run link or
results to the PR so the bedrock-runtime-compatible-anthropic-e2e path is
verified.

In `@test/e2e/test-full-e2e.sh`:
- Around line 72-73: Update the E2E run to trigger the two nightly lanes for the
full onboard path: from the test script (test-full-e2e.sh) invoke the GitHub
workflow "nightly-e2e.yaml" for your branch using the GitHub CLI and specify
jobs=cloud-e2e and jobs=openclaw-onboard-security-posture-e2e (i.e., run the
workflow with gh workflow run nightly-e2e.yaml --ref <branch> -f
jobs=cloud-e2e,openclaw-onboard-security-posture-e2e) so the cloud-e2e and
openclaw-onboard-security-posture-e2e lanes are executed.

In `@test/e2e/test-messaging-compatible-endpoint.sh`:
- Around line 40-41: The PR needs the nightly messaging-compatible-endpoint E2E
lane run; trigger the GitHub Actions workflow for this branch by running: gh
workflow run nightly-e2e.yaml --ref <branch> -f
jobs=messaging-compatible-endpoint-e2e, ensuring you use the branch name for
this PR and confirm the job name matches the
test/e2e/test-messaging-compatible-endpoint.sh recommendation
(messaging-compatible-endpoint-e2e); if the workflow is missing, add or update
the nightly-e2e.yaml to include that job before triggering.

In `@test/e2e/test-openclaw-inference-switch.sh`:
- Around line 296-297: The test suite file test-openclaw-inference-switch.sh
should trigger the new nightly job after migrating to the shared parser; run the
GitHub Actions workflow with the command gh workflow run nightly-e2e.yaml --ref
<branch> -f jobs=openclaw-inference-switch-e2e (replace <branch> with your
branch) so the openclaw-inference-switch-e2e lane is executed; confirm the job
runs and update any CI docs or PR description to note the nightly invocation for
test-openclaw-inference-switch.sh.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f1bc1ef0-bfda-476d-823b-64ee47950bea

📥 Commits

Reviewing files that changed from the base of the PR and between 1bdb519 and f177ded.

📒 Files selected for processing (8)
  • test/e2e/lib/openclaw-json.sh
  • test/e2e/test-bedrock-runtime-compatible-anthropic.sh
  • test/e2e/test-brave-search-e2e.sh
  • test/e2e/test-full-e2e.sh
  • test/e2e/test-launchable-smoke.sh
  • test/e2e/test-messaging-compatible-endpoint.sh
  • test/e2e/test-openclaw-inference-switch.sh
  • test/e2e/test-sandbox-operations.sh

@jyaunches jyaunches merged commit ad5e185 into NVIDIA:main May 22, 2026
78 checks passed
@github-actions
Copy link
Copy Markdown
Contributor

Selective E2E Results — ✅ All requested jobs passed

Run: 26263753126
Target ref: fix/openclaw-json-e2e-parser
Requested jobs: messaging-compatible-endpoint-e2e,openclaw-inference-switch-e2e,bedrock-runtime-compatible-anthropic-e2e,cloud-e2e,openclaw-onboard-security-posture-e2e
Summary: 5 passed, 0 failed, 0 skipped

Job Result
bedrock-runtime-compatible-anthropic-e2e ✅ success
cloud-e2e ✅ success
messaging-compatible-endpoint-e2e ✅ success
openclaw-inference-switch-e2e ✅ success
openclaw-onboard-security-posture-e2e ✅ success

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants