Skip to content

test(e2e): add dashboard remote-bind regression guard for #3259#3410

Merged
ericksoa merged 5 commits into
mainfrom
test/dashboard-remote-bind-e2e-guard
May 13, 2026
Merged

test(e2e): add dashboard remote-bind regression guard for #3259#3410
ericksoa merged 5 commits into
mainfrom
test/dashboard-remote-bind-e2e-guard

Conversation

@jyaunches
Copy link
Copy Markdown
Contributor

@jyaunches jyaunches commented May 12, 2026

Summary

Adds a failing-test-first regression E2E guard for #3259 in the new regression-e2e.yaml holding-pen workflow.

This guard verifies that an explicitly opted-in remote dashboard bind restarts the OpenShell dashboard forward on all interfaces (0.0.0.0:<port>), rather than leaving the dashboard localhost-only.

Regression holding pen

This is intentionally not added to scheduled nightly-e2e.yaml. The job lives in regression-e2e.yaml so it can be explicitly dispatched while the fix is in flight and later reviewed/promoted into nightly when stable.

Expected red-on-unfixed-code behavior

Until #3259 is fixed, the regression job is expected to fail with:

FAIL: Dashboard forward is still localhost-only; expected 0.0.0.0:18789

Once this merges, /skill:nemoclaw-issue-kickoff #3259 can produce the fix PR against this acceptance criterion.

Validation

  • bash -n test/e2e/test-dashboard-remote-bind.sh
  • YAML parse for regression-e2e.yaml, e2e-branch-validation.yaml, nightly-e2e.yaml
  • npm run build:cli

Related: #3259

Summary by CodeRabbit

  • Tests

    • Added a new E2E test validating dashboard remote bind behavior plus a supporting remote-check script.
    • Tightened timeout on an existing CLI debug test.
  • CI / Workflows

    • Added a manually-triggerable regression E2E workflow.
    • Extended E2E dispatch options to include the new dashboard-remote-bind test suite.

Review Change Stack

@jyaunches jyaunches added the v0.0.40 Release target label May 12, 2026
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 12, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5fd68e80-3c3d-4fbd-be79-8b7b4b868849

📥 Commits

Reviewing files that changed from the base of the PR and between a4753b8 and adbc9cc.

📒 Files selected for processing (1)
  • test/e2e/test-dashboard-remote-bind.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/e2e/test-dashboard-remote-bind.sh

📝 Walkthrough

Walkthrough

Adds a new E2E test suite dashboard-remote-bind: registers it in the branch validation workflow, adds a manual regression-e2e workflow, integrates a conditional Brev test that runs a remote validation script, and implements a Bash script that verifies remote dashboard forwards bind to all interfaces.

Changes

Dashboard Remote Bind E2E Test Suite

Layer / File(s) Summary
Test Suite Registration
.github/workflows/e2e-branch-validation.yaml
Register dashboard-remote-bind in the workflow header documentation and add it as a workflow_dispatch input option.
Regression E2E Workflow
.github/workflows/regression-e2e.yaml
New workflow enabling manual dispatch of regression E2E tests with configurable pr_number, jobs, and keep_alive. Delegates to the reusable e2e-branch-validation workflow, passing dashboard-remote-bind and parameters.
Test Case Integration
test/e2e/brev-e2e.test.ts
Add new test case gated to run when TEST_SUITE === "dashboard-remote-bind" or "all". Executes remote script test/e2e/test-dashboard-remote-bind.sh and asserts PASS present and no FAIL: output (300s timeout).
Remote Dashboard Bind Validation Script
test/e2e/test-dashboard-remote-bind.sh
Bash script with helpers and env derivation; stops existing OpenShell forward, runs nemoclaw ... connect with NEMOCLAW_DASHBOARD_BIND=0.0.0.0, inspects OpenShell forwards and asserts bind address indicates all-interface binding (not localhost-only); prints PASS on success.
Test Timeout Adjustment
test/cli.test.ts
Tighten a unit test timeout by passing 15_000 to testTimeoutOptions(...).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 I hopped through workflows, tests, and shell,
I nudged the bind so dashboards ring a bell,
From dispatch to remote, the checks align,
I scanned the forwards, ensured they span each line,
A cheerful rabbit claps: "All hosts — you're fine!"

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the primary change: adding a dashboard remote-bind regression guard for issue #3259, which aligns with the PR's objectives.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/dashboard-remote-bind-e2e-guard

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: regression-e2e/dashboard-remote-bind-e2e, e2e-branch-validation/full

Dispatch hint: dashboard-remote-bind-e2e

Workflow run

Full advisor summary

Pi Semantic E2E Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. This PR is purely CI/test infrastructure: it adds a new failing-test-first coverage guard (test-dashboard-remote-bind.sh), wires it into the existing brev-e2e vitest harness and the e2e-branch-validation reusable workflow, and introduces a regression-e2e.yaml holding-pen workflow to dispatch it. No product source files under src/, bin/, scripts/ (runtime), or installer paths are modified, so no user-facing behavior, security boundary, inference route, or onboarding path can regress from this change. The new guard itself is expected to fail on main until the owning fix lands (per the script and workflow headers), so requiring it would incorrectly block this test-only PR.

Optional E2E

  • regression-e2e/dashboard-remote-bind-e2e (~$0.10 Brev CPU credits, ~10-15 min): Exercises the newly-wired regression-e2e.yaml -> e2e-branch-validation.yaml -> test-dashboard-remote-bind.sh path to confirm the CI wiring invokes the test script correctly. Per the script's own header and the regression-e2e.yaml description, this is a failing-test-first coverage guard expected to FAIL on main until the remote-bind fix lands, so it is informational and MUST NOT block merge.
  • e2e-branch-validation/full (~$0.10 Brev CPU credits, ~10 min): The reusable workflow e2e-branch-validation.yaml was edited (new choice option + passthrough). Running the 'full' suite once confirms the existing test_suite switch still resolves correctly and no other caller (nightly-e2e, etc.) regressed. Low signal-to-cost; recommended only if other workflow callers are in flight.

New E2E recommendations

  • None.

Dispatch hint

  • Workflow: regression-e2e.yaml
  • jobs input: dashboard-remote-bind-e2e

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/regression-e2e.yaml:
- Around line 39-41: The concurrency group key used in the workflow (the
concurrency: group value: regression-e2e-${{ github.event_name }}-${{ github.ref
}}-${{ inputs.jobs || 'all' }}) is too coarse and can cancel unrelated PR runs;
update the group to include a per-run discriminator such as inputs.pr_number (or
another unique run identifier) so manual runs for different PRs or runs against
the same ref won’t cancel each other — modify the group expression to append ${
{ inputs.pr_number } } (or equivalent) to the existing key used by the
concurrency block.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b895cd5d-fb13-4622-8cd8-bab7418edd2e

📥 Commits

Reviewing files that changed from the base of the PR and between edb7478 and 1d24d82.

📒 Files selected for processing (4)
  • .github/workflows/e2e-branch-validation.yaml
  • .github/workflows/regression-e2e.yaml
  • test/e2e/brev-e2e.test.ts
  • test/e2e/test-dashboard-remote-bind.sh

Comment thread .github/workflows/regression-e2e.yaml
Comment thread test/e2e/test-dashboard-remote-bind.sh Fixed
@wscurran wscurran added E2E End-to-end testing — Brev infrastructure, test cases, nightly failures, and coverage gaps enhancement: testing Use this label to identify requests to improve NemoClaw test coverage. labels May 12, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/test-dashboard-remote-bind.sh`:
- Around line 41-45: The test currently writes logs to a fixed path
/tmp/nemoclaw-dashboard-remote-bind-connect.log which can collide across runs;
change the test to create a unique temp file (mktemp) and store its path in a
variable, use that variable in the nemoclaw connect redirection (when invoking
with NEMOCLAW_DASHBOARD_BIND and SANDBOX_NAME), and register a cleanup trap to
remove the temp file on exit and to cat it on failure; ensure references to the
temp file variable replace the hardcoded path in both the success/failure
branches.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fecc7ae3-6841-4518-9809-65b98679ff87

📥 Commits

Reviewing files that changed from the base of the PR and between 1d24d82 and ed46d04.

📒 Files selected for processing (2)
  • .github/workflows/regression-e2e.yaml
  • test/e2e/test-dashboard-remote-bind.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • .github/workflows/regression-e2e.yaml

Comment thread test/e2e/test-dashboard-remote-bind.sh Outdated
Copy link
Copy Markdown
Contributor

@ericksoa ericksoa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the current head against current main. This is a CI-only regression guard for #3259, merges cleanly, CodeRabbit's prior comments are addressed, and the remaining risk is limited to the manual regression E2E wiring.

@ericksoa ericksoa merged commit 30b2061 into main May 13, 2026
23 of 24 checks passed
@ericksoa ericksoa deleted the test/dashboard-remote-bind-e2e-guard branch May 13, 2026 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

E2E End-to-end testing — Brev infrastructure, test cases, nightly failures, and coverage gaps enhancement: testing Use this label to identify requests to improve NemoClaw test coverage. v0.0.40 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants