✨ Add dry-run + kind cluster E2E tests for Mission Control#4233
✨ Add dry-run + kind cluster E2E tests for Mission Control#4233clubanderson merged 1 commit intomainfrom
Conversation
Two new test files: mission-control-dry-run.spec.ts (9 tests): Mock mode (6): isDryRun persistence, Dry Run button visibility, default value, DRY RUN badge, state round-trip, LaunchSequence headers Real clusters (3): dry-run cert-manager on vllm-d, observability on platform-eval, multi-project across both — NO resources created mission-control-kind-e2e.spec.ts (8 tests): 1. Create 3 kind clusters via console Local Clusters API 2-3. Deploy + verify observability stack (cert-manager, Prometheus, Grafana) 4-5. Deploy + verify security compliance (OPA Gatekeeper, Kyverno) 6. Deploy + verify GitOps pipeline (ArgoCD, cert-manager) 7. Multi-project stress: 6 projects across 2 kind clusters 8. Cleanup: delete all kind clusters via console API Kind clusters created via POST to kc-agent /local-clusters endpoint (same API the Settings > Local Clusters UI uses). Real deploys ONLY on local kind clusters — production clusters (vllm-d, platform-eval) use dry-run mode exclusively. Run mock tests: MOCK_AI=true npx playwright test e2e/mission-control-dry-run.spec.ts Run kind tests: KC_AGENT=true npx playwright test e2e/mission-control-kind-e2e.spec.ts Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kubestellar-prow
bot
added
the
dco-signoff: yes
✅ Deploy Preview for kubestellarconsole ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
👋 Hey @clubanderson — thanks for opening this PR!
This is an automated message. |
kubestellar-prow
bot
added
the
size/XXL
|
Thank you for your contribution! Your PR has been merged. Check out what's new:
Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey |
There was a problem hiding this comment.
Pull request overview
Adds Playwright E2E coverage for Mission Control’s dry-run mode and for real deployments onto locally-provisioned kind clusters, to validate UI state management and end-to-end deployment workflows.
Changes:
- Add Mission Control dry-run Playwright spec covering UI state persistence and (optionally) real-cluster dry-run validation.
- Add kind-based Mission Control E2E spec that provisions kind clusters via the Local Clusters API, deploys stacks, verifies resources, and cleans up.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 10 comments.
| File | Description |
|---|---|
| web/e2e/mission-control-dry-run.spec.ts | New Playwright suite for dry-run UI behavior and optional real-cluster dry-run validation. |
| web/e2e/mission-control-kind-e2e.spec.ts | New Playwright suite that provisions kind clusters via kc-agent, drives Mission Control deployments, verifies workloads/webhooks, and performs cleanup. |
| localStorage.setItem('kc_demo_mode', 'true') | ||
| localStorage.setItem('kc_onboarded', 'true') | ||
| localStorage.setItem('kc_user_cache', JSON.stringify({ |
There was a problem hiding this comment.
The demo/onboarding/user-cache localStorage keys here don’t match the app’s canonical keys (see web/src/lib/constants/storage.ts: STORAGE_KEY_DEMO_MODE='kc-demo-mode', STORAGE_KEY_ONBOARDED='demo-user-onboarded', STORAGE_KEY_USER_CACHE='kc-user-cache'). Using the underscore variants means demo mode/onboarding/user cache won’t be read by auth/demoMode logic, which can break navigation/auth in these tests. Update these setItem() calls to the canonical keys (or import the constants via an e2e helper).
| localStorage.setItem('kc_demo_mode', 'true') | |
| localStorage.setItem('kc_onboarded', 'true') | |
| localStorage.setItem('kc_user_cache', JSON.stringify({ | |
| localStorage.setItem('kc-demo-mode', 'true') | |
| localStorage.setItem('demo-user-onboarded', 'true') | |
| localStorage.setItem('kc-user-cache', JSON.stringify({ |
| localStorage.setItem('token', 'demo-token') | ||
| localStorage.setItem('kc_demo_mode', 'true') | ||
| localStorage.setItem('kc_onboarded', 'true') | ||
| localStorage.setItem('kc_user_cache', JSON.stringify({ | ||
| id: 'demo-user', github_id: '12345', github_login: 'demo-user', | ||
| email: 'demo@example.com', role: 'viewer', onboarded: true, | ||
| })) | ||
| }) |
There was a problem hiding this comment.
Same issue as above: these localStorage keys use underscore names that the app doesn’t read. This can lead to the app clearing the demo token or not recognizing onboarded/user state, making the test flaky or failing unexpectedly. Use 'kc-demo-mode', 'demo-user-onboarded', and 'kc-user-cache' (per web/src/lib/constants/storage.ts).
| if (clicked) { | ||
| } |
There was a problem hiding this comment.
This click path can silently do nothing: clicked is computed but never asserted, and the subsequent if (clicked) {} block is empty. If the button isn’t found (UI change/phase mismatch), the test will still proceed and may produce false results. Assert clicked is true (or fall back to a Playwright locator click with force: true) and remove the empty block.
| if (clicked) { | |
| } | |
| expect(clicked).toBe(true) |
| if (btn) { (btn as HTMLElement).click(); return true } | ||
| return false | ||
| }) | ||
| if (clicked) { |
There was a problem hiding this comment.
Same silent-noop issue here: clicked is not asserted and the empty if (clicked) {} block makes it easy for the test to pass without ever triggering a dry-run. Please assert the button was found/clicked (or use a locator-based click fallback) and drop the empty block.
| if (clicked) { | |
| // Fallback to a locator-based click so the test fails if the button cannot be triggered | |
| if (!clicked) { | |
| await page.getByRole('button', { name: 'Dry Run' }).click() |
| /** Markers the AI should include in dry-run output */ | ||
| const DRY_RUN_PROMPT_MARKER = '--dry-run=server' | ||
| const DRY_RUN_COMPLETION_MARKER = 'DRY RUN COMPLETE' | ||
|
|
There was a problem hiding this comment.
These constants are currently unused, which creates eslint warnings and also indicates the intended assertions aren’t implemented. Either remove them or update the real-cluster tests to assert the mission output includes '--dry-run=server' and a completion marker (as described in the header comment).
| /** Markers the AI should include in dry-run output */ | |
| const DRY_RUN_PROMPT_MARKER = '--dry-run=server' | |
| const DRY_RUN_COMPLETION_MARKER = 'DRY RUN COMPLETE' |
| test('9. dry-run multi-project across both clusters — verify DRY RUN markers', async ({ page }) => { | ||
| test.skip(!AGENT_MODE, 'Requires KC_AGENT=true and kc-agent running') | ||
|
|
||
| await seedAndOpenMC(page, { | ||
| phase: 'blueprint', | ||
| description: 'Multi-cluster dry-run: observability on vllm-d, security on platform-eval', | ||
| title: 'DR: Multi-Cluster Validation', | ||
| isDryRun: true, | ||
| projects: [...OBSERVABILITY_PROJECTS, ...SECURITY_PROJECTS], | ||
| assignments: [ | ||
| { | ||
| clusterName: CLUSTER_VLLM_D, clusterContext: CLUSTER_VLLM_D, provider: 'openshift', | ||
| projectNames: ['prometheus', 'grafana', 'cert-manager'], | ||
| warnings: [], readiness: { cpuHeadroomPercent: 50, memHeadroomPercent: 60, storageHeadroomPercent: 70, overallScore: 60 }, | ||
| }, | ||
| { | ||
| clusterName: CLUSTER_PLATFORM_EVAL, clusterContext: CLUSTER_PLATFORM_EVAL, provider: 'openshift', | ||
| projectNames: ['falco', 'kyverno'], | ||
| warnings: [], readiness: { cpuHeadroomPercent: 45, memHeadroomPercent: 55, storageHeadroomPercent: 75, overallScore: 58 }, | ||
| }, | ||
| ], | ||
| phases: [ | ||
| { phase: 1, name: 'Infrastructure', projectNames: ['cert-manager'], estimatedSeconds: 60 }, | ||
| { phase: 2, name: 'Observability', projectNames: ['prometheus', 'grafana'], estimatedSeconds: 120 }, | ||
| { phase: 3, name: 'Security', projectNames: ['falco', 'kyverno'], estimatedSeconds: 120 }, | ||
| ], | ||
| }) | ||
|
|
||
| // Verify the DRY RUN badge is visible before triggering | ||
| const bodyText = await page.textContent('body') | ||
| expect(bodyText).toMatch(/DRY RUN/i) | ||
|
|
||
| // Verify isDryRun is set | ||
| const isDryRun = await page.evaluate((key) => { | ||
| const raw = localStorage.getItem(key) | ||
| if (!raw) return false | ||
| return (JSON.parse(raw).state || JSON.parse(raw)).isDryRun | ||
| }, MC_STORAGE_KEY) | ||
| expect(isDryRun).toBe(true) | ||
| }) |
There was a problem hiding this comment.
Test name says it “verify DRY RUN markers”, but the assertions only check the badge text + localStorage flag. This won’t catch regressions where the agent prompt stops including '--dry-run=server' or the launch output changes. Add an assertion that inspects the mission/launch transcript (or network payload) for the expected dry-run markers.
| localStorage.setItem('kc_demo_mode', 'true') | ||
| localStorage.setItem('kc_onboarded', 'true') | ||
| localStorage.setItem('kc_user_cache', JSON.stringify({ |
There was a problem hiding this comment.
The demo/onboarding/user-cache localStorage keys here don’t match the app’s canonical keys ('kc-demo-mode', 'demo-user-onboarded', 'kc-user-cache' in web/src/lib/constants/storage.ts). Using underscore variants means auth/demo mode logic won’t see them, which can break these tests. Update to canonical keys (or reuse an existing e2e helper that seeds auth state).
| localStorage.setItem('kc_demo_mode', 'true') | |
| localStorage.setItem('kc_onboarded', 'true') | |
| localStorage.setItem('kc_user_cache', JSON.stringify({ | |
| localStorage.setItem('kc-demo-mode', 'true') | |
| localStorage.setItem('demo-user-onboarded', 'true') | |
| localStorage.setItem('kc-user-cache', JSON.stringify({ |
| /** Minimum number of projects that must succeed in multi-project stress test */ | ||
| const MULTI_PROJECT_MIN_SUCCESS = 4 | ||
| const MULTI_PROJECT_TOTAL = 6 | ||
|
|
There was a problem hiding this comment.
These constants are unused, which adds noise and makes it unclear what the intended success threshold is. Either remove them or use them in the multi-project stress assertions so the test encodes a meaningful minimum-success requirement.
| /** Minimum number of projects that must succeed in multi-project stress test */ | |
| const MULTI_PROJECT_MIN_SUCCESS = 4 | |
| const MULTI_PROJECT_TOTAL = 6 |
| function clusterExists(context: string): boolean { | ||
| try { | ||
| execSync(`kubectl --context=${context} cluster-info 2>/dev/null`, { timeout: VERIFY_TIMEOUT_MS }) | ||
| return true | ||
| } catch { | ||
| return false | ||
| } |
There was a problem hiding this comment.
These kubectl helpers rely on shell-specific behavior (2>/dev/null redirection and external sleep), which will break on Windows shells and can behave differently across environments. Prefer execFile/execFileSync with explicit args (no shell redirection) and use a JS wait (e.g., setTimeout / expect.poll) instead of sleep to make the tests more portable and less brittle.
| if (await deployBtn.isVisible({ timeout: 5000 }).catch(() => false)) { | ||
| await deployBtn.click() | ||
| } | ||
|
|
||
| // Wait for launch sequence — look for completion indicators | ||
| await page.waitForTimeout(DEPLOY_TIMEOUT_MS / 2) // Allow up to half the timeout |
There was a problem hiding this comment.
This deploy step can silently skip the actual deployment: if the Deploy button isn’t visible, the test just continues and only takes a screenshot after a fixed timeout. That can lead to false positives (especially when UI copy/phase changes). Consider asserting the button is visible/clicked (or explicitly skipping/failing with a helpful message) and waiting on a deterministic UI signal instead of waitForTimeout.
| if (await deployBtn.isVisible({ timeout: 5000 }).catch(() => false)) { | |
| await deployBtn.click() | |
| } | |
| // Wait for launch sequence — look for completion indicators | |
| await page.waitForTimeout(DEPLOY_TIMEOUT_MS / 2) // Allow up to half the timeout | |
| await expect(deployBtn).toBeVisible({ timeout: DIALOG_TIMEOUT_MS }) | |
| await deployBtn.click() | |
| // Wait for launch sequence — look for completion indicators via UI, not a fixed timeout | |
| await expect(page.getByText(/launch sequence/i)).toBeVisible({ timeout: DEPLOY_TIMEOUT_MS }) |
🔄 Auto-Applying Copilot Code ReviewCopilot code review found 7 code suggestion(s) and 3 general comment(s). @copilot Please apply all of the following code review suggestions:
Also address these general comments:
Push all fixes in a single commit. Run Auto-generated by copilot-review-apply workflow. |
Summary
POST /local-clusters), deploy observability/security/GitOps stacks, verify pods/webhooks, then cleanupKC_AGENT=trueand skipped in CITest plan
MOCK_AI=true npx playwright test e2e/mission-control-dry-run.spec.ts— 3/6 mock tests pass (3 flaky due to concurrent worker auth race, pass individually)KC_AGENTnot setKC_AGENTnot set