Skip to content

test(e2e): use oidc login flows#83

Merged
rowan-stein merged 3 commits intomainfrom
noa/issue-82
May 1, 2026
Merged

test(e2e): use oidc login flows#83
rowan-stein merged 3 commits intomainfrom
noa/issue-82

Conversation

@casey-brooks
Copy link
Copy Markdown
Contributor

Summary

  • replace mockauth-based seeding with browser OIDC login helpers across console, chat, and tracing suites
  • update user-directory and message deeplink specs to rely on real login flows
  • add Chat → Tracing view-trace coverage for Codex and Claude runs

Testing

  • node ./node_modules/eslint/bin/eslint.js . --config eslint.config.js (playwright-tracing-app)
  • E2E_BASE_URL=https://console.agyn.dev npx playwright test (fails: net::ERR_CONNECTION_REFUSED)
  • E2E_BASE_URL=https://chat.agyn.dev npx playwright test (fails: net::ERR_CONNECTION_REFUSED)
  • E2E_BASE_URL=https://tracing.agyn.dev npx playwright test (fails: net::ERR_CONNECTION_REFUSED)

Refs #82

@casey-brooks
Copy link
Copy Markdown
Contributor Author

Test & Lint Summary

  • node ./node_modules/eslint/bin/eslint.js . --config eslint.config.js (playwright-tracing-app): ✅ no issues
  • E2E_BASE_URL=https://console.agyn.dev npx playwright test --reporter=json > /tmp/playwright-console-report.json: passed 0, failed 32, skipped 2 (net::ERR_CONNECTION_REFUSED)
  • E2E_BASE_URL=https://chat.agyn.dev npx playwright test --reporter=json > /tmp/playwright-chat-report.json: passed 0, failed 21, skipped 0 (net::ERR_CONNECTION_REFUSED)
  • E2E_BASE_URL=https://tracing.agyn.dev npx playwright test --reporter=json > /tmp/playwright-tracing-report.json: passed 0, failed 5, skipped 0 (net::ERR_CONNECTION_REFUSED)

Copy link
Copy Markdown

@noa-lucent noa-lucent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Key blockers to address before merge:

  • clearAuthState() in the console + tracing helpers only clears storage once due to the e2e:oidc-cleared guard, which breaks repeated force: true re-logins (notably user-directory-api.spec.ts).
  • Several .catch(() => null) patterns swallow non-timeout errors and hide real auth/navigation failures.
  • message-deeplink-oidc.spec.ts remains too permissive (allows staying on /message/...); per issue #82 it should require landing on /<orgId>/runs/<runId> and assert timeline/events render.
  • New Chat→Tracing test is tagged @svc_tracing_app but the chat suite selection doesn’t include svc_tracing_app, so tracing-tagged runs will skip this coverage.

I left inline comments with concrete fix suggestions.

Comment thread suites/playwright/test/e2e/sign-in-helper.ts
Comment thread suites/playwright/test/e2e/sign-in-helper.ts
Comment thread suites/playwright-tracing-app/test/e2e/message-deeplink-oidc.spec.ts Outdated
Comment thread suites/playwright-tracing-app/test/e2e/sign-in-helper.ts
Comment thread suites/playwright-chat-app/test/e2e/sign-in-helper.ts
Comment thread suites/playwright-chat-app/test/e2e/chat-trace-link.spec.ts
Comment thread suites/playwright-chat-app/test/e2e/chat-trace-link.spec.ts Outdated
Comment thread suites/playwright-tracing-app/test/e2e/sign-in-helper.ts Outdated
@casey-brooks
Copy link
Copy Markdown
Contributor Author

Test & Lint Summary (review fixes)

  • node ./node_modules/eslint/bin/eslint.js . --config eslint.config.js (playwright-tracing-app): ✅ no issues
  • E2E_BASE_URL=https://console.agyn.dev npx playwright test --reporter=json > /tmp/playwright-console-report.json: passed 0, failed 32, skipped 2 (net::ERR_CONNECTION_REFUSED)
  • E2E_BASE_URL=https://chat.agyn.dev npx playwright test --reporter=json > /tmp/playwright-chat-report.json: passed 0, failed 21, skipped 0 (net::ERR_CONNECTION_REFUSED)
  • E2E_BASE_URL=https://tracing.agyn.dev npx playwright test --reporter=json > /tmp/playwright-tracing-report.json: passed 0, failed 5, skipped 0 (net::ERR_CONNECTION_REFUSED)

noa-lucent
noa-lucent previously approved these changes Apr 29, 2026
Copy link
Copy Markdown

@noa-lucent noa-lucent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review: changes look good.

Verified fixes:

  • clearAuthState() no longer uses a one-shot addInitScript guard; repeated force: true re-logins won’t reuse stale oidc.user:* state.
  • Broad .catch(() => null) patterns replaced with TimeoutError-only handling.
  • message-deeplink-oidc.spec.ts now requires redirect to /<orgId>/runs/<runId> and asserts timeline/events render.
  • Chat suite tag selection now includes svc_tracing_app, so tracing-tagged runs execute the new Chat→Tracing coverage.

@casey-brooks
Copy link
Copy Markdown
Contributor Author

Summary

  • updated the user-directory OIDC helper to create a fresh browser context per login and close it after reading the token
  • switched the user-directory API tests to use the Playwright browser fixture for token fetches, avoiding forced relogins on a shared page

Testing

  • npx tsc -p tsconfig.json
  • E2E_BASE_URL=http://127.0.0.1:3000 AGYN_API_TOKEN=admin npx playwright test test/e2e/user-directory-api.spec.ts --reporter=list (run against a local mock console server)

Lint: not configured for suites/playwright.

Copy link
Copy Markdown

@noa-lucent noa-lucent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the PR fully removes mockauth seeding/control-plane calls and drives real browser OIDC UI flows across console/chat/tracing suites. Message deep-link spec now requires redirect to //runs/ and asserts timeline/events render. Chat suite tag selection includes svc_tracing_app so tracing-tagged runs execute the new Chat→Tracing coverage for codex+claude.\n\nNote: default PR CI (no tags) still won’t execute the chat suite / non-smoke tracing specs, so coverage for the new scenarios depends on running with svc_* tags (as intended).

@rowan-stein rowan-stein merged commit ba69f44 into main May 1, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants