-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
E2E tests are failing locally and in CI #37
Comments
I updated #38 to upload the Playwright traces. They can be downloaded individually and inspected with
Here's what the one "create a new site" shows: Two things to notice:
|
I have a crude fix for the onboarding issue at |
* WIP * Makes onboarding work! * Make create site work * Make delete site work * All E2E tests work 🎉 * Fix E2E tearDown cleanup condition * Cleanup * Add a check for consistency * Only run onboarding when needed * Improve debugging * Remove useless comment * Address lint issues * Run lint, unit, and E2E tests on Buildkite (issue with caching) * Remove E2E step – It's failing and don't want to slow the unit tests * Add step to run E2E tests in Buildkite Currently failing, see #37 * Add build matrix with Windows and Mac queues for E2E tests * Call E2E with `bash` explicitly * Add OS name in E2E tests label * Add traces to E2E runs * Fix E2E matrix combinations * Fix lint issues * Make paths system agnostic * Remove no longer valid fixme comment * Clarify comment * Use latest CI plugin version * Use new variables in pipeline * Fix executable path of Windows build in E2E tests * Use `throw new Error` instead of undefined `fail` to fail E2E teardown * Run E2E in Buildkite with `DEBUG=pw:browser` * Remove a trailing whitespace * Test - Run unit tests on Windows, too * Remove Windows test steps in Buildkite Deferring enabling them till we'll have found a solution for Windows-specific issues. In the meantime, I aim to run E2E macOS tests on Buildkite instead of GitHub Actions in all PRs. * Remove macOS as a target for E2E tests in GitHub Actions * Extract CI toolkit version in dedicated var This should make it easier to locate and update the version if needed * Use shared var to define NVM CI plugin * Remove `agents: queue: mac` from the pipeline root We have a variety of steps and platforms in the pipeline now. It no longer makes sense to default to macOS. --------- Co-authored-by: Wojtek Naruniec <wojtek@naruniec.me> Co-authored-by: Carlos Garcia <fluiddot@gmail.com>
I'll take a look at the failures on Windows (#76). |
Quick summary
The E2E tests are currently failing, both locally and in CI.
I think the reason this hasn't been noticed in PRs is that the workflow step is configured with
continue-on-error: true
.Steps to reproduce
Locally, run
npm install && npm run package && npm run e2e
.On CI, visit a recent PR and look at its actions. For example, #56: https://github.com/Automattic/studio/actions/runs/8830011430/job/24242246592?pr=27
What you expected to happen
E2E tests should be able to create tmp sites and move forward with the testing.
What actually happened
Locally, you can see the app being stuck on the "Add your first site" screen:
I assume the same happens in CI, based on the way the tests fail, but I have no way to verify it.
Impact
One
Available workarounds?
There is no user impact
Platform
No response
Logs or notes
Here's my local test log
The text was updated successfully, but these errors were encountered: