Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

E2E Tests: In CI, also run in Firefox #17893

Closed
wants to merge 2 commits into from
Closed

E2E Tests: In CI, also run in Firefox #17893

wants to merge 2 commits into from

Conversation

ockham
Copy link
Contributor

@ockham ockham commented Nov 24, 2020 •
edited

Fixes #17874

Changes proposed in this Pull Request:

WIP. First stab at running e2e tests in Firefox, when run in CI.

Jetpack product discussion

N/A

Does this pull request change what data or activity we track or use?

No.

Testing instructions:

See if the e2e tests GH Action passes 馃

To run locally (with tests visible):

PUPPETEER_HEADLESS=false PUPPETEER_PRODUCT=firefox yarn test-e2e

Proposed changelog entry for your changes:

E2E Tests: Also run in Firefox, when invoked from CI.

TODO

  • Do the same for the gutenberg plugin e2e tests (further down in that file).
  • Once that's done: Revert 2260dfc and 417ed45 and (temporarily) push to this PR to verify that without those fixes, e2e tests fail on Firefox for Gutenberg 9.4.1.
  • Can we share setup parts across jobs in that file? Or make it a proper matrix ([ 'chrome', 'firefox' ])? See e.g. here for inspiration.
  • Add instructions to README on how to run this locally.

cc/ @brbrr @bsessions85

@ockham ockham self-assigned this Nov 24, 2020
@jetpackbot
Copy link

jetpackbot commented Dec 2, 2020 •
edited

Warnings
鈿狅笍

The PR is missing at least one [Status] label. Suggestions: [Status] In Progress, [Status] Needs Review

This is an automated check which relies on PULL_REQUEST_TEMPLATE. We encourage you to follow that template as it helps Jetpack maintainers do their job. If you think 'Testing instructions' or 'Proposed changelog entry' are not needed for your PR - please explain why you think so. Thanks for cooperation 馃

Generated by 馃毇 dangerJS against 89109a3

@ockham ockham mentioned this pull request Dec 2, 2020
Merged
@ockham
Copy link
Contributor Author

ockham commented Dec 2, 2020

Let's wait for #17950, which will allow us to have a matrix for both plugin/no plugin, and Chrome/Firefox.

ockham added a commit that referenced this pull request Dec 14, 2020
@ockham @anomiex
E2E Tests: Use matrix to run tests with and without GB plugin (#17950)
0ad576b 
We currently run our e2e tests in two different ways:
- Once for a snapshot of the Jetpack plugin per the current PR.
- Once with the latest Gutenberg plugin installed.

The latter is supposed to help us uncover issues with our own GB blocks that depend on GB, in case the new GB plugin version makes changes that can cause breakage.

There might be opportunity to extend this further, such as also running them with the latest Gutenberg plugin _Release Candidate_ (rather than stable release) installed; or in different browsers (see #17893 for an ongoing experiment).

Anyway, the two ways we're currently running the e2e tests are implemented separately, even though they share most of the logic.

This PR thus attempts to merge them into one single flow, with a [build matrix](https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/managing-complex-workflows#using-a-build-matrix) to distinguish the different bits. This should make the GH action more maintainable and extensible.

Co-authored-by: Brad Jorsch <anomiex@users.noreply.github.com>
@ockham
Copy link
Contributor Author

ockham commented Dec 14, 2020

Let's wait for #17950, which will allow us to have a matrix for both plugin/no plugin, and Chrome/Firefox.

This has been merged. I've accordingly rebased.

@ockham ockham mentioned this pull request Jan 26, 2021
Closed
@brbrr brbrr added this to To do in E2E tests Feb 2, 2021
@brbrr brbrr moved this from To do to In progress in E2E tests Feb 2, 2021
@jeherve
Copy link
Member

jeherve commented Mar 2, 2022

I'll close this PR for now because of the lack of activity on this. We can always reopen in the future if needed, but it will need a rebase, so it may be easier to start a new PR at this point.

@jeherve jeherve closed this Mar 2, 2022
E2E tests automation moved this from In progress to Done Mar 2, 2022
@ockham ockham deleted the add/e2e-tests-firefox-ci branch May 6, 2022 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@ockham ockham

Labels
E2E Tests [Type] Infrastructure
Projects
E2E tests
Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

e2e Tests: Run against Firefox
3 participants
@ockham @jetpackbot @jeherve