Enable "Smoke test release" workflow #36598

rodelgc · 2023-01-25T11:32:30Z

All Submissions:

Have you followed the WooCommerce Contributing guideline?
Does your code follow the WordPress' coding standards?
Have you checked to ensure there aren't other open Pull Requests for the same update/change?

Changes proposed in this Pull Request:

Closes:

This PR contains the following changes:

Smoke test release workflow now uses the Playwright tests, both API and E2E.
Run the Smoke test release workflow:
- automatically when a new release is published, including pre-releases.
- manually, by running it from the Smoke test release page.

The workflow runs the API & E2E tests in the following environments:

WP & PHP version	Site
WP Latest, PHP 8.0	Release Smoke Test site in Bluehost
WP Latest-1	`wp-env` localhost
WP Latest-2	`wp-env` localhost
PHP 7.4	`wp-env` localhost
PHP 8.1	`wp-env` localhost

Publish Allure report for each test on WooCommerce Test Reports > Smoke Tests on Releases page.
Address several more flakiness discovered when running the E2E tests against the Release Smoke Test Site.
The UPDATE_WC environment variable now accepts the WooCommerce version you want to run the tests against.

For now, this PR does not include the following because the Smoke test release file is getting huge already:

Running of tests with top plugins (WooCommerce Payments, WooCommerce Subscriptions, etc.).
Sending Slack notifications on failed tests.
You would still see duplicated steps throughout the Smoke test release file. They will be made into reusable workflows in the future.

The plugin tests and Slack notifications will be done on a separate PR, probably on a separate workflow file.

How to test the changes in this Pull Request:

Scenario 1: Test using a valid WooCommerce release version as input

Go to Smoke test release.
Trigger a new run using these steps:
1. Click Run workflow.
2. Select this branch (e2e/playwright-release).
3. In the WooCommerce Release Tag field enter a valid WooCommerce release version like 7.4.0-beta.1 or 7.0.0. Do not use 7.2.0-rc.2 because it doesn't have a woocommerce.zip asset. Also, choose a version that's not yet listed in WooCommerce Test Reports > Smoke Tests on Releases.
4. Start the run.
Go to the workflow run you started.
Wait for the Test WooCommerce update job to finish. It runs the update-woocommerce.spec.js E2E test against the Release Smoke test site to see update failures early on.
Go to WooCommerce Test Reports > Smoke Tests on Releases. The WooCommerce version number you entered earlier should appear on the list. If not, wait for 1-2 more minutes to give GitHub Pages more time to upload the new report from Test WooCommerce update job.
Click on the version number to expand it.
In the WP Latest row:
1. Verify the "E2E" column using these steps:
  1. You should see "E2E ✅" which means that update-woocommerce.spec.js passed on the Release smoke test site.
  2. Click on the "E2E" link. You should see the Allure report containing the passing update-woocommerce.spec.js test. It's the only test you'd see for now, since the other tests haven't ran yet.
  3. It's fine if you see can run the database update marked as "skipped" because the WC database version in our Release Smoke Test site is already the latest.
2. Verify the "API" column using these steps:
  1. You should see the "API" text to be grayed out for now because the API tests haven't finished yet. Same with the other "E2E" and "API" tests of other environments in the list.
We will now verify if the previous 2 versions of WordPress were correctly obtained. Go back to the Smoke test release run.
Wait for the Get WP L-1 & L-2 version numbers job to finish, if it hasn't yet.
Click Get WP L-1 & L-2 version numbers to see its steps.
Click the step Get version numbers. It should print the correct L-1 and L-2 versions obtained from the WordPress.org API version check endpoint. Verify that they're correct.
Now, verify if the obtained PHP versions were correct. Click on any of the Test against PHP X.X jobs.
Expand the Verify PHP version step. Verify that the PHP version found in WP Env environment is correct when compared to the Expected PHP version as shown below.
To verify that the tests ran on the WP L-1 and L-2 versions, click on any of the Test against WP Latest-X jobs.
Look for the step Verify environment details and click to expand it. This step basically runs shell commands that print certain environment details like the WordPress version installed.
Verify that the printed WP version is correct.
Still under the Verify environment details step, verify that the installed WooCommerce version is the one that you selected earlier.
Wait for all the jobs to finish.
Go back to WooCommerce Test Reports > Smoke Tests on Releases and expand the WooCommerce version you chose.
You should see the table filled up now with status icons, except for PHP 8.0 because it's already covered by WP Latest which ran against the Release Smoke Test site, which has PHP 8.0. Refer to the "Legend" section to see what each icon means.
In the "WP Latest" row, click on the "E2E" link again. You should now see the Allure report of all tests, including the update-woocommerce.spec.js test you saw earlier.
Click on the E2E and API links of other rows to see if you're taken to the correct Allure report.
In the Allure report, verify the correctness of the following details:

Scenario 2: Test using an invalid WooCommerce release version as input

Go to Smoke test release.
Trigger a new run using these steps:
1. Click Run workflow.
2. Select this branch (e2e/playwright-release).
3. In the WooCommerce Release Tag field enter an invalid value like 7.6.0 or 7.3.0-beta.3.
4. Start the run.
Open the workflow run you just started.
After a few moments, verify that:
1. The "Get WooCommerce release tag" job fails
2. All the other jobs did not execute.
3. Under the "Validate tag" step, the error release not found is printed.

Scenario 3: Test using an valid WooCommerce release version, but does not have a `woocommerce.zip` release asset

Go to Smoke test release.
Trigger a new run using these steps:
1. Click Run workflow.
2. Select this branch (e2e/playwright-release).
3. In the WooCommerce Release Tag field enter a valid release tag in the WooCommerce Core repo, but does not have a woocommerce.zip asset like 7.2.0-rc.2 or wc-beta-tester-2.2.0.
4. Start the run.
Open the workflow run.
After a few moments, verify that:
1. The "Get WooCommerce release tag" job fails.
2. All the other jobs did not execute.
3. The "Verify woocommerce.zip asset" step fails because there's no woocommerce.zip found. Note that the error message may differ.
  
  Error message on wc-beta-tester-2.2.0
  
  Error message on 7.2.0-rc.2

Other information:

Have you added an explanation of what your changes do and why you'd like us to include them?
Have you created a changelog file for each project being changed, ie pnpm --filter=<project> changelog add?

FOR PR REVIEWER ONLY:

I have reviewed that everything is sanitized/escaped appropriately for any SQL or XSS injection possibilities. I made sure Linting is not ignored or disabled.

…ng's

nigeljamesstevenson

Looks good to me Del! Thanks a lot for this 👏🏻

nigeljamesstevenson · 2023-02-17T19:03:39Z

Hi @rodelgc - unfortunately this branch has a conflict. If you resolve it then I will be happy to merge - thanks again!

rodelgc · 2023-02-22T08:35:15Z

Hi @rodelgc - unfortunately this branch has a conflict. If you resolve it then I will be happy to merge - thanks again!

Resolved in latest commit.

nigeljamesstevenson

Looks great! Thanks for the update 👏🏻

codecov · 2023-02-22T11:04:56Z

Codecov Report

Merging #36598 (7227f86) into trunk (f9f6e68) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff            @@
##             trunk   #36598   +/-   ##
========================================
  Coverage     46.7%    46.7%           
  Complexity   17178    17178           
========================================
  Files          429      429           
  Lines        64779    64779           
========================================
  Hits         30240    30240           
  Misses       34539    34539

nigeljamesstevenson

Looks good Del - thanks!

* Add an encoding selector to the product importer * Add changelog file * Don't assume the character encoding parameter is present * Add woocommerce_attributes_saved trigger. Closes #35004. * Fix create wc extension script (#36917) * Run create-extension for create-wc-extension script * Add changelog --------- Co-authored-by: Sam Seay <samueljseay@gmail.com> * Update product editor package (#36830) * Add missing dev packages to product-editor package * Create components folder for organization * Move product field, section and tab slots over to product-editor package * Move use of product slot fills to product-editor package * Sync dependencies * Add changelogs * Update README's and add constant for default values * Update README's in product-editor package * Change SORT_STRING to SORT_NATURAL for the encodings list Co-authored-by: Barry Hughes <3594411+barryhughes@users.noreply.github.com> * Add changelog * Add custom rendering logic to the item label (#36476) * Add type definitions * Add custom rendering logic to the item label * Add stories * Add changelog file * Fix rebase merge issue * Fix up stories after rebase --------- Co-authored-by: Matt Sherman <matt@jam123.com> * Enable "Smoke test release" workflow (#36598) * First pass at updating release test workflow * Add changelog * Set dir env variables * Update to workflow * Fix indent * Fix indent * Clean up indent * Re-order steps * Change order of jobs * Added common php versions * Update pipeline * Update some labels * Simplify for testing * Update paths * Create tmp folder * Fix path * Paths * Try outputting some debugging * Add step ID back * Remove working directory * Another path tweak * Add API release tests * Add k6 tests * Add PHP tests * Launch wp-env during PHP tests * Try default values * Tweak some settings, add WP testing * Tweak some settings * Re-order e2e steps * Update step descriptions * Reorganize tests, add plugin tests * Enable only e2e job * Initial set up to run against release smoke test site * Fix syntax * Temporarily disable update wc spec * Temporarily disable downloading woocommerce zip * Download release zip using tag name * Fix wrong job name * Fix wrong job name * Fix dir * Delete fetch-asset-id.js * REfactor update-woocommerce spec * Add error handling * Download release zip by tag * Refactor update woo spec to download zip by tag * Correct job name * fail test on invalid tag * Enable all e2e tests * Run api tests before e2e tests * Fix job dependency * Add customer credentials to api job * Separate job for WC Update * Combine e2e allure-results, then report * Enable report job * Fix context * Change job and artifact names * Use test s3 path * Minor job name change * Upload artifacts to bucket * Correct s3 path * Add quiet option * Retain video on failures * Finalize s3 path * Try WP latest-1 * Revert to wp latest * Refine search for woocommerce zip asset * Get created-at * Specify repo in gh command * Slugify env description * Trim space * Sync with bucket instead of copy * Remove invalid --recursive flag * Re-add missing step to combine e2e results from update wc test * Ensure artifact upload on test failure * Enable all e2e tests on WP latest * Retain existing data before updating WC * Make test compatible with 'Canceled' and 'Cancelled' * Set env_desc as env var * Re-add deleted file * Fix UPDATE_WC in daily smoke test workflow * Add tracing in global setup * Remove tracing * Temporarily run only basic spec * Job for WP Latest-1 & 2 * Fix "Required input 'created_at' not provided" * Minor rename * Remove install filter * Install deps in get-wp-matrix * Delete get-wp-versions.js * Add get-wp-versions.js to e2e-pw folder * REname file * REfactor * Refactor script for getting WP prev versions * Update job dependencies * Temporarily remove disabled jobs * Allow e2e-wp-latest after api test failure * Update L-1 & L-2 job deps * Fix report-wp-latest * Fix failing api test * Make get-wp-versions quicker * Publish report immediately after test * Test reporting in e2e-update-wc * Fix missing parameter * Fix env_desc, re-enable other jobs * Enable all e2e tests * Minor job name change * Fix flaky test * Add php version testing * stringify php versions * Re-enable all e2e tests * Up timeout to 2min * Remove PHP 8.0 * Add missing conditionals * Fix php version verification script * Fix starting dir * Fix flakiness * Skip e2e if api failed * Verify woocommerce.zip early * Add token * Delete test summary on github for the meantime * Use default playwright config * More meaningful variable names * Update step titles based on review * Use expect.poll() * Minor spacing corrections * Use `stable-check` endpoint, delete unnecessary loop * Update locators to be JN-compatible * Fix erroneous getting of release tag * Fix conflict of "No thanks" button locator with that of WP Mail Logging's * Update github-script action to v6 * Revert to 'Cancelled' * Remove unnecessary step * Provide missing env variables --------- Co-authored-by: Jon Lane <jon.lane@automattic.com> Co-authored-by: Jonathan Lane <lanej0@users.noreply.github.com> * Rename workflow file --------- Co-authored-by: Nestor Soriano <konamiman@konamiman.com> Co-authored-by: helgatheviking <507025+helgatheviking@users.noreply.github.com> Co-authored-by: louwie17 <lourensschep@gmail.com> Co-authored-by: Sam Seay <samueljseay@gmail.com> Co-authored-by: Barry Hughes <3594411+barryhughes@users.noreply.github.com> Co-authored-by: Maikel David Pérez Gómez <mdperez86@gmail.com> Co-authored-by: Matt Sherman <matt@jam123.com> Co-authored-by: Jon Lane <jon.lane@automattic.com> Co-authored-by: Jonathan Lane <lanej0@users.noreply.github.com>

* First pass at updating release test workflow * Add changelog * Set dir env variables * Update to workflow * Fix indent * Fix indent * Clean up indent * Re-order steps * Change order of jobs * Added common php versions * Update pipeline * Update some labels * Simplify for testing * Update paths * Create tmp folder * Fix path * Paths * Try outputting some debugging * Add step ID back * Remove working directory * Another path tweak * Add API release tests * Add k6 tests * Add PHP tests * Launch wp-env during PHP tests * Try default values * Tweak some settings, add WP testing * Tweak some settings * Re-order e2e steps * Update step descriptions * Reorganize tests, add plugin tests * Enable only e2e job * Initial set up to run against release smoke test site * Fix syntax * Temporarily disable update wc spec * Temporarily disable downloading woocommerce zip * Download release zip using tag name * Fix wrong job name * Fix wrong job name * Fix dir * Delete fetch-asset-id.js * REfactor update-woocommerce spec * Add error handling * Download release zip by tag * Refactor update woo spec to download zip by tag * Correct job name * fail test on invalid tag * Enable all e2e tests * Run api tests before e2e tests * Fix job dependency * Add customer credentials to api job * Separate job for WC Update * Combine e2e allure-results, then report * Enable report job * Fix context * Change job and artifact names * Use test s3 path * Minor job name change * Upload artifacts to bucket * Correct s3 path * Add quiet option * Retain video on failures * Finalize s3 path * Try WP latest-1 * Revert to wp latest * Refine search for woocommerce zip asset * Get created-at * Specify repo in gh command * Slugify env description * Trim space * Sync with bucket instead of copy * Remove invalid --recursive flag * Re-add missing step to combine e2e results from update wc test * Ensure artifact upload on test failure * Enable all e2e tests on WP latest * Retain existing data before updating WC * Make test compatible with 'Canceled' and 'Cancelled' * Set env_desc as env var * Re-add deleted file * Fix UPDATE_WC in daily smoke test workflow * Add tracing in global setup * Remove tracing * Temporarily run only basic spec * Job for WP Latest-1 & 2 * Fix "Required input 'created_at' not provided" * Minor rename * Remove install filter * Install deps in get-wp-matrix * Delete get-wp-versions.js * Add get-wp-versions.js to e2e-pw folder * REname file * REfactor * Refactor script for getting WP prev versions * Update job dependencies * Temporarily remove disabled jobs * Allow e2e-wp-latest after api test failure * Update L-1 & L-2 job deps * Fix report-wp-latest * Fix failing api test * Make get-wp-versions quicker * Publish report immediately after test * Test reporting in e2e-update-wc * Fix missing parameter * Fix env_desc, re-enable other jobs * Enable all e2e tests * Minor job name change * Fix flaky test * Add php version testing * stringify php versions * Re-enable all e2e tests * Up timeout to 2min * Remove PHP 8.0 * Add missing conditionals * Fix php version verification script * Fix starting dir * Fix flakiness * Skip e2e if api failed * Verify woocommerce.zip early * Add token * Delete test summary on github for the meantime * Use default playwright config * More meaningful variable names * Update step titles based on review * Use expect.poll() * Minor spacing corrections * Use `stable-check` endpoint, delete unnecessary loop * Update locators to be JN-compatible * Fix erroneous getting of release tag * Fix conflict of "No thanks" button locator with that of WP Mail Logging's * Update github-script action to v6 * Revert to 'Cancelled' * Remove unnecessary step * Provide missing env variables --------- Co-authored-by: Jon Lane <jon.lane@automattic.com> Co-authored-by: Jonathan Lane <lanej0@users.noreply.github.com>

Jon Lane added 30 commits October 27, 2022 14:10

First pass at updating release test workflow

d719823

Add changelog

6ebfab8

Merge branch 'trunk' into e2e/update-release-e2e-testing

3cf065e

Set dir env variables

08a4743

Merge branch 'trunk' into e2e/update-release-e2e-testing

92e2e1b

Update to workflow

cee2065

Fix indent

662a540

Fix indent

2b5fb57

Clean up indent

b67f936

Re-order steps

4d6a330

Change order of jobs

47e21db

Added common php versions

a4fc52d

Update pipeline

8ec5474

Update some labels

0d764e6

Simplify for testing

8bf1940

Update paths

22f0d2a

Create tmp folder

efbdb27

Fix path

3cf5de4

Paths

65ed740

Try outputting some debugging

2ebf470

Add step ID back

83b0fd7

Remove working directory

edb08f8

Another path tweak

5ffbb01

Add API release tests

01cd444

Add k6 tests

b78d5cd

Add PHP tests

678fd11

Launch wp-env during PHP tests

3adc597

Try default values

0f33673

Tweak some settings, add WP testing

c10ccaa

Tweak some settings

a327e54

rodelgc added 7 commits February 1, 2023 13:54

Minor spacing corrections

92fad6f

Use stable-check endpoint, delete unnecessary loop

94552a6

Update locators to be JN-compatible

2c1a510

Fix erroneous getting of release tag

d3983c5

Fix conflict of "No thanks" button locator with that of WP Mail Loggi…

a7dded3

…ng's

Update github-script action to v6

09c0bb8

Revert to 'Cancelled'

bf4fa5a

nigeljamesstevenson self-requested a review February 15, 2023 11:11

nigeljamesstevenson previously approved these changes Feb 17, 2023

View reviewed changes

rodelgc added 2 commits February 22, 2023 16:30

Resolve conflicts with trunk

777bb53

Remove unnecessary step

146d986

rodelgc dismissed nigeljamesstevenson’s stale review via 146d986 February 22, 2023 08:33

rodelgc requested review from alopezari and nigeljamesstevenson February 22, 2023 08:34

nigeljamesstevenson previously approved these changes Feb 22, 2023

View reviewed changes

Provide missing env variables

5f70773

rodelgc dismissed nigeljamesstevenson’s stale review via 5f70773 February 22, 2023 10:45

Merge branch 'trunk' into e2e/playwright-release

bfdbe88

rodelgc requested a review from nigeljamesstevenson February 23, 2023 03:43

Merge branch 'trunk' into e2e/playwright-release

b666211

nigeljamesstevenson approved these changes Feb 23, 2023

View reviewed changes

Merge branch 'trunk' into e2e/playwright-release

7227f86

rodelgc merged commit ea1925c into trunk Feb 27, 2023

rodelgc deleted the e2e/playwright-release branch February 27, 2023 04:11

github-actions bot added this to the 7.6.0 milestone Feb 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable "Smoke test release" workflow #36598

Enable "Smoke test release" workflow #36598

rodelgc commented Jan 25, 2023 •

edited

nigeljamesstevenson left a comment

nigeljamesstevenson commented Feb 17, 2023

rodelgc commented Feb 22, 2023

nigeljamesstevenson left a comment

codecov bot commented Feb 22, 2023 •

edited

nigeljamesstevenson left a comment

Enable "Smoke test release" workflow #36598

Enable "Smoke test release" workflow #36598

Conversation

rodelgc commented Jan 25, 2023 • edited

All Submissions:

Changes proposed in this Pull Request:

How to test the changes in this Pull Request:

Scenario 1: Test using a valid WooCommerce release version as input

Scenario 2: Test using an invalid WooCommerce release version as input

Scenario 3: Test using an valid WooCommerce release version, but does not have a woocommerce.zip release asset

Other information:

FOR PR REVIEWER ONLY:

nigeljamesstevenson left a comment

Choose a reason for hiding this comment

nigeljamesstevenson commented Feb 17, 2023

rodelgc commented Feb 22, 2023

nigeljamesstevenson left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 22, 2023 • edited

Codecov Report

nigeljamesstevenson left a comment

Choose a reason for hiding this comment

rodelgc commented Jan 25, 2023 •

edited

Scenario 3: Test using an valid WooCommerce release version, but does not have a `woocommerce.zip` release asset

codecov bot commented Feb 22, 2023 •

edited