Allow running of "Smoke test release" on draft releases #36997

rodelgc · 2023-02-28T12:13:48Z

All Submissions:

Have you followed the WooCommerce Contributing guideline?
Does your code follow the WordPress' coding standards?

Changes proposed in this Pull Request:

This PR would allow the Smoke test release workflow to be run on a draft release. Also, it contains a few several fixes in the workflow file.

The main reason why the Smoke test release workflow previously couldn't run on draft releases was because it's using the default github.token when retrieving the list of releases. Since this token has limited permissions, it can only access releases/pre-releases that are already published.

To fix this, the Smoke test release workflow now uses the same E2E_GH_TOKEN used in other workflows. With this token, draft releases can already be accessed.

How to test the changes in this Pull Request:

Have you followed the Writing high-quality testing instructions guide?

Create a draft release. Make sure it has:
1. A woocommerce.zip asset, and
2. A temporary tag for testing purposes, like test-pr-36997.
Go to the Smoke test release page.
In the "Use workflow from" dropdown, select this branch e2e/release-include-drafts.
In the "WooCommerce Release Tag" box, enter the tag you created in step 1.
Click "Run workflow" to manually start it.
Wait for it to finish. All jobs should be executed like so:
Go to WooCommerce Test Reports > Smoke Tests on Releases page.
You should see the tag listed in the page, and that the API and E2E test results are displayed, regardless of whether they passed or failed.

If you're taken to a 404 page, chances are the reports are still being uploaded. Wait for a few more minutes, then refresh the page.
Delete the draft release after testing.

Other information:

Have you added an explanation of what your changes do and why you'd like us to include them?
Have you created a changelog file for each project being changed, ie pnpm --filter=<project> changelog add?
Have you included testing instructions?

FOR PR REVIEWER ONLY:

I have reviewed that everything is sanitized/escaped appropriately for any SQL or XSS injection possibilities. I made sure Linting is not ignored or disabled.

github-actions · 2023-02-28T12:37:08Z

Test Results Summary

Commit SHA: 718ef83

Test 🧪	Passed ✅	Failed 🚨	Broken 🚧	Skipped ⏭️	Unknown ❔	Total 📊	Duration ⏱️
API Tests	259	0	0	2	0	261	0m 57s
E2E Tests	189	0	0	6	0	195	16m 20s

To view the full API test report, click here.
To view the full E2E test report, click here.
To view all test reports, visit the WooCommerce Test Reports Dashboard.

codecov · 2023-02-28T12:42:53Z

Codecov Report

Merging #36997 (718ef83) into trunk (7c76118) will decrease coverage by 0.0%.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##             trunk   #36997     +/-   ##
==========================================
- Coverage     46.7%    46.7%   -0.0%     
  Complexity   17188    17188             
==========================================
  Files          429      429             
  Lines        64820    64821      +1     
==========================================
- Hits         30253    30251      -2     
- Misses       34567    34570      +3

Impacted Files	Coverage Δ
plugins/woocommerce/includes/class-wc-tax.php	`78.6% <0.0%> (-0.5%)`	⬇️
...ugins/woocommerce/includes/class-wc-post-types.php	`2.9% <0.0%> (-<0.1%)`	⬇️

rodelgc · 2023-03-02T05:29:52Z

.github/workflows/smoke-test-release.yml

-        types: [published]
+        types: [released, prereleased]


[Fix]

I noticed in the 7.4.1 and 7.5.0-beta.2 releases that this workflow wasn't triggered after they were made public. I figured it's because these releases may have already been published, and were just updated to be a pre-release or a release later on. Therefore, the published release type wasn't triggered because they were already in a published state to begin with.

Setting the release type to released and prereleased I think would be more appropriate. But, is it, @woocommerce/atlas? Thanks 😃

I figured it's because these releases may have already been published, and were just updated to be a pre-release or a release later on.

I'm not sure that this is the case, as I always make sure to select one of those two before publishing a release... but if released and prereleased trigger this workflow as expected, I think this should be okay.

I saw this in the release event documentation:

Note: The prereleased type will not trigger for pre-releases published from draft releases, but the published type will trigger. If you want a workflow to run when stable and pre-releases publish, subscribe to published instead of released and prereleased.

Might this conflict with this change? 🤔

rodelgc · 2023-03-02T05:38:31Z

.github/workflows/smoke-test-release.yml

-    group: ${{ github.workflow }}-${{ github.ref }}
+    group: ${{ github.workflow }}-${{ github.event.release.tag_name || inputs.tag }}


[Fix]

I noticed that the concurrency group was erroneous because the main intention of this workflow is to group the release smoke tests based on the release version (tag) they're testing, not based on github.ref. Leaving the concurrency group to use github.ref will cause a smoke test for release 7.5.0, for example, to unexpectedly cancel the smoke test of another release version.

I fixed it here so that the release smoke tests are grouped by release tag, (whether this tag came from the release or workflow_dispatch event) thereby preventing concurrent release smoke tests on different tags/versions from cancelling each other.

Good catch! 🙌

rodelgc · 2023-03-02T05:43:48Z

.github/workflows/smoke-test-release.yml

            created: ${{ steps.created-at.outputs.created }}
        steps:
            - name: Validate tag
              if: ${{ github.event_name == 'workflow_dispatch' }}
              env:
-                  GH_TOKEN: ${{ github.token }}
-              run: gh release view "${{ github.event.inputs.tag }}" --repo=woocommerce/woocommerce
+                  GH_TOKEN: ${{ secrets.E2E_GH_TOKEN }}


As noted in the PR description, I replaced the default github.token with E2E_GH_TOKEN. You would see similar diffs like this throughout this PR.

rodelgc · 2023-03-02T05:50:05Z

.github/workflows/smoke-test-release.yml

-            - name: Get tag
-              uses: actions/github-script@v6
-              id: tag
-              with:
-                  result-encoding: string
-                  script: |
-                      console.log( "${{ github.event_name }}" );
-                      return "${{ github.event.release.tag_name }}" || "${{ github.event.inputs.tag }}"
+            - name: Get tag from triggered event
+              id: get-tag
+              env:
+                  RELEASE_TAG: ${{ github.event.release.tag_name || inputs.tag }}
+              run: |
+                  echo "Triggered event: ${{ github.event_name }}"
+                  echo "Tag from event: $RELEASE_TAG"
+                  echo "tag=$RELEASE_TAG" >> $GITHUB_OUTPUT


[Fix]

The return line was erroneous because the || should be inside ${{ }}. Fixed it here by doing just that. Also removed the use of the actions/github-script action in order to simplify the logic of this step and make it more readable (I hope 🤞) .

rodelgc · 2023-03-02T05:50:44Z

.github/workflows/smoke-test-release.yml

-              uses: aws-actions/configure-aws-credentials@v1
+              uses: aws-actions/configure-aws-credentials@v1-node16


[Fix]

This fixes the node v12 deprecation messages.

rodelgc · 2023-03-02T06:00:00Z

plugins/woocommerce/tests/e2e-pw/tests/smoke-tests/update-woocommerce.spec.js

+	if ( GITHUB_TOKEN ) {
+		requestConfig.headers.Authorization = `Bearer ${ GITHUB_TOKEN }`;
+	}


The changes in this update-woocommerce.spec.js file centers around this particular line. This makes sure that the supplied token will be used when downloading the woocommerce.zip file from a draft release.

jonathansadowski

👍 nice job. I verified that this selected the correct asset with a draft release that I created. The smoke tests themselves failed for various reasons unrelated to this PR (such as my tag name, 7.4.1-test not matching the version from the asset I created, 7.4.1), but I believe that's expected given the package that I used.

jonathansadowski · 2023-03-02T19:12:27Z

.github/workflows/smoke-test-release.yml

-        types: [published]
+        types: [released, prereleased]


I figured it's because these releases may have already been published, and were just updated to be a pre-release or a release later on.

I'm not sure that this is the case, as I always make sure to select one of those two before publishing a release... but if released and prereleased trigger this workflow as expected, I think this should be okay.

nigeljamesstevenson

Based on Jonathan's validation, this is looking good!

alopezari

Looks good to me @rodelgc!

Based on this change, it would be great if we could confirm that smoke tests also run on the following scenarios:

Release is directly published, without any draft previous state.
Draft release is published, in other words, a draft release is created first (so smoke tests would be run considering the new changes from this PR), but once we remove the draft state and it becomes an actual release, smoke tests should be run again if I'm not wrong).

This way we would cover potential regression issues related to the event triggers change.

alopezari · 2023-03-03T10:23:06Z

.github/workflows/smoke-test-release.yml

-        types: [published]
+        types: [released, prereleased]


I saw this in the release event documentation:

Note: The prereleased type will not trigger for pre-releases published from draft releases, but the published type will trigger. If you want a workflow to run when stable and pre-releases publish, subscribe to published instead of released and prereleased.

Might this conflict with this change? 🤔

alopezari · 2023-03-03T10:32:26Z

.github/workflows/smoke-test-release.yml

-    group: ${{ github.workflow }}-${{ github.ref }}
+    group: ${{ github.workflow }}-${{ github.event.release.tag_name || inputs.tag }}


Good catch! 🙌

rodelgc · 2023-03-06T03:08:30Z

I saw this in the release event documentation:

Note: The prereleased type will not trigger for pre-releases published from draft releases, but the published type will trigger. If you want a workflow to run when stable and pre-releases publish, subscribe to published instead of released and prereleased.

Might this conflict with this change? 🤔

It actually did @alopezari! Thanks for bringing this up. When I tested it in a private repo, the workflow indeed wasn't triggered when I saved the release as a draft first, then published it as a pre-release.

To fix this, I pushed 718ef83 so that the release types are now [released, prereleased, published]. When I tested these 3 release types on a private repo, the workflow was triggered as expected on these actions:

Save a draft first, then published as a Prerelease
Save a draft first, then published without checking any of the checkboxes in the GitHub UI
Save a draft first, then published as a Latest release
Published as a Prerelease directly, didn't create a draft
Published directly without checking any checkboxes, didn't create a draft
Published directly as a Latest release, didn't create a draft
Start with an already published release, then update it to a Prerelease
Start with an already published pre-release, then update it to a Latest release
Start with an already published pre-release, then update it to a release (without checking Latest)

Workflow wasn't triggered on the scenario where an already published non-latest release was changed to Latest, and vice-versa, and I think this is fine because there's really no need to re-run the tests when this happens.

Ready again for another review.

alopezari

Save a draft first, then published as a Prerelease
Save a draft first, then published without checking any of the checkboxes in the GitHub UI
Save a draft first, then published as a Latest release
Published as a Prerelease directly, didn't create a draft
Published directly without checking any checkboxes, didn't create a draft
Published directly as a Latest release, didn't create a draft
Start with an already published release, then update it to a Prerelease
Start with an already published pre-release, then update it to a Latest release
Start with an already published pre-release, then update it to a release (without checking Latest)

Thanks for such extensive testing of this change @rodelgc!

I'm approving this PR 👍

alopezari · 2023-03-06T11:17:58Z

If @jonathansadowski & @nigeljamesstevenson are happy with the latest change, I'm happy to merge this 👍

nigeljamesstevenson · 2023-03-07T10:40:55Z

@alopezari - Great spot on noticing:

The prereleased type will not trigger for pre-releases published from draft releases

and amazing testing all the combinations @rodelgc ! - This is looking great 😊

alopezari · 2023-03-08T09:23:35Z

Awesome, thanks @nigeljamesstevenson! I'll merge this 👍

rodelgc added 6 commits February 28, 2023 17:17

Conditionally use authorization token when getting WC Zip download URL.

8fef083

Exponse error message

f3a96be

Handle undefined authorization token more gracefully

3f5a8e4

Specify GH token in workflow

2d65274

Add changelog

3f26383

Use E2E_GH_TOKEN instead of default github.token

1e811c1

github-actions bot added the plugin: woocommerce Issues related to the WooCommerce Core plugin. label Feb 28, 2023

rodelgc added 11 commits February 28, 2023 21:09

Provide GitHub token to other jobs

e0369af

Fix getting of tag name from triggered event

a6c7f44

Use "inputs" context instead

7916b15

Add release version to workflow name

6d905d5

Correct concurrency group

d1da273

Merge branch 'trunk' into e2e/release-include-drafts

1f2d46a

Fix workflow name

02d97b2

Add --archive option

8ee473f

Fix output setting

d5d2ffb

Fix script for verifying asset

a91073d

Remove unnecessary 'uses' line

3d4cd21

rodelgc changed the title ~~E2e/release include drafts~~ Run "Smoke test release" on draft releases Mar 2, 2023

Remove node 12 deprecation messages

3761267

rodelgc changed the title ~~Run "Smoke test release" on draft releases~~ Allow running of "Smoke test release" on draft releases Mar 2, 2023

rodelgc commented Mar 2, 2023

View reviewed changes

rodelgc marked this pull request as ready for review March 2, 2023 06:56

rodelgc requested review from a team and jonathansadowski and removed request for a team March 2, 2023 06:56

rodelgc added the focus: smoke tests These Issues/PRs relate to smoke testing. label Mar 2, 2023

jonathansadowski previously approved these changes Mar 2, 2023

View reviewed changes

nigeljamesstevenson previously approved these changes Mar 3, 2023

View reviewed changes

alopezari reviewed Mar 3, 2023

View reviewed changes

Re-add "published" release type

718ef83

rodelgc dismissed stale reviews from nigeljamesstevenson and jonathansadowski via 718ef83 March 6, 2023 02:57

alopezari approved these changes Mar 6, 2023

View reviewed changes

alopezari merged commit b60cc12 into trunk Mar 8, 2023
19 of 20 checks passed

alopezari deleted the e2e/release-include-drafts branch March 8, 2023 09:23

github-actions bot added this to the 7.6.0 milestone Mar 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow running of "Smoke test release" on draft releases #36997

Allow running of "Smoke test release" on draft releases #36997

rodelgc commented Feb 28, 2023 •

edited

github-actions bot commented Feb 28, 2023 •

edited

codecov bot commented Feb 28, 2023 •

edited

rodelgc Mar 2, 2023 •

edited

jonathansadowski Mar 2, 2023

alopezari Mar 3, 2023

rodelgc Mar 2, 2023 •

edited

alopezari Mar 3, 2023

rodelgc Mar 2, 2023 •

edited

rodelgc Mar 2, 2023 •

edited

rodelgc Mar 2, 2023

rodelgc Mar 2, 2023 •

edited

jonathansadowski left a comment

jonathansadowski Mar 2, 2023

nigeljamesstevenson left a comment

alopezari left a comment •

edited

alopezari Mar 3, 2023

alopezari Mar 3, 2023

rodelgc commented Mar 6, 2023 •

edited

alopezari left a comment

alopezari commented Mar 6, 2023

nigeljamesstevenson commented Mar 7, 2023

alopezari commented Mar 8, 2023

		group: ${{ github.workflow }}-${{ github.ref }}
		group: ${{ github.workflow }}-${{ github.event.release.tag_name \|\| inputs.tag }}

		uses: aws-actions/configure-aws-credentials@v1
		uses: aws-actions/configure-aws-credentials@v1-node16

Allow running of "Smoke test release" on draft releases #36997

Allow running of "Smoke test release" on draft releases #36997

Conversation

rodelgc commented Feb 28, 2023 • edited

All Submissions:

Changes proposed in this Pull Request:

How to test the changes in this Pull Request:

Other information:

FOR PR REVIEWER ONLY:

github-actions bot commented Feb 28, 2023 • edited

Test Results Summary

codecov bot commented Feb 28, 2023 • edited

Codecov Report

rodelgc Mar 2, 2023 • edited

Choose a reason for hiding this comment

jonathansadowski Mar 2, 2023

Choose a reason for hiding this comment

alopezari Mar 3, 2023

Choose a reason for hiding this comment

rodelgc Mar 2, 2023 • edited

Choose a reason for hiding this comment

alopezari Mar 3, 2023

Choose a reason for hiding this comment

rodelgc Mar 2, 2023 • edited

Choose a reason for hiding this comment

rodelgc Mar 2, 2023 • edited

Choose a reason for hiding this comment

rodelgc Mar 2, 2023

Choose a reason for hiding this comment

rodelgc Mar 2, 2023 • edited

Choose a reason for hiding this comment

jonathansadowski left a comment

Choose a reason for hiding this comment

jonathansadowski Mar 2, 2023

Choose a reason for hiding this comment

nigeljamesstevenson left a comment

Choose a reason for hiding this comment

alopezari left a comment • edited

Choose a reason for hiding this comment

alopezari Mar 3, 2023

Choose a reason for hiding this comment

alopezari Mar 3, 2023

Choose a reason for hiding this comment

rodelgc commented Mar 6, 2023 • edited

alopezari left a comment

Choose a reason for hiding this comment

alopezari commented Mar 6, 2023

nigeljamesstevenson commented Mar 7, 2023

alopezari commented Mar 8, 2023

rodelgc commented Feb 28, 2023 •

edited

github-actions bot commented Feb 28, 2023 •

edited

codecov bot commented Feb 28, 2023 •

edited

rodelgc Mar 2, 2023 •

edited

rodelgc Mar 2, 2023 •

edited

rodelgc Mar 2, 2023 •

edited

rodelgc Mar 2, 2023 •

edited

rodelgc Mar 2, 2023 •

edited

alopezari left a comment •

edited

rodelgc commented Mar 6, 2023 •

edited