Update plugin(s) build setup for distribution #1033

thelovekesh · 2024-03-06T16:50:26Z

Summary

Fixes task 3 of #1012

As mentioned by @westonruter in #893 (comment), we are now moving towards use cases where we can have to build files that are not in the git tree and hence will be ignored by git archive.

Also as outlined in #1024 (comment), in the new mono repo setup we will require a build setup because the repo level .gitattributes or .distignore won't work just to ignore files from standalone distributed plugins.

This PR proposes a simple Webpack-based plugin bundling solution that solves both problems.

Relevant technical choices

Switch to Webpack for handling plugin(s) building tasks with copy-webpack-plugin Webpack plugin.
Remove dependency from custom-written CLI scripts to:
- Get the plugin version from the readme.txt file. This task is now handled by Webpack which creates a manifest.json with plugin versions.
- Get the plugin path. This is not required as we are completely removing the modules.
- Build the standalone plugins. This task is now handled by Webpack.
Add plugin(s) deployment status on the repo.

Screenshot

Add a workflow step to add the plugin(s) built to release assets with their checksum to ensure assets' integrity.

Screenshot

Checklist

PR has either [Focus] or Infrastructure label.
PR has a [Type] label.
PR has a milestone or the no milestone label.

github-actions · 2024-03-06T16:50:49Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: thelovekesh <thelovekesh@git.wordpress.org>
Co-authored-by: felixarntz <flixos90@git.wordpress.org>
Co-authored-by: mukeshpanchal27 <mukesh27@git.wordpress.org>
Co-authored-by: swissspidy <swissspidy@git.wordpress.org>
Co-authored-by: westonruter <westonruter@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

.github/workflows/deploy-standalone-plugins.yml

felixarntz

@thelovekesh For building the standalone plugins, this looks solid so far.

However, I don't think it makes sense to introduce a build process for Performance Lab, this is a solution in search of a problem IMO. And I don't see a good reason to use .distignore over .gitattributes, please clarify why you're proposing this change.

.github/workflows/deploy-dotorg.yml

.distignore

package.json

.github/workflows/deploy-standalone-plugins.yml

thelovekesh · 2024-03-08T21:22:50Z

package.json

  },
  "scripts": {
    "changelog": "./bin/plugin/cli.js changelog",
    "since": "./bin/plugin/cli.js since",
    "readme": "./bin/plugin/cli.js readme",
    "translations": "./bin/plugin/cli.js translations",
-    "build-plugins": "./bin/plugin/cli.js build-plugins",
+    "build-plugins": "npm-run-all 'build:plugin:!(performance-lab)'",
+    "build:plugin:performance-lab": "rm -rf build/performance-lab && mkdir -p build/performance-lab && git archive HEAD | tar -x -C build/performance-lab",


@felixarntz Added this command, in case we ever need to confirm what's being added in the build by deploy action.

@felixarntz I believe that we should use the above command to verify distribution files(maybe just before a release?). I can see that the build-cs directory is being committed in the svn at https://plugins.trac.wordpress.org/browser/performance-lab/trunk for the most current release 😞.

Also, it would be good if we log an issue to remove such redundant files/dirs from all our plugins in the next release.

felixarntz

@thelovekesh This looks quite good to me now in terms of the overall approach. A few small comments/questions below.

I'm not very familiar with GitHub workflows, so it would be great to get @swissspidy's feedback on those changes in particular.

.github/workflows/deploy-standalone-plugins.yml

swissspidy · 2024-03-10T13:04:55Z

.github/workflows/deploy-standalone-plugins.yml

+      - name: Generate checksum
+        working-directory: ./build/dist
+        run: |
+          mkdir -p $RUNNER_TEMP/plugin-checksums
+          find . -type f -print0 | sort -z | xargs -r0 shasum -a 256 -b | sed 's# \*\./# *#' > $RUNNER_TEMP/plugin-checksums/checksums.txt
+          shasum -a 256 -U -c $RUNNER_TEMP/plugin-checksums/checksums.txt
+          cat $RUNNER_TEMP/plugin-checksums/checksums.txt | while read sum file; do echo "$sum $file" > ${file#\*}.sha256; done


Do we really need to create checksums? A bit overkill IMO. I doubt anyone would ever look at these.

If someone wants checksum verification, WordPress.org already provides that information anyway through a nice API.

It's not a "must use," but providing a means to verify the integrity of your distributed assets is never bad.

.github/workflows/deploy-standalone-plugins.yml

swissspidy

Overall looks good 👍 Looking forward to seeing this in action.

mukeshpanchal27

@thelovekesh, thank you for the updates.

It seems that npm run test-plugins is broken. Have you checked the automated and manual plugin release workflow with dry-run set to true? This way, we can thoroughly test all steps of the workflow. Could you please ensure the following:

Added a pull_request for checking the automated workflow.
Set both auto and manual workflow DRY Run to true for security.
Updated SVN credentials for double security.

thelovekesh · 2024-03-11T18:45:13Z

Thanks @mukeshpanchal27. It seems like breaking changes are unrelated to this PR. I can see them on #1028 as well - https://github.com/WordPress/performance/actions/runs/8182298569/job/22373438106

Have you checked the automated and manual plugin release workflow with dry-run set to true?

Yes, I have tested them on my fork.

Standalone plugins release logs - https://github.com/thelovekesh/performance/actions/workflows/deploy-standalone-plugins.yml
PL plugin release logs - https://github.com/thelovekesh/performance/actions/workflows/deploy-dotorg.yml
Release assets linking - https://github.com/thelovekesh/performance/releases/tag/test-release
Deployment status - https://github.com/thelovekesh/performance/deployments/

felixarntz

@thelovekesh @swissspidy How does this PR determine which plugins to deploy an update for? Here's what I mean:

When a new Performance Lab release is triggered for instance, not all standalone plugins will need to be updated.
A plugin should only be updated if its new version is greater than the latest version currently available on wordpress.org.

I'm not certain the workflow currently accounts for that. I know the 10up deploy action handles this internally already, but I don't see anything in the other steps of our workflow that take that into account. Can you double check? 🤔

Also, another somewhat related comment on which ZIP files get attached to which GitHub release. FWIW, if that complicates getting this merged, we could always leave that out and add separately, also to solve it holistically for Performance Lab as well. We need to rethink our release strategy anyway.

.gitattributes

.github/workflows/deploy-standalone-plugins.yml

thelovekesh · 2024-03-12T20:34:14Z

I'm not certain the workflow currently accounts for that. I know the 10up deploy action handles this internally already, but I don't see anything in the other steps of our workflow that take that into account. Can you double check? 🤔

@felixarntz At this point, no it doesn't take that into account and it's that way only. We need to re-consider on how we release our plugins and IMO that should be a part of different issue/PR.

felixarntz · 2024-03-12T20:40:13Z

@thelovekesh

I'm not certain the workflow currently accounts for that. I know the 10up deploy action handles this internally already, but I don't see anything in the other steps of our workflow that take that into account. Can you double check? 🤔

@felixarntz At this point, no it doesn't take that into account and it's that way only. We need to re-consider on how we release our plugins and IMO that should be a part of different issue/PR.

I agree with you regarding the attachment of ZIP files to the release that it can be solved separately. However, the part about detecting whether a plugin even needs to be updated has to be handled here, otherwise it breaks existing behavior. For example:

Let's say "WebP Uploads" is currently live on wordpress.org in version 1.0.5.
In this repository, it currently has version 1.0.5 in its plugin main file and readme.txt.
Publishing a new release on GitHub must not attach a ZIP file for WebP Uploads to that release, because 1.0.5 was already published before.

In other words, this would be incorrect since the 1.0.5 ZIP generated wouldn't actually be the 1.0.5 release. That's why this needs to be fixed here to prevent this bug.

This reverts commit 65e4273.

…eady

thelovekesh · 2024-03-18T12:27:25Z

sure @felixarntz. I have updated the workflow only to deploy a plugin if it's not already deployed on the WPOrg.

felixarntz

@thelovekesh Looks great to me!

Before merge, @swissspidy could you take another brief look at the new additions since you had previously approved?

westonruter · 2024-03-18T18:17:05Z

.github/workflows/deploy-standalone-plugins.yml

+      - name: Upload release assets
+        if: steps.artifact-existence.outputs.exists == 'true'
+        uses: softprops/action-gh-release@v1
+        with:
+          files: |
+            ./build/dist/${{ matrix.plugin }}.zip
+            ./build/dist/${{ matrix.plugin }}.zip.sha256


So this will attach all standalone plugin ZIPs to the Performance Lab plugin release? If so, it seems like a missing piece in general for the standalone plugin story is how to trigger a release of a standalone plugin without having to do a release of Performance Lab as well. I guess that can still be done by manually invoking the workflow, but then there would be no record of the standalone plugin release among the GitHub releases.

Yes, that's right. For now, there are two scenarios:

Release trigger.

On workflow dispatch.

In the second case, there will be no release record as mentioned by @westonruter. In the last Performance chat, we discussed opening a new issue which I have just opened #1061 to discuss it.

@westonruter Correct, at the moment the manual workflow is the way to do it, if we need to release something apart from a PL release.

A discussion on how to better address that is long overdue and has come up in a few places before. Let's tackle that in a separate issue. Potentially we can just create GitHub releases for the individual plugins and have the workflows adjust based on e.g. the release name or something like that what gets deployed.

So this will attach all standalone plugin ZIPs to the Performance Lab plugin release?

Only for such plugins that are not already released on WPOrg. See 0d5451d

bin/webpack/utils.js

swissspidy

Still looks good to me now. I'd say let's just get this in and give it a try, we can always refine later if needed.

thelovekesh added [Type] Enhancement A suggestion for improvement of an existing feature Infrastructure Issues for the overall performance plugin infrastructure no milestone PRs that do not have a defined milestone for release labels Mar 6, 2024

thelovekesh self-assigned this Mar 6, 2024

thelovekesh requested review from mukeshpanchal27 and felixarntz as code owners March 6, 2024 16:50

thelovekesh commented Mar 6, 2024

View reviewed changes

.github/workflows/deploy-standalone-plugins.yml Show resolved Hide resolved

felixarntz requested changes Mar 6, 2024

View reviewed changes

mukeshpanchal27 requested changes Mar 7, 2024

View reviewed changes

.github/workflows/deploy-standalone-plugins.yml Show resolved Hide resolved

thelovekesh marked this pull request as draft March 7, 2024 08:51

thelovekesh force-pushed the update/plugin-bundling branch from 873a924 to 1d82457 Compare March 7, 2024 08:52

thelovekesh marked this pull request as ready for review March 7, 2024 17:04

thelovekesh requested review from westonruter, mukeshpanchal27 and felixarntz March 7, 2024 17:04

thelovekesh changed the title ~~Update plugin(s) build for distribution~~ Update plugin(s) build setup for distribution Mar 7, 2024

thelovekesh commented Mar 8, 2024

View reviewed changes

felixarntz reviewed Mar 8, 2024

View reviewed changes

.github/workflows/deploy-standalone-plugins.yml Show resolved Hide resolved

.github/workflows/deploy-standalone-plugins.yml Show resolved Hide resolved

felixarntz requested a review from swissspidy March 8, 2024 21:26

swissspidy reviewed Mar 10, 2024

View reviewed changes

.github/workflows/deploy-standalone-plugins.yml Show resolved Hide resolved

swissspidy approved these changes Mar 10, 2024

View reviewed changes

mukeshpanchal27 reviewed Mar 11, 2024

View reviewed changes

felixarntz reviewed Mar 11, 2024

View reviewed changes

.gitattributes Show resolved Hide resolved

.github/workflows/deploy-standalone-plugins.yml Outdated Show resolved Hide resolved

thelovekesh force-pushed the update/plugin-bundling branch from cfd474d to beb0500 Compare March 12, 2024 20:38

Remove .gitattributes

981dc73

thelovekesh added 16 commits March 18, 2024 15:34

Add debug steps

0416f3d

Add build dir deletion and manifest generation on appropriate hooks

19d29c6

Remove redundant error handling

c08ed0f

Remove get-plugin-version and get-plugin-dir command

0a5310d

Add deployments step in the release workflow

a7371d8

Fix status in deployments step in the release worflow

4aa56af

Update deployment status name

d5dc998

Move deploy.yml to deploy-standalone-plugins.yml

be8c207

Revert deploy-dotorg.yml workflow

8f22f4a

Add empty line in EOF

6a5173e

Remove build step for PL plugin

3dfa114

Remove .distignore

eeb26db

Revert "Remove .gitattributes"

e03fca0

This reverts commit 65e4273.

Add build-cs and webpack config

d471080

Update build command

bb98c66

Update standalone plugins workflow to only deploy if not deployed alr…

0d5451d

…eady

thelovekesh force-pushed the update/plugin-bundling branch from beb0500 to 0d5451d Compare March 18, 2024 12:21

thelovekesh requested review from felixarntz and mukeshpanchal27 March 18, 2024 12:27

thelovekesh mentioned this pull request Mar 18, 2024

Remove modules infrastructure and UI from the plugin #1060

Merged

felixarntz approved these changes Mar 18, 2024

View reviewed changes

westonruter reviewed Mar 18, 2024

View reviewed changes

thelovekesh mentioned this pull request Mar 18, 2024

Streamline PL and standalone plugins release #1061

Open

1 task

mukeshpanchal27 reviewed Mar 19, 2024

View reviewed changes

bin/webpack/utils.js Show resolved Hide resolved

swissspidy approved these changes Mar 19, 2024

View reviewed changes

mukeshpanchal27 approved these changes Mar 19, 2024

View reviewed changes

mukeshpanchal27 merged commit eaac49c into trunk Mar 19, 2024
49 checks passed

mukeshpanchal27 deleted the update/plugin-bundling branch March 19, 2024 15:50

thelovekesh mentioned this pull request Mar 22, 2024

DX improvements in new monorepo #1012

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update plugin(s) build setup for distribution #1033

Update plugin(s) build setup for distribution #1033

thelovekesh commented Mar 6, 2024 •

edited

github-actions bot commented Mar 6, 2024 •

edited

felixarntz left a comment

thelovekesh Mar 8, 2024

thelovekesh Mar 9, 2024

felixarntz left a comment

swissspidy Mar 10, 2024

thelovekesh Mar 11, 2024

swissspidy left a comment

mukeshpanchal27 left a comment

thelovekesh commented Mar 11, 2024 •

edited

felixarntz left a comment

thelovekesh commented Mar 12, 2024

felixarntz commented Mar 12, 2024 •

edited

thelovekesh commented Mar 18, 2024

felixarntz left a comment

westonruter Mar 18, 2024

thelovekesh Mar 18, 2024

felixarntz Mar 18, 2024

thelovekesh Mar 18, 2024

swissspidy left a comment

Update plugin(s) build setup for distribution #1033

Update plugin(s) build setup for distribution #1033

Conversation

thelovekesh commented Mar 6, 2024 • edited

Summary

Relevant technical choices

Checklist

github-actions bot commented Mar 6, 2024 • edited

felixarntz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felixarntz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swissspidy left a comment

Choose a reason for hiding this comment

mukeshpanchal27 left a comment

Choose a reason for hiding this comment

thelovekesh commented Mar 11, 2024 • edited

felixarntz left a comment

Choose a reason for hiding this comment

thelovekesh commented Mar 12, 2024

felixarntz commented Mar 12, 2024 • edited

thelovekesh commented Mar 18, 2024

felixarntz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swissspidy left a comment

Choose a reason for hiding this comment

thelovekesh commented Mar 6, 2024 •

edited

github-actions bot commented Mar 6, 2024 •

edited

thelovekesh commented Mar 11, 2024 •

edited

felixarntz commented Mar 12, 2024 •

edited