New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Try parallel app processing #73557
Try parallel app processing #73557
Conversation
This PR does not affect the size of JS and CSS bundles shipped to the user's browser. Generated by performance advisor bot at iscalypsofastyet.com. |
1c10170
to
ac47dce
Compare
This PR modifies the release build for the following Calypso Apps: For info about this notification, see here: PCYsg-OT6-p2
To test WordPress.com changes, run "install-plugin.sh try-parallel-app-processing" on your sandbox. |
f15d61e
to
4f7f88a
Compare
@@ -22,7 +22,7 @@ | |||
"start": "yarn run clean && yarn run build:notifications && yarn run dev-server", | |||
"dev": "yarn run calypso-apps-builder --localPath dist --remotePath /home/wpcom/public_html/widgets.wp.com/notifications", | |||
"build": "NODE_ENV=production yarn dev", | |||
"build-ci": "yarn build && ./bin/prepArtifactForCI.sh" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Prep artifact actually requires the "prev build" to be downloaded already, so it should be in the normalization script
I spun up a PR to hopefully address that. I didn't want to do it in the main stream of the CI optimization work, because (IIUC) the changeover to |
4f7f88a
to
f12bbfe
Compare
@noahtallen This PR is still in draft, but I think it's ready to review and possibly merge. What do you think? |
I'll take a look at the CI error and get it ready! You're probably right, we can test it out more after merging and getting more apps set up with it. |
f12bbfe
to
4503eab
Compare
537ca76
to
7e36a07
Compare
new webpack.NormalModuleReplacementPlugin( | ||
package, | ||
pkg, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Weird issue, but can consistently replicate when running the translation script in happy blocks of all places
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice catch!
done | ||
|
||
# Add the index.php file | ||
cp ./index.php ./README.md ./dist/* ./release-files/ | ||
cp ./index.php ./README.md ./release-files/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dist isn't used anymore, which can be verified by checking the trunk artifacts
This PR modifies the release build for happy-blocks To test your changes on WordPress.com, run To deploy your changes after merging, see the documentation: PCYsg-r7r-p2 |
mkdir -p ./release-files/${block//\.\.\//}; | ||
cp -r $block/build/* ./release-files/${block//\.\.\//}/; | ||
mkdir -p "./release-files/${block//\.\.\//}"; | ||
cp -r $block/build/* "./release-files/${block//\.\.\//}/"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a feeling this code shouldn't be in this script. It seems necessary for the plugin to work normally, but it's in a CI-only script. So the normal yarn build
script won't work
@worldomonation I wonder if this means we should move the CI related stuff back to .teamcity, so it's clearer that it should only be used for CI workflows, not for local builds 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@noahtallen It's been close to a month so my recall is a bit hazy but I do think we can remove all build-ci
yarn scripts, prepArtifactForCI.sh
bash scripts and such from the repo for now. IIRC I tried to incorporate the processing into TeamCity in a follow-up branch I had.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should I push to your branch to do that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok cool! I don't think we need to do it in this branch. If we merge this separately we can split that up into other PRs which are easier to look at individually
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gotcha. I'll rebase my WIP branch after we merge, and get rid of these files and scripts.
This PR creates an all-in-one Calypso app artifact processor. In the previous approach, with each app processed in an individual TeamCity build, the artifact process script only had to care about one app.
The major change here is to parallelize and combine in the following ways:
Promise.all
over functions which return boolean, which in turn gives us a list of apps which are changing.All of this is re-written in NodeJS to be (hopefully!) easier to read than bash script in kotlin string! It's also easier to manage data and parallelization.
I've tested the following parts:
What's missing: the previous artifact processor added a build_meta.txt file. I'd like to deprecate that in favor of build_meta.json. Some builds generate that with webpack, but we need some way to get it in all builds.