New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Try parallel app processing #73557
Merged
Merged
Try parallel app processing #73557
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
1cc07eb
Tweaks to existing builds
noahtallen 34e5cd9
Add function which can detect a changing calypso app
noahtallen 4503eab
Add generalized script for processing calypso apps
noahtallen 7e36a07
Fix weird error related to reserved keywords
noahtallen 4c04bbf
Fix issue with build script
noahtallen c44dbfb
Restore missing part of package.json
noahtallen File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -41,9 +41,10 @@ const excludedPackages = [ | |
]; | ||
|
||
const excludedPackagePlugins = excludedPackages.map( | ||
( package ) => | ||
// Note: apparently the word "package" is a reserved keyword here for some reason | ||
( pkg ) => | ||
new webpack.NormalModuleReplacementPlugin( | ||
package, | ||
pkg, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Weird issue, but can consistently replicate when running the translation script in happy blocks of all places There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nice catch! |
||
path.resolve( __dirname, 'src/components/nothing' ) | ||
) | ||
); | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
import { exec as _exec } from 'node:child_process'; | ||
import { createWriteStream } from 'node:fs'; | ||
import { Readable } from 'node:stream'; | ||
import { finished } from 'node:stream/promises'; | ||
import util from 'node:util'; | ||
const exec = util.promisify( _exec ); | ||
|
||
export default async function didCalypsoAppChange( { slug, dir, newReleaseDir, customNormalize } ) { | ||
await downloadPrevBuild( slug, dir ); | ||
await customNormalize?.(); | ||
try { | ||
await exec( | ||
`diff -rq --exclude="*.js.map" --exclude="*.asset.php" --exclude="build_meta.json" --exclude="README.md" ${ newReleaseDir } ${ dir }/prev-release/`, | ||
{ encoding: 'UTF-8', cwd: dir, stdio: 'inherit' } | ||
); | ||
return false; | ||
} catch ( { code, stdout, stderr } ) { | ||
if ( code === 1 ) { | ||
console.info( `The build for ${ slug } changed. Cause:` ); | ||
console.info( stdout ); | ||
return true; | ||
} | ||
throw new Error( `Unexpected error code ${ code } while diffing ${ slug } build: ${ stderr }` ); | ||
} | ||
} | ||
|
||
async function downloadPrevBuild( appSlug, dir ) { | ||
const prevBuildZip = `${ dir }/prev-archive-download.zip`; | ||
const stream = createWriteStream( prevBuildZip ); | ||
|
||
const prevBuildUrl = `${ process.env.tc_sever_url }/repository/download/calypso_calypso_WPComPlugins_Build_Plugins/${ appSlug }-release-build.tcbuildtag/${ appSlug }.zip?guest=1&branch=try-parallel-app-processing`; | ||
console.info( `Fetching previous release build for ${ appSlug } from ${ prevBuildUrl }` ); | ||
|
||
const { body, status } = await fetch( prevBuildUrl ); | ||
if ( status !== 200 ) { | ||
throw new Error( `Could not fetch previous build! Response code ${ status }.` ); | ||
} | ||
|
||
console.info( `Extracting downloaded archive for ${ appSlug }...` ); | ||
await finished( Readable.fromWeb( body ).pipe( stream ) ); | ||
await exec( `unzip -q ${ prevBuildZip } -d ${ dir }/prev-release`, { | ||
encoding: 'UTF-8', | ||
stdio: 'inherit', | ||
} ); | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,165 @@ | ||
import { exec as _exec } from 'node:child_process'; | ||
import { createHmac } from 'node:crypto'; | ||
import path from 'node:path'; | ||
import { fileURLToPath } from 'node:url'; | ||
import util from 'node:util'; | ||
import didCalypsoAppChange from './did-calypso-app-change.mjs'; | ||
|
||
checkEnvVars(); | ||
|
||
const exec = util.promisify( _exec ); | ||
const SKIP_BUILD_DIFF = process.env.skip_build_diff === 'true'; | ||
const IS_DEFAULT_BRANCH = process.env.is_default_branch === 'true'; | ||
const dirname = fileURLToPath( new URL( '.', import.meta.url ) ); | ||
const appRoot = path.resolve( dirname, '../apps' ); | ||
|
||
const apps = [ | ||
{ | ||
slug: 'happy-blocks', | ||
dir: path.resolve( appRoot, 'happy-blocks' ), | ||
newReleaseDir: path.resolve( appRoot, 'happy-blocks/release-files' ), | ||
slackNotify: true, | ||
}, | ||
]; | ||
|
||
// STEP 1: Check if any apps have changed. If skipping the diff, continue as if all apps have changed. | ||
const changedApps = SKIP_BUILD_DIFF | ||
? apps | ||
: ( | ||
await Promise.all( | ||
apps.map( async ( app ) => ( ( await didCalypsoAppChange( app ) ) ? app : null ) ) | ||
) | ||
).filter( Boolean ); | ||
|
||
if ( changedApps.length ) { | ||
console.info( | ||
'The following apps changed: ', | ||
changedApps.map( ( { slug } ) => slug ).join( ', ' ) | ||
); | ||
} else { | ||
console.info( 'No apps changed.' ); | ||
} | ||
|
||
// STEP 2: Tag the build in TeamCity. This will let future builds identify the previous release. | ||
const finalTasks = []; | ||
if ( changedApps.length ) { | ||
console.info( 'Tagging build...' ); | ||
finalTasks.push( tagBuild( changedApps ) ); | ||
} | ||
|
||
// STEP 3: Notify the author. On trunk, send a Slack notification. On a PR, a GitHub commnent. | ||
if ( ! IS_DEFAULT_BRANCH ) { | ||
console.info( 'Running GitHub comment...' ); | ||
finalTasks.push( addGitHubComment( changedApps ) ); | ||
} else { | ||
console.info( 'Running Slack notification...' ); | ||
finalTasks.push( sendSlackNotification( changedApps ) ); | ||
} | ||
|
||
await Promise.all( finalTasks ); | ||
console.log( 'Success!' ); | ||
|
||
async function tagBuild( _changedApps ) { | ||
const tags = _changedApps.map( ( app ) => `${ app.slug }-release-build` ); | ||
|
||
const tagurl = `https://teamcity.a8c.com/httpAuth/app/rest/builds/id:${ process.env.build_id }/tags/`; | ||
console.info( `Adding tags (${ tags }) to current build at URL ${ tagurl }` ); | ||
|
||
const jsonTags = JSON.stringify( { | ||
count: tags.length, | ||
tag: tags.map( ( tag ) => ( { | ||
name: tag, | ||
} ) ), | ||
} ); | ||
|
||
const res = await fetch( tagurl, { | ||
method: 'POST', | ||
headers: new Headers( { | ||
'Content-Type': 'application/json', | ||
Authorization: `Basic ${ Buffer.from( process.env.tc_auth ).toString( 'base64' ) }`, | ||
} ), | ||
body: jsonTags, | ||
} ); | ||
if ( res.status !== 200 ) { | ||
console.error( 'Tagging the build failed!' ); | ||
} | ||
} | ||
|
||
async function addGitHubComment( _changedApps ) { | ||
const notifyApps = _changedApps.filter( ( { ghNotify = true } ) => ghNotify ); | ||
|
||
const commentWatermark = 'calypso-app-artifacts'; | ||
const ghCommentCmd = `./bin/add-pr-comment.sh ${ process.env.git_branch } ${ commentWatermark }`; | ||
|
||
if ( ! notifyApps.length ) { | ||
console.info( 'No apps to notify about. Deleting existing comment if exists.' ); | ||
// Delete the existing comment, since there are no apps to notify about. | ||
return await exec( `${ ghCommentCmd } delete <<< "" || true`, { | ||
encoding: 'UTF-8', | ||
stdio: 'inherit', | ||
} ); | ||
} | ||
|
||
const header = '**This PR modifies the release build for the following Calypso Apps:**'; | ||
const docsMsg = '_For info about this notification, see here: PCYsg-OT6-p2_'; | ||
const changedAppsMsg = notifyApps.map( ( { slug } ) => `* ${ slug }` ).join( '\n' ); | ||
const testMsg = `To test WordPress.com changes, run "install-plugin.sh $pluginSlug ${ process.env.git_branch }" on your sandbox.`; | ||
|
||
const appMsg = `${ header }\n\n${ docsMsg }\n\n${ changedAppsMsg }\n\n${ testMsg }`; | ||
|
||
await exec( `${ ghCommentCmd } <<- EOF || true\n${ appMsg }\nEOF`, { | ||
encoding: 'UTF-8', | ||
} ); | ||
} | ||
|
||
async function sendSlackNotification( _changedApps ) { | ||
const notifyApps = _changedApps.filter( ( { slackNotify = false } ) => slackNotify ); | ||
|
||
if ( ! notifyApps.length ) { | ||
console.info( 'No apps to notify about. Skipping Slack notification.' ); | ||
return; | ||
} | ||
|
||
// TODO: move from one to multiple plugins! | ||
const body = `commit=${ process.env.commit_sha }&plugin=$pluginSlug`; | ||
|
||
const signature = createHmac( 'sha256', process.env.mc_auth_secret ) | ||
.update( body ) | ||
.digest( 'hex' ); | ||
|
||
console.log( `Sending data to slack endpoint: ${ body }` ); | ||
const res = await fetch( `${ process.env.mc_post_root }?plugin-deploy-reminder`, { | ||
method: 'POST', | ||
headers: new Headers( { | ||
'Content-Type': 'application/x-www-form-urlencoded', | ||
'TEAMCITY-SIGNATURE': signature, | ||
} ), | ||
body, | ||
} ); | ||
if ( res.status !== 200 ) { | ||
console.error( 'Slack notification failed!' ); | ||
console.error( 'Details: ', await res.text() ); | ||
} | ||
} | ||
|
||
function checkEnvVars() { | ||
const requiredVars = [ | ||
'tc_auth', | ||
'git_branch', | ||
'build_id', | ||
'GH_TOKEN', | ||
'is_default_branch', | ||
'skip_build_diff', | ||
'mc_auth_secret', | ||
'commit_sha', | ||
'mc_post_root', | ||
'tc_sever_url', | ||
]; | ||
|
||
// Undefined and empty strings will be detected. | ||
const missingVars = requiredVars.filter( ( varName ) => ! process.env[ varName ] ); | ||
if ( missingVars.length > 0 ) { | ||
console.error( `Missing required environment variables: ${ missingVars.join( ', ' ) }` ); | ||
process.exit( 1 ); | ||
} | ||
} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Prep artifact actually requires the "prev build" to be downloaded already, so it should be in the normalization script