Skip to content

Commit

Permalink
Distribute bundles more evenly into CI shards (#26208)
Browse files Browse the repository at this point in the history
## Summary

Previously, we distributed bundles into shards and then checked if we
need to actually build that bundle. This can under-utilize shards
heavily (e.g.
https://app.circleci.com/pipelines/github/facebook/react/38611/workflows/df9e56e7-d476-49ee-9392-d8b37c81aa66/jobs/630545/parallel-runs/28?filterBy=ALL
only building a single bundle).

This won't result in an optimal distribution but, if we're lucky, we
might end up with shard #26 not taking 7mins anymore. The slowest shard
ultimately decicdes when we can start with testing builds.

## How did you test this change?

- [x] `CIRCLE_NODE_INDEX=28 CIRCLE_NODE_TOTAL=40 yarn build` building
more than 1 bundle
- [x] Check timings of `yarn_build` so that we don't up with an
over-stuffed shard (e.g. a shard having to build all the expensive
bundles). Casually dropping 60min idle time 🎉:
- Before:
https://app.circleci.com/pipelines/github/facebook/react/38683/workflows/a41533d7-811c-439d-9751-214ba06035c5/jobs/632230/timing
- After:
https://app.circleci.com/pipelines/github/facebook/react/38686/workflows/8a770df6-5b3e-41ea-b3b5-10abeae703e7/jobs/632247/timing
  • Loading branch information
eps1lon committed Feb 20, 2023
1 parent bb1e3d0 commit c7967b1
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions scripts/rollup/build.js
Expand Up @@ -527,10 +527,6 @@ function resolveEntryFork(resolvedEntry, isFBBundle) {
}

async function createBundle(bundle, bundleType) {
if (shouldSkipBundle(bundle, bundleType)) {
return;
}

const filename = getFilename(bundle, bundleType);
const logKey =
chalk.white.bold(filename) + chalk.dim(` (${bundleType.toLowerCase()})`);
Expand Down Expand Up @@ -766,6 +762,10 @@ async function buildEverything() {
);
}

bundles = bundles.filter(([bundle, bundleType]) => {
return !shouldSkipBundle(bundle, bundleType);
});

if (process.env.CIRCLE_NODE_TOTAL) {
// In CI, parallelize bundles across multiple tasks.
const nodeTotal = parseInt(process.env.CIRCLE_NODE_TOTAL, 10);
Expand Down

0 comments on commit c7967b1

Please sign in to comment.