-
Notifications
You must be signed in to change notification settings - Fork 1.9k
build: parallelize engine builds with turbo #8716
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
0f6db02
build: parallelize engine builds with turbo
kpal81xd 8a0e4f6
build: move engine orchestration into turbo
kpal81xd bd9cbd2
build: remove build banner logging
kpal81xd 3f42af8
build: clean rollup output in turbo logs
kpal81xd 201b276
build: update turbo
kpal81xd 764dcc9
refactor: reorder scripts
kpal81xd 4a3b882
build: update build script options for consistency
kpal81xd 0e7b14d
build: rename release build type to rel
kpal81xd f130386
build: restore visualizer build scripts
kpal81xd 068bceb
build: preserve rollup output colors
kpal81xd 336a369
build: fix examples engine build types
kpal81xd 624a3fd
build: add typed watch scripts
kpal81xd 0552e0d
build: clean output before turbo builds
kpal81xd 863c0cb
build: require node with parseArgs support
kpal81xd d8eb160
build: allow direct min builds without rel output
kpal81xd 57c777d
docs: fix build helper flag comments
kpal81xd d7ba508
build: preserve minified esm banners
kpal81xd 6871dd5
build: fix visualizer and clean task behavior
kpal81xd 742c53e
build: simplify rollup target generation
kpal81xd 9fb6cf2
chore: fix typescript errors in rollup-build-target.mjs
kpal81xd 1de46fd
build: restore rollup watch scripts
kpal81xd 2c7eb1b
chore: reorder clean script
kpal81xd f499bb2
build: preserve minified sourcemaps
kpal81xd File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,6 +2,7 @@ | |
| .cursor | ||
| .idea/ | ||
| .vscode/ | ||
| .turbo | ||
| build | ||
| tree*.*.html | ||
| coverage | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,45 +1,182 @@ | ||
| /** | ||
| * Build helper scripts | ||
| * Usage: node build.mjs [options] -- [rollup options] | ||
| * Usage: node build.mjs [options] | ||
| * | ||
| * Options: | ||
| * target[:<moduleFormat>][:<buildType>][:<bundleState>] - Specify the target | ||
| * - moduleFormat (esm, umd) | ||
| * - buildType (release, debug, profiler, min) | ||
| * - bundleState (unbundled, bundled) | ||
| * Example: target:esm:release:bundled | ||
| * --type - Specify the build type: rel, dbg, prf, min, types. | ||
| * --format - Specify the module format: esm, umd. | ||
| * --watch - Rebuild the Rollup leaf build when inputs change. | ||
| * --sourcemaps - Build with source maps using Rollup directly. | ||
| * --clean - Remove build output. | ||
| * | ||
| * treemap - Enable treemap build visualization (release only). | ||
| * treenet - Enable treenet build visualization (release only). | ||
| * treesun - Enable treesun build visualization (release only). | ||
| * treeflame - Enable treeflame build visualization (release only). | ||
| * --treemap - Enable treemap build visualization (rel only). | ||
| * --treenet - Enable treenet build visualization (rel only). | ||
| * --treesun - Enable treesun build visualization (rel only). | ||
| * --treeflame - Enable treeflame build visualization (rel only). | ||
| */ | ||
|
|
||
| import { execSync } from 'child_process'; | ||
| import { spawn } from 'node:child_process'; | ||
| import { rm } from 'node:fs/promises'; | ||
| import path from 'node:path'; | ||
| import { parseArgs, stripVTControlCharacters } from 'node:util'; | ||
|
|
||
| const args = process.argv.slice(2); | ||
| const JS_TYPES = /** @type {const} */ (['rel', 'dbg', 'prf', 'min']); | ||
| const BUILD_TYPES = /** @type {const} */ ([...JS_TYPES, 'types']); | ||
| const MODULE_FORMATS = /** @type {const} */ (['umd', 'esm']); | ||
|
|
||
| const ENV_START_MATCHES = [ | ||
| 'target', | ||
| 'treemap', | ||
| 'treenet', | ||
| 'treesun', | ||
| 'treeflame' | ||
| ]; | ||
| const TREE_FLAGS = ['treemap', 'treenet', 'treesun', 'treeflame']; | ||
| const BIN_DIR = path.join('node_modules', '.bin'); | ||
| const USAGE = `Usage: node build.mjs [options] | ||
|
|
||
| const env = []; | ||
| for (let i = 0; i < args.length; i++) { | ||
| if (ENV_START_MATCHES.some(match => args[i].startsWith(match)) && args[i - 1] !== '--environment') { | ||
| env.push(`--environment ${args[i]}`); | ||
| args.splice(i, 1); | ||
| i--; | ||
| continue; | ||
| Options: | ||
| --type <rel|dbg|prf|min|types> (default: rel) | ||
| --format <esm|umd> (default: esm, tree visualizers default to both) | ||
| --watch, -w | ||
| --sourcemaps, -m | ||
| --clean | ||
| --treemap, --treenet, --treesun, --treeflame | ||
|
|
||
| Use npm run build or turbo run build:all for aggregate builds.`; | ||
|
|
||
| const { values } = parseArgs({ | ||
| args: process.argv.slice(2), | ||
| options: { | ||
| type: { type: 'string' }, | ||
| format: { type: 'string' }, | ||
| watch: { type: 'boolean', short: 'w' }, | ||
| sourcemaps: { type: 'boolean', short: 'm' }, | ||
| clean: { type: 'boolean' }, | ||
| treemap: { type: 'boolean' }, | ||
| treenet: { type: 'boolean' }, | ||
| treesun: { type: 'boolean' }, | ||
| treeflame: { type: 'boolean' }, | ||
| help: { type: 'boolean', short: 'h' } | ||
| }, | ||
| allowPositionals: false | ||
| }); | ||
|
|
||
| const hasType = values.type !== undefined; | ||
| const type = values.type ?? 'rel'; | ||
| const hasFormat = values.format !== undefined; | ||
| const format = values.format ?? 'esm'; | ||
| const trees = TREE_FLAGS.filter(flag => values[flag]); | ||
|
|
||
| const pipe = (input, output) => { | ||
| let text = ''; | ||
| let style = ''; | ||
| input.setEncoding('utf8'); | ||
| input.on('data', (chunk) => { | ||
| text += chunk.replace(/\r/g, '\n'); | ||
| const lines = text.split('\n'); | ||
| text = lines.pop(); | ||
| for (const line of lines) { | ||
| const raw = line.trimEnd(); | ||
| const out = stripVTControlCharacters(line).trimEnd(); | ||
| if (out.trim()) { | ||
| output.write(`${style}${raw}\n`); | ||
| style = ''; | ||
| } else if (raw) { | ||
| style += raw; | ||
| } | ||
| } | ||
| }); | ||
| input.on('end', () => { | ||
| const raw = text.trimEnd(); | ||
| const out = stripVTControlCharacters(text).trimEnd(); | ||
| if (out.trim()) { | ||
| output.write(`${style}${raw}\n`); | ||
| } | ||
| }); | ||
| }; | ||
|
|
||
| const run = (cmd, args) => { | ||
| const child = spawn(cmd, args, { | ||
| shell: process.platform === 'win32', | ||
| stdio: values.watch ? 'inherit' : ['inherit', 'pipe', 'pipe'] | ||
| }); | ||
| if (child.stdout) { | ||
| pipe(child.stdout, process.stdout); | ||
| } | ||
| if (child.stderr) { | ||
| pipe(child.stderr, process.stderr); | ||
| } | ||
| return new Promise((resolve) => { | ||
| child.on('error', (err) => { | ||
| console.error(err.message); | ||
| resolve(1); | ||
| }); | ||
| child.on('close', code => resolve(code ?? 1)); | ||
| }); | ||
| }; | ||
|
|
||
| const fail = (msg) => { | ||
| console.error(msg); | ||
| process.exit(1); | ||
| }; | ||
|
|
||
| const bin = name => path.join(BIN_DIR, process.platform === 'win32' ? `${name}.cmd` : name); | ||
|
|
||
| const getRollupBuild = () => { | ||
| if (!BUILD_TYPES.includes(type)) { | ||
| fail(`--type must be one of: ${BUILD_TYPES.join(', ')}`); | ||
| } | ||
|
|
||
| if (hasFormat && !MODULE_FORMATS.includes(format)) { | ||
| fail(`--format must be one of: ${MODULE_FORMATS.join(', ')}`); | ||
| } | ||
|
|
||
| if (trees.length && type !== 'rel') { | ||
| fail('tree visualizers only support --type=rel'); | ||
| } | ||
|
|
||
| if (values.watch && !hasType && !hasFormat) { | ||
| return null; | ||
| } | ||
|
|
||
| if ((values.watch || trees.length) && !hasFormat) { | ||
| return `build:${type}`; | ||
| } | ||
|
|
||
| if (type === 'types') { | ||
| if (values.format) { | ||
| fail('--type=types cannot be combined with --format'); | ||
| } | ||
| return 'build:types'; | ||
| } | ||
|
|
||
| return `build:${format}:${type}`; | ||
| }; | ||
|
|
||
| const runRollup = () => { | ||
| const env = []; | ||
| const build = getRollupBuild(); | ||
| if (build) { | ||
| env.push(build); | ||
| } | ||
| env.push(...trees); | ||
|
|
||
| const args = ['-c']; | ||
| if (values.sourcemaps) { | ||
| args.push('-m'); | ||
| } | ||
| for (const item of env) { | ||
| args.push('--environment', item); | ||
| } | ||
| if (values.watch) { | ||
| args.push('-w', '--no-watch.clearScreen'); | ||
| } | ||
|
|
||
| return run(bin('rollup'), args); | ||
| }; | ||
|
|
||
| if (values.help) { | ||
| console.log(USAGE); | ||
| process.exit(0); | ||
| } | ||
|
|
||
| const cmd = `rollup -c ${args.join(' ')} ${env.join(' ')}`; | ||
| try { | ||
| execSync(cmd, { stdio: 'inherit' }); | ||
| } catch (e) { | ||
| console.error(e.message); | ||
| if (values.clean) { | ||
| await rm('build', { recursive: true, force: true }); | ||
| process.exit(0); | ||
| } | ||
|
|
||
| process.exitCode = await runRollup(); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.