Skip to content

Commit 150f9cf

Browse files
nathanwhitnathanwhit
authored
perf: order startup functions in release builds (#36219)
## Summary - generate an exact function-entry order from a first-pass release binary on Linux x86-64/arm64 and macOS arm64 - relink the same revision with the generated order, without committing revision-specific order artifacts - verify that the final binary preserves symbol coverage and improves ordering relative to the first-pass binary ## Technique Release CI now uses two passes for the supported targets: 1. Build the normal, unstripped release executable and preserve it as the baseline. 2. Run representative startup workloads three times each: empty JavaScript, cached TypeScript, an active timer, `deno serve`, `deno test`, `node:crypto`, and `deno fmt`. 3. Trap the first execution of each function entry and restore its original instruction. Linux uses `INT3` on x86-64 or `BRK` on arm64 with `memfd` RX/RW aliases; macOS uses `BRK` with shared-memory RX/RW aliases. Both arm64 tracers synchronize the instruction cache before resuming. A small runner suspends the generator while workloads execute so the generator's own V8 threads do not contaminate the trace. 4. Resolve traced addresses to linker-visible symbols and relink with LLD's `--symbol-ordering-file` on Linux or Apple's `-order_file` on macOS. 5. Compare the first-pass and final binaries. The verifier requires high symbol coverage and rejects regressions, while allowing an already well-ordered baseline to remain unchanged. Generating the order from the same binary that consumes it is important because Rust symbol hashes, outlined functions, and layout change across revisions. The generated order and verification report are uploaded as short-lived CI artifacts rather than checked into the repository. ## Results Release builds from the investigation: | Idle workload | Baseline | Page order | Exact function order | | --- | ---: | ---: | ---: | | Linux, no timers | 47.15 MiB | 34.63 MiB | **28.80 MiB** | | Linux, timer active | 53.77 MiB | 40.14 MiB | **33.58 MiB** | The exact Linux order saves 18.34 MiB without timers and 20.19 MiB with a timer versus the unordered release build. It also saves another 5.82-6.57 MiB over page-granularity ordering by reducing the order from 72,876 symbols to 22,963 startup functions. | Target | Ordered symbols | Timer-free RSS delta | Empty JS startup delta | Cached TypeScript startup delta | | --- | ---: | ---: | ---: | ---: | | Linux x86-64 | 22,963 | **-18,784 KiB** | **-1.897 ms** | **-2.009 ms** | | macOS arm64 | 46,869 | **-10,960 KiB** | **-1.052 ms** | **-1.078 ms** | Fresh macOS orders independently reproduced roughly 11.1 MiB RSS savings after source changes, while stale or symbol-remapped orders lost a substantial part of the benefit. This is why the workflow regenerates the order for every release build. Linux arm64 uses the same exact-entry workload and LLD ordering policy. Its native tracer path was validated separately below; a release RSS figure is intentionally omitted until measured from the full CI-built binary. ## Validation - `./tools/format.js` - `./tools/lint.js` - type-check all startup-order TypeScript tools with the local Deno build - compile both native tracers with `-Wall -Wextra -Werror` - run the Linux arm64 tracer in a native container through its production `memfd` RX/RW aliases, including 20 iterations of a 32-thread simultaneous-entry race - run the complete Linux arm64 generator/runner/tracer pipeline across all seven workload categories with three repeats and zero unresolved trace addresses - generate fresh macOS and Linux traces and confirm every traced address resolves to a symbol - run the relative verifier against unordered, ordered, and already-ordered control binaries on both platforms
1 parent 9dcb61d commit 150f9cf

10 files changed

Lines changed: 3070 additions & 1 deletion

‎.github/workflows/ci.generated.yml‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1876,6 +1876,45 @@ jobs:
18761876
cargo build --release --locked -p deno -p denort -p test_server --bin deno --bin denort --bin test_server --features=deno/panic-trace
18771877
cargo build --release --locked -p denort_desktop
18781878
df -h
1879+
- name: Trace startup order
1880+
if: '!(!contains(github.event.pull_request.labels.*.name, ''ci-full'') && github.event_name == ''pull_request'') && github.repository == ''denoland/deno'''
1881+
env:
1882+
NO_COLOR: 1
1883+
run: |-
1884+
cp -p target/release/deno target/release/deno-before-startup-order
1885+
target/release/deno run -A tools/startup_order/generate_macos_function_orderfile.ts \
1886+
--binary $GITHUB_WORKSPACE/target/release/deno-before-startup-order \
1887+
--output $GITHUB_WORKSPACE/target/release/startup-order-aarch64-apple-darwin.order \
1888+
--repeats 3 \
1889+
--workload-profile run-first
1890+
- name: Relink release deno with startup order
1891+
if: '!(!contains(github.event.pull_request.labels.*.name, ''ci-full'') && github.event_name == ''pull_request'') && github.repository == ''denoland/deno'''
1892+
env:
1893+
DENO_SNAPSHOT_MINIFY_SOURCES: '1'
1894+
DENO_USE_STARTUP_ORDER: '1'
1895+
DENO_STARTUP_ORDER_FILE: '${{ github.workspace }}/target/release/startup-order-aarch64-apple-darwin.order'
1896+
run: cargo build --release --locked -p deno -p denort -p test_server --bin deno --bin denort --bin test_server --features=deno/panic-trace
1897+
- name: Verify startup order
1898+
if: '!(!contains(github.event.pull_request.labels.*.name, ''ci-full'') && github.event_name == ''pull_request'') && github.repository == ''denoland/deno'''
1899+
env:
1900+
NO_COLOR: 1
1901+
run: |-
1902+
target/release/deno run -A tools/startup_order/verify_orderfile.ts \
1903+
--baseline-binary target/release/deno-before-startup-order \
1904+
--binary target/release/deno \
1905+
--order target/release/startup-order-aarch64-apple-darwin.order \
1906+
--output target/release/startup-order-aarch64-apple-darwin.order.verify.json
1907+
- name: Upload startup order
1908+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
1909+
if: '!(!contains(github.event.pull_request.labels.*.name, ''ci-full'') && github.event_name == ''pull_request'') && github.repository == ''denoland/deno'' && always()'
1910+
with:
1911+
name: startup-order-release-macos-aarch64
1912+
path: |-
1913+
target/release/startup-order-aarch64-apple-darwin.order
1914+
target/release/startup-order-aarch64-apple-darwin.order.json
1915+
target/release/startup-order-aarch64-apple-darwin.order.verify.json
1916+
retention-days: 7
1917+
if-no-files-found: warn
18791918
- name: Generate symcache
18801919
if: '!(!contains(github.event.pull_request.labels.*.name, ''ci-full'') && github.event_name == ''pull_request'') && github.repository == ''denoland/deno'''
18811920
env:
@@ -4392,6 +4431,43 @@ jobs:
43924431
cargo build --release --locked -p deno -p denort -p test_server --bin deno --bin denort --bin test_server --features=deno/panic-trace
43934432
cargo build --release --locked -p denort_desktop
43944433
df -h
4434+
- name: Trace startup order
4435+
env:
4436+
NO_COLOR: 1
4437+
run: |-
4438+
cp -p target/release/deno target/release/deno-before-startup-order
4439+
target/release/deno run -A tools/startup_order/generate_linux_function_orderfile.ts \
4440+
--binary $GITHUB_WORKSPACE/target/release/deno-before-startup-order \
4441+
--output $GITHUB_WORKSPACE/target/release/startup-order-x86_64-unknown-linux-gnu.order \
4442+
--repeats 3 \
4443+
--workload-profile run-first
4444+
- name: Relink release deno with startup order
4445+
env:
4446+
DENO_SNAPSHOT_MINIFY_SOURCES: '1'
4447+
DENO_USE_STARTUP_ORDER: '1'
4448+
DENO_STARTUP_ORDER_FILE: '${{ github.workspace }}/target/release/startup-order-x86_64-unknown-linux-gnu.order'
4449+
run: cargo build --release --locked -p deno -p denort -p test_server --bin deno --bin denort --bin test_server --features=deno/panic-trace
4450+
- name: Verify startup order
4451+
env:
4452+
NO_COLOR: 1
4453+
run: |-
4454+
target/release/deno run -A tools/startup_order/verify_orderfile.ts \
4455+
--baseline-binary target/release/deno-before-startup-order \
4456+
--binary target/release/deno \
4457+
--order target/release/startup-order-x86_64-unknown-linux-gnu.order \
4458+
--output target/release/startup-order-x86_64-unknown-linux-gnu.order.verify.json
4459+
- name: Upload startup order
4460+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
4461+
if: always()
4462+
with:
4463+
name: startup-order-release-linux-x86_64
4464+
path: |-
4465+
target/release/startup-order-x86_64-unknown-linux-gnu.order
4466+
target/release/startup-order-x86_64-unknown-linux-gnu.order.json
4467+
target/release/startup-order-x86_64-unknown-linux-gnu.order.starts.json
4468+
target/release/startup-order-x86_64-unknown-linux-gnu.order.verify.json
4469+
retention-days: 7
4470+
if-no-files-found: warn
43954471
- name: Check release snapshot flags
43964472
run: |-
43974473
if strings target/release/deno | grep -F -- '--no-lazy --no-lazy-eval --no-lazy-streaming'; then
@@ -6443,6 +6519,46 @@ jobs:
64436519
cargo build --release --locked -p deno -p denort -p test_server --bin deno --bin denort --bin test_server --features=deno/panic-trace
64446520
cargo build --release --locked -p denort_desktop
64456521
df -h
6522+
- name: Trace startup order
6523+
if: '!(!contains(github.event.pull_request.labels.*.name, ''ci-full'') && github.event_name == ''pull_request'')'
6524+
env:
6525+
NO_COLOR: 1
6526+
run: |-
6527+
cp -p target/release/deno target/release/deno-before-startup-order
6528+
target/release/deno run -A tools/startup_order/generate_linux_function_orderfile.ts \
6529+
--binary $GITHUB_WORKSPACE/target/release/deno-before-startup-order \
6530+
--output $GITHUB_WORKSPACE/target/release/startup-order-aarch64-unknown-linux-gnu.order \
6531+
--repeats 3 \
6532+
--workload-profile run-first
6533+
- name: Relink release deno with startup order
6534+
if: '!(!contains(github.event.pull_request.labels.*.name, ''ci-full'') && github.event_name == ''pull_request'')'
6535+
env:
6536+
DENO_SNAPSHOT_MINIFY_SOURCES: '1'
6537+
DENO_USE_STARTUP_ORDER: '1'
6538+
DENO_STARTUP_ORDER_FILE: '${{ github.workspace }}/target/release/startup-order-aarch64-unknown-linux-gnu.order'
6539+
run: cargo build --release --locked -p deno -p denort -p test_server --bin deno --bin denort --bin test_server --features=deno/panic-trace
6540+
- name: Verify startup order
6541+
if: '!(!contains(github.event.pull_request.labels.*.name, ''ci-full'') && github.event_name == ''pull_request'')'
6542+
env:
6543+
NO_COLOR: 1
6544+
run: |-
6545+
target/release/deno run -A tools/startup_order/verify_orderfile.ts \
6546+
--baseline-binary target/release/deno-before-startup-order \
6547+
--binary target/release/deno \
6548+
--order target/release/startup-order-aarch64-unknown-linux-gnu.order \
6549+
--output target/release/startup-order-aarch64-unknown-linux-gnu.order.verify.json
6550+
- name: Upload startup order
6551+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
6552+
if: '!(!contains(github.event.pull_request.labels.*.name, ''ci-full'') && github.event_name == ''pull_request'') && always()'
6553+
with:
6554+
name: startup-order-release-linux-aarch64
6555+
path: |-
6556+
target/release/startup-order-aarch64-unknown-linux-gnu.order
6557+
target/release/startup-order-aarch64-unknown-linux-gnu.order.json
6558+
target/release/startup-order-aarch64-unknown-linux-gnu.order.starts.json
6559+
target/release/startup-order-aarch64-unknown-linux-gnu.order.verify.json
6560+
retention-days: 7
6561+
if-no-files-found: warn
64466562
- name: Check release snapshot flags
64476563
if: '!(!contains(github.event.pull_request.labels.*.name, ''ci-full'') && github.event_name == ''pull_request'')'
64486564
run: |-

‎.github/workflows/ci.ts‎

Lines changed: 71 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -602,6 +602,15 @@ const buildItems = handleBuildItems([{
602602

603603
const buildJobs = buildItems.map((rawBuildItem) => {
604604
const buildItem = defineExprObj(rawBuildItem);
605+
const usesStartupOrder = rawBuildItem.profile === "release" &&
606+
((rawBuildItem.os === "linux" &&
607+
(rawBuildItem.arch === "x86_64" || rawBuildItem.arch === "aarch64")) ||
608+
(rawBuildItem.os === "macos" && rawBuildItem.arch === "aarch64"));
609+
const startupOrderTarget = rawBuildItem.os === "macos"
610+
? "aarch64-apple-darwin"
611+
: `${rawBuildItem.arch}-unknown-linux-gnu`;
612+
const startupOrderPath =
613+
`target/release/startup-order-${startupOrderTarget}.order`;
605614
const isLinux = buildItem.os.equals("linux");
606615
const isWindows = buildItem.os.equals("windows");
607616
const isMacos = buildItem.os.equals("macos");
@@ -899,6 +908,8 @@ const buildJobs = buildItems.map((rawBuildItem) => {
899908
.map((name) => `-p ${name}`).join(" ");
900909
const binsToBuild = ["deno", "denort", "test_server"]
901910
.map((name) => `--bin ${name}`).join(" ");
911+
const cargoBuildReleaseCommand =
912+
`cargo build --release --locked ${packagesToBuild} ${binsToBuild} --features=deno/panic-trace`;
902913
const cargoBuildReleaseStep = step
903914
.if(
904915
isRelease.and(isDenoland.or(buildItem.use_sysroot)),
@@ -936,14 +947,73 @@ const buildJobs = buildItems.map((rawBuildItem) => {
936947
"fi",
937948
// output fs space before and after building
938949
"df -h",
939-
`cargo build --release --locked ${packagesToBuild} ${binsToBuild} --features=deno/panic-trace`,
950+
cargoBuildReleaseCommand,
940951
// Build the desktop runtime shared library (libdenort cdylib) for
941952
// laufey-based desktop apps. Separate invocation because the
942953
// panic-trace feature only applies to the deno/denort binaries.
943954
"cargo build --release --locked -p denort_desktop",
944955
"df -h",
945956
],
946957
},
958+
...(usesStartupOrder
959+
? [{
960+
name: "Trace startup order",
961+
run: rawBuildItem.os === "macos"
962+
? [
963+
"cp -p target/release/deno target/release/deno-before-startup-order",
964+
"target/release/deno run -A tools/startup_order/generate_macos_function_orderfile.ts \\",
965+
" --binary $GITHUB_WORKSPACE/target/release/deno-before-startup-order \\",
966+
` --output $GITHUB_WORKSPACE/${startupOrderPath} \\`,
967+
" --repeats 3 \\",
968+
" --workload-profile run-first",
969+
]
970+
: [
971+
"cp -p target/release/deno target/release/deno-before-startup-order",
972+
"target/release/deno run -A tools/startup_order/generate_linux_function_orderfile.ts \\",
973+
" --binary $GITHUB_WORKSPACE/target/release/deno-before-startup-order \\",
974+
` --output $GITHUB_WORKSPACE/${startupOrderPath} \\`,
975+
" --repeats 3 \\",
976+
" --workload-profile run-first",
977+
],
978+
env: { NO_COLOR: 1 },
979+
}, {
980+
name: "Relink release deno with startup order",
981+
run: cargoBuildReleaseCommand,
982+
env: {
983+
DENO_SNAPSHOT_MINIFY_SOURCES: "1",
984+
DENO_USE_STARTUP_ORDER: "1",
985+
DENO_STARTUP_ORDER_FILE:
986+
`\${{ github.workspace }}/${startupOrderPath}`,
987+
},
988+
}, {
989+
name: "Verify startup order",
990+
run: [
991+
"target/release/deno run -A tools/startup_order/verify_orderfile.ts \\",
992+
" --baseline-binary target/release/deno-before-startup-order \\",
993+
" --binary target/release/deno \\",
994+
` --order ${startupOrderPath} \\`,
995+
` --output ${startupOrderPath}.verify.json`,
996+
],
997+
env: { NO_COLOR: 1 },
998+
}, {
999+
name: "Upload startup order",
1000+
uses: "actions/upload-artifact@v6",
1001+
if: conditions.status.always(),
1002+
with: {
1003+
name: `startup-order-${profileName}`,
1004+
path: [
1005+
startupOrderPath,
1006+
`${startupOrderPath}.json`,
1007+
...(rawBuildItem.os === "linux"
1008+
? [`${startupOrderPath}.starts.json`]
1009+
: []),
1010+
`${startupOrderPath}.verify.json`,
1011+
].join("\n"),
1012+
"retention-days": 7,
1013+
"if-no-files-found": "warn",
1014+
},
1015+
}]
1016+
: []),
9471017
{
9481018
name: "Check release snapshot flags",
9491019
if: isLinux,

‎cli/build.rs‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
use std::env;
44
use std::io::Write;
55
use std::path::Path;
6+
use std::path::PathBuf;
67

78
use deno_runtime::*;
89

@@ -431,6 +432,109 @@ fn emit_dts_rerun_if_changed() {
431432
}
432433
}
433434

435+
#[allow(clippy::disallowed_methods, reason = "build code")]
436+
fn emit_startup_order_link_args(out_dir: &Path) {
437+
const ENABLE_ENV: &str = "DENO_USE_STARTUP_ORDER";
438+
const ORDER_FILE_ENV: &str = "DENO_STARTUP_ORDER_FILE";
439+
440+
println!("cargo:rerun-if-env-changed={ENABLE_ENV}");
441+
println!("cargo:rerun-if-env-changed={ORDER_FILE_ENV}");
442+
443+
if env::var_os(ENABLE_ENV).is_none() {
444+
return;
445+
}
446+
if env::var(ENABLE_ENV).ok().as_deref() != Some("1") {
447+
panic!("{ENABLE_ENV} must be unset or set to 1");
448+
}
449+
450+
let profile = env::var("PROFILE").unwrap();
451+
if profile != "release" {
452+
panic!("startup ordering is only supported by the release profile");
453+
}
454+
455+
let target = env::var("TARGET").unwrap();
456+
match target.as_str() {
457+
"aarch64-apple-darwin"
458+
| "aarch64-unknown-linux-gnu"
459+
| "x86_64-unknown-linux-gnu" => {}
460+
_ => return,
461+
}
462+
463+
let source =
464+
PathBuf::from(env::var_os(ORDER_FILE_ENV).unwrap_or_else(|| {
465+
panic!(
466+
"{ORDER_FILE_ENV} must point to an order generated from the baseline \
467+
release binary using the default linker layout"
468+
)
469+
}));
470+
let source = source.canonicalize().unwrap_or_else(|error| {
471+
panic!(
472+
"failed to resolve startup order {}: {error}",
473+
source.display()
474+
)
475+
});
476+
println!("cargo:rerun-if-changed={}", source.display());
477+
478+
let contents = if source.extension().is_some_and(|ext| ext == "zst") {
479+
let compressed = std::fs::read(&source).unwrap_or_else(|error| {
480+
panic!("failed to read startup order {}: {error}", source.display())
481+
});
482+
zstd::stream::decode_all(compressed.as_slice()).unwrap_or_else(|error| {
483+
panic!(
484+
"failed to decompress startup order {}: {error}",
485+
source.display()
486+
)
487+
})
488+
} else {
489+
std::fs::read(&source).unwrap_or_else(|error| {
490+
panic!("failed to read startup order {}: {error}", source.display())
491+
})
492+
};
493+
494+
// Always use the same linker path for generated orders. Full-LTO output can
495+
// change when only the order-file path changes.
496+
let order_file = out_dir.join(format!("startup-order-{target}.order"));
497+
std::fs::write(&order_file, contents).unwrap_or_else(|error| {
498+
panic!(
499+
"failed to write startup order {}: {error}",
500+
order_file.display()
501+
)
502+
});
503+
504+
let contents = std::fs::read_to_string(&order_file).unwrap_or_else(|error| {
505+
panic!(
506+
"failed to validate startup order {}: {error}",
507+
order_file.display()
508+
)
509+
});
510+
let symbol_count = contents
511+
.lines()
512+
.filter(|line| !line.is_empty() && !line.starts_with('#'))
513+
.count();
514+
if symbol_count < 1_000 {
515+
panic!(
516+
"startup order {} has only {symbol_count} symbols",
517+
order_file.display()
518+
);
519+
}
520+
521+
let order_file = order_file.canonicalize().unwrap();
522+
match target.as_str() {
523+
"aarch64-apple-darwin" => println!(
524+
"cargo:rustc-link-arg-bin=deno=-Wl,-order_file,{}",
525+
order_file.display()
526+
),
527+
"aarch64-unknown-linux-gnu" | "x86_64-unknown-linux-gnu" => {
528+
println!(
529+
"cargo:rustc-link-arg-bin=deno=-Wl,--symbol-ordering-file={}",
530+
order_file.display()
531+
);
532+
println!("cargo:rustc-link-arg-bin=deno=-Wl,--no-warn-symbol-ordering");
533+
}
534+
_ => unreachable!(),
535+
}
536+
}
537+
434538
fn main() {
435539
let out_dir = std::path::PathBuf::from(std::env::var_os("OUT_DIR").unwrap());
436540

@@ -457,6 +561,7 @@ fn main() {
457561
// To debug snapshot issues uncomment:
458562
// op_fetch_asset::trace_serializer();
459563

564+
emit_startup_order_link_args(&out_dir);
460565
process_node_types(&out_dir);
461566

462567
// Always emit rerun-if-changed for dts files (they are included via

0 commit comments

Comments
 (0)