Skip to content

fix: test-cargo-unit CI job timing out since LTO was enabled#91424

Merged
lukesandberg merged 2 commits intocanaryfrom
fix_unit_test_timeouts_lol_lto
Mar 15, 2026
Merged

fix: test-cargo-unit CI job timing out since LTO was enabled#91424
lukesandberg merged 2 commits intocanaryfrom
fix_unit_test_timeouts_lol_lto

Conversation

@lukesandberg
Copy link
Contributor

@lukesandberg lukesandberg commented Mar 15, 2026

What

Fixes the test cargo unit CI job which has been consistently timing out (30 min) on canary since #91343 landed on Mar 14.

Why

Two issues were contributing to the timeout:

  1. LTO on unit test binaries: Enable thin LTO for release builds #91343 added lto = "thin" to [profile.release]. The release-with-assertions profile (used by test-cargo-unit) inherits from release, so it inherited LTO. Unlike the native binary build which only links one artifact, cargo nextest run --workspace builds test binaries for dozens of crates — each requiring its own LTO link pass. This pushed the build from ~8 minutes to 30+ minutes, exceeding the job timeout.

  2. No remote cache timeout or streaming output: The turbo run test-cargo-unit command had no --remote-cache-timeout (unlike the build step which uses --remote-cache-timeout 60) and used turbo's default grouped log output, which buffers all output until the task completes. This made the job appear completely silent for 30 minutes before being killed, making it impossible to diagnose.

How

  • Added a release-with-assertions-no-lto cargo profile that inherits from release-with-assertions but disables LTO, and switched test-cargo-unit to use it. The production native build continues to use release-with-assertions with LTO.
  • Added --remote-cache-timeout 60 --log-order stream to the turbo command for test-cargo-unit for parity with the build step and real-time output visibility.

Copy link
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@lukesandberg lukesandberg changed the title add a remote-cache-timeout timeout to the 'test cargo unit' job fix: test-cargo-unit CI job timing out since LTO was enabled Mar 15, 2026
@lukesandberg lukesandberg changed the title fix: test-cargo-unit CI job timing out since LTO was enabled disab Mar 15, 2026
@lukesandberg lukesandberg changed the title disab fix: test-cargo-unit CI job timing out since LTO was enabled Mar 15, 2026
@lukesandberg lukesandberg marked this pull request as ready for review March 15, 2026 19:51
@lukesandberg lukesandberg requested a review from a team March 15, 2026 19:52
@nextjs-bot
Copy link
Collaborator

nextjs-bot commented Mar 15, 2026

Tests Passed

@codspeed-hq
Copy link

codspeed-hq bot commented Mar 15, 2026

Merging this PR will not alter performance

✅ 17 untouched benchmarks
⏩ 3 skipped benchmarks1


Comparing fix_unit_test_timeouts_lol_lto (d401492) with canary (0f59973)

Open in CodSpeed

Footnotes

  1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@nextjs-bot
Copy link
Collaborator

Stats from current PR

✅ No significant changes detected

📊 All Metrics
📖 Metrics Glossary

Dev Server Metrics:

  • Listen = TCP port starts accepting connections
  • First Request = HTTP server returns successful response
  • Cold = Fresh build (no cache)
  • Warm = With cached build artifacts

Build Metrics:

  • Fresh = Clean build (no .next directory)
  • Cached = With existing .next directory

Change Thresholds:

  • Time: Changes < 50ms AND < 10%, OR < 2% are insignificant
  • Size: Changes < 1KB AND < 1% are insignificant
  • All other changes are flagged to catch regressions

⚡ Dev Server

Metric Canary PR Change Trend
Cold (Listen) 455ms 456ms ▄▆█▁▁
Cold (Ready in log) 440ms 440ms ▃▄█▁▁
Cold (First Request) 1.172s 1.170s ▂▄█▁▁
Warm (Listen) 456ms 456ms ▄▆█▁▁
Warm (Ready in log) 445ms 444ms ▃▄█▁▁
Warm (First Request) 348ms 350ms ▂▃█▁▁
📦 Dev Server (Webpack) (Legacy)

📦 Dev Server (Webpack)

Metric Canary PR Change Trend
Cold (Listen) 455ms 455ms █▁▁▁▁
Cold (Ready in log) 436ms 434ms █▁▃▄▁
Cold (First Request) 1.840s 1.856s █▁▂▂▁
Warm (Listen) 456ms 456ms █▁▁▁▁
Warm (Ready in log) 434ms 435ms █▁▃▄▁
Warm (First Request) 1.861s 1.857s █▁▂▂▁

⚡ Production Builds

Metric Canary PR Change Trend
Fresh Build 3.816s 3.720s ▂▅█▁▁
Cached Build 3.792s 3.741s ▃▅█▂▁
📦 Production Builds (Webpack) (Legacy)

📦 Production Builds (Webpack)

Metric Canary PR Change Trend
Fresh Build 14.192s 14.174s █▁▁▁▁
Cached Build 14.340s 14.277s █▁▁▁▁
node_modules Size 483 MB 483 MB ▁▁███
📦 Bundle Sizes

Bundle Sizes

⚡ Turbopack

Client

Main Bundles
Canary PR Change
0-m3lxn9ivp_d.js gzip 162 B N/A -
0.-r71sd1j_zb.js gzip 7.61 kB N/A -
0~-plqowrxaiq.js gzip 170 B N/A -
0~lwfcrlb4v_9.css gzip 115 B 115 B
00h0nz7r436~l.js gzip 13.3 kB N/A -
029_0n_61aap-.js gzip 70.8 kB N/A -
02ku7edzc_wf7.js gzip 450 B N/A -
03~yq9q893hmn.js gzip 39.4 kB 39.4 kB
092lcb3fqrrf9.js gzip 8.52 kB N/A -
0ac.phskjau0b.js gzip 158 B N/A -
0aj~xs1l1g8tg.js gzip 8.53 kB N/A -
0g0avicw0a5.l.js gzip 156 B N/A -
0gcm6h40wgorw.js gzip 48.2 kB N/A -
0h~_p6c6krw3f.js gzip 155 B N/A -
0h35gmp9u328z.js gzip 8.54 kB N/A -
0h6fkavebp.iz.js gzip 8.47 kB N/A -
0ino_yf1k3h6k.js gzip 10.4 kB N/A -
0k.sixc712dq1.js gzip 10.1 kB N/A -
0kkm7tesfinr6.js gzip 12.9 kB N/A -
0l4bu6-iylytu.js gzip 156 B N/A -
0moy~uao4dl.m.js gzip 9.19 kB N/A -
0oxfh9vf9xol0.js gzip 156 B N/A -
0q50rtpusjy90.js gzip 2.28 kB N/A -
0smgy2grrrlka.js gzip 8.58 kB N/A -
0t1dzhdfh0txh.js gzip 215 B 215 B
0u2bvb3fwjc.j.js gzip 155 B N/A -
0vwpsns7_n8j5.js gzip 13.6 kB N/A -
0yedy2nlxl.ia.js gzip 162 B N/A -
0zid7o0-vupvp.js gzip 225 B N/A -
10p-bsrowjca0.js gzip 158 B N/A -
11s6x9pcj4vxt.js gzip 157 B N/A -
11yo3xfd6b147.js gzip 12.9 kB N/A -
13.84hqxl_1p7.js gzip 9.76 kB N/A -
14v2a7u4xu22z.js gzip 157 B N/A -
1554wr-t7p6z-.js gzip 8.55 kB N/A -
15s.go9_6chuz.js gzip 65.7 kB N/A -
15tjst79~qy3_.js gzip 1.46 kB N/A -
15z_v00ne4ud0.js gzip 8.47 kB N/A -
17d_m3p4j9w6r.js gzip 5.62 kB N/A -
17yu~3yiu7d2m.js gzip 8.52 kB N/A -
188gain_2u3k9.js gzip 157 B N/A -
turbopack-0...b8_0.js gzip 4.16 kB N/A -
turbopack-03..u6-n.js gzip 4.16 kB N/A -
turbopack-06...odu.js gzip 4.14 kB N/A -
turbopack-09..jw41.js gzip 4.16 kB N/A -
turbopack-09..i1z9.js gzip 4.16 kB N/A -
turbopack-0m..vxy3.js gzip 4.16 kB N/A -
turbopack-0m..~hqc.js gzip 4.15 kB N/A -
turbopack-0o..8~3x.js gzip 4.16 kB N/A -
turbopack-0q..mhg_.js gzip 4.16 kB N/A -
turbopack-0s..gliq.js gzip 4.16 kB N/A -
turbopack-0u..k9zu.js gzip 4.16 kB N/A -
turbopack-10..8sax.js gzip 4.15 kB N/A -
turbopack-11..4bv0.js gzip 4.16 kB N/A -
turbopack-15..tsj9.js gzip 4.17 kB N/A -
01umcdex4f~m9.js gzip N/A 13.6 kB -
02gzbp8wsyfo-.js gzip N/A 70.8 kB -
03t__~.5lvgeu.js gzip N/A 5.62 kB -
04d6ll75jqx3r.js gzip N/A 9.19 kB -
04yegd4pwdu3f.js gzip N/A 152 B -
05.z3nadwcjhm.js gzip N/A 155 B -
0508ye.nu_ii2.js gzip N/A 157 B -
05673jb9k5ren.js gzip N/A 157 B -
0583exyh-yhc7.js gzip N/A 9.76 kB -
05b07nana~4_..js gzip N/A 10.1 kB -
072lv63r8dcz~.js gzip N/A 8.58 kB -
076vw6tg50945.js gzip N/A 153 B -
08refq~-egobr.js gzip N/A 156 B -
09j2nyjh-gde8.js gzip N/A 158 B -
0ar1~bwpezfgw.js gzip N/A 13.3 kB -
0c3leaugxcvgb.js gzip N/A 155 B -
0c99mq1ez2bke.js gzip N/A 450 B -
0cq-cmde_ws6u.js gzip N/A 8.47 kB -
0fwf102w10o9~.js gzip N/A 8.52 kB -
0gtmn.q_j1v5r.js gzip N/A 10.4 kB -
0l4d~z7ixj3wz.js gzip N/A 12.9 kB -
0n7bzm036.uw3.js gzip N/A 160 B -
0nclq9z6yzzm5.js gzip N/A 1.46 kB -
0nvnmuxpdyn-t.js gzip N/A 65.7 kB -
0nzumcogektg7.js gzip N/A 8.55 kB -
0s.c-cn5eebrx.js gzip N/A 8.47 kB -
0s5ej0rhsf.3a.js gzip N/A 169 B -
0tna7lg6q4zne.js gzip N/A 12.9 kB -
0votdfxr5fb5u.js gzip N/A 2.28 kB -
0xkuhv202qqhu.js gzip N/A 7.6 kB -
0y~srtt-b14d..js gzip N/A 155 B -
0ykl9bs_qj.5..js gzip N/A 8.52 kB -
0zfen0tnxp4gh.js gzip N/A 8.55 kB -
10wkq1h9jzkg..js gzip N/A 225 B -
13ncdn~mslimg.js gzip N/A 162 B -
149ndfh8zfcaz.js gzip N/A 8.53 kB -
14baw4rjxv_ds.js gzip N/A 48.1 kB -
14ix9emhgsyxn.js gzip N/A 154 B -
turbopack-0_..zz7r.js gzip N/A 4.16 kB -
turbopack-0...0dw_.js gzip N/A 4.16 kB -
turbopack-0~..tg3z.js gzip N/A 4.16 kB -
turbopack-0~..azp4.js gzip N/A 4.16 kB -
turbopack-0c..2hn8.js gzip N/A 4.16 kB -
turbopack-0h..-gyh.js gzip N/A 4.16 kB -
turbopack-0i..np7y.js gzip N/A 4.14 kB -
turbopack-0p..~xt1.js gzip N/A 4.16 kB -
turbopack-0r..lj0f.js gzip N/A 4.16 kB -
turbopack-0v..mnsx.js gzip N/A 4.16 kB -
turbopack-0y..jxh8.js gzip N/A 4.16 kB -
turbopack-10..yp1k.js gzip N/A 4.16 kB -
turbopack-11..47ja.js gzip N/A 4.17 kB -
turbopack-16..~v78.js gzip N/A 4.16 kB -
Total 463 kB 462 kB ✅ -15 B

Server

Middleware
Canary PR Change
middleware-b..fest.js gzip 712 B 716 B
Total 712 B 716 B ⚠️ +4 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 426 B 429 B
Total 426 B 429 B ⚠️ +3 B

📦 Webpack

Client

Main Bundles
Canary PR Change
5528-HASH.js gzip 5.54 kB N/A -
6280-HASH.js gzip 60.1 kB N/A -
6335.HASH.js gzip 169 B N/A -
912-HASH.js gzip 4.59 kB N/A -
e8aec2e4-HASH.js gzip 62.7 kB N/A -
framework-HASH.js gzip 59.7 kB 59.7 kB
main-app-HASH.js gzip 255 B 254 B
main-HASH.js gzip 39.2 kB 39.2 kB
webpack-HASH.js gzip 1.68 kB 1.68 kB
262-HASH.js gzip N/A 4.59 kB -
2889.HASH.js gzip N/A 169 B -
5602-HASH.js gzip N/A 5.55 kB -
6948ada0-HASH.js gzip N/A 62.7 kB -
9544-HASH.js gzip N/A 60.8 kB -
Total 234 kB 235 kB ⚠️ +690 B
Polyfills
Canary PR Change
polyfills-HASH.js gzip 39.4 kB 39.4 kB
Total 39.4 kB 39.4 kB
Pages
Canary PR Change
_app-HASH.js gzip 194 B 194 B
_error-HASH.js gzip 183 B 180 B 🟢 3 B (-2%)
css-HASH.js gzip 331 B 330 B
dynamic-HASH.js gzip 1.81 kB 1.81 kB
edge-ssr-HASH.js gzip 256 B 256 B
head-HASH.js gzip 351 B 352 B
hooks-HASH.js gzip 384 B 383 B
image-HASH.js gzip 580 B 581 B
index-HASH.js gzip 260 B 260 B
link-HASH.js gzip 2.51 kB 2.51 kB
routerDirect..HASH.js gzip 320 B 319 B
script-HASH.js gzip 386 B 386 B
withRouter-HASH.js gzip 315 B 315 B
1afbb74e6ecf..834.css gzip 106 B 106 B
Total 7.98 kB 7.98 kB ✅ -1 B

Server

Edge SSR
Canary PR Change
edge-ssr.js gzip 125 kB 125 kB
page.js gzip 268 kB 267 kB
Total 393 kB 392 kB ✅ -309 B
Middleware
Canary PR Change
middleware-b..fest.js gzip 616 B 612 B
middleware-r..fest.js gzip 156 B 155 B
middleware.js gzip 43.9 kB 43.9 kB
edge-runtime..pack.js gzip 842 B 842 B
Total 45.5 kB 45.5 kB ✅ -16 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 715 B 718 B
Total 715 B 718 B ⚠️ +3 B
Build Cache
Canary PR Change
0.pack gzip 4.26 MB 4.25 MB 🟢 12.2 kB (0%)
index.pack gzip 108 kB 108 kB
index.pack.old gzip 108 kB 107 kB
Total 4.47 MB 4.46 MB ✅ -12.9 kB

🔄 Shared (bundler-independent)

Runtimes
Canary PR Change
app-page-exp...dev.js gzip 332 kB 332 kB
app-page-exp..prod.js gzip 180 kB 180 kB
app-page-tur...dev.js gzip 332 kB 332 kB
app-page-tur..prod.js gzip 180 kB 180 kB
app-page-tur...dev.js gzip 328 kB 328 kB
app-page-tur..prod.js gzip 178 kB 178 kB
app-page.run...dev.js gzip 329 kB 329 kB
app-page.run..prod.js gzip 179 kB 179 kB
app-route-ex...dev.js gzip 76 kB 76 kB
app-route-ex..prod.js gzip 51.7 kB 51.7 kB
app-route-tu...dev.js gzip 76 kB 76 kB
app-route-tu..prod.js gzip 51.7 kB 51.7 kB
app-route-tu...dev.js gzip 75.6 kB 75.6 kB
app-route-tu..prod.js gzip 51.5 kB 51.5 kB
app-route.ru...dev.js gzip 75.5 kB 75.5 kB
app-route.ru..prod.js gzip 51.4 kB 51.4 kB
dist_client_...dev.js gzip 324 B 324 B
dist_client_...dev.js gzip 326 B 326 B
dist_client_...dev.js gzip 318 B 318 B
dist_client_...dev.js gzip 317 B 317 B
pages-api-tu...dev.js gzip 43.3 kB 43.3 kB
pages-api-tu..prod.js gzip 33 kB 33 kB
pages-api.ru...dev.js gzip 43.3 kB 43.3 kB
pages-api.ru..prod.js gzip 33 kB 33 kB
pages-turbo....dev.js gzip 52.7 kB 52.7 kB
pages-turbo...prod.js gzip 38.6 kB 38.6 kB
pages.runtim...dev.js gzip 52.7 kB 52.7 kB
pages.runtim..prod.js gzip 38.6 kB 38.6 kB
server.runti..prod.js gzip 62.4 kB 62.4 kB
Total 2.95 MB 2.95 MB ⚠️ +2 B
📎 Tarball URL
https://vercel-packages.vercel.app/next/commits/267e4d2476ddde656accfd632097728b38f7232a/next

@lukesandberg lukesandberg merged commit 4a99afc into canary Mar 15, 2026
428 of 440 checks passed
Copy link
Contributor Author

Merge activity

@lukesandberg lukesandberg deleted the fix_unit_test_timeouts_lol_lto branch March 15, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants