external node_modules for SSR #3361

ForsakenHarmony · 2023-01-18T14:56:21Z

Adds resolve plugins (currently only running after_resolve, could add resolve later)

Had to fix ResolveResult::merge_alternatives because it merged two external results into Unresolvable

vercel · 2023-01-18T14:56:27Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
examples-cra-web	🔄 Building (Inspect)			Jan 31, 2023 at 4:38PM (UTC)
examples-nonmonorepo	🔄 Building (Inspect)			Jan 31, 2023 at 4:38PM (UTC)

9 Ignored Deployments

Name	Status	Preview	Updated
examples-basic-web	⬜️ Ignored (Inspect)		Jan 31, 2023 at 4:38PM (UTC)
examples-designsystem-docs	⬜️ Ignored (Inspect)		Jan 31, 2023 at 4:38PM (UTC)
examples-kitchensink-blog	⬜️ Ignored (Inspect)		Jan 31, 2023 at 4:38PM (UTC)
examples-native-web	⬜️ Ignored (Inspect)		Jan 31, 2023 at 4:38PM (UTC)
examples-svelte-web	⬜️ Ignored (Inspect)		Jan 31, 2023 at 4:38PM (UTC)
examples-tailwind-web	⬜️ Ignored (Inspect)		Jan 31, 2023 at 4:38PM (UTC)
examples-vite-web	⬜️ Ignored (Inspect)		Jan 31, 2023 at 4:38PM (UTC)
turbo-site	⬜️ Ignored (Inspect)	Visit Preview	Jan 31, 2023 at 4:38PM (UTC)
turbo-vite-web	⬜️ Ignored (Inspect)		Jan 31, 2023 at 4:38PM (UTC)

ForsakenHarmony · 2023-01-18T14:56:31Z

Current dependencies on/for this PR:

main
- PR add full next.js config #3343
  - PR external node_modules for SSR #3361 👈

This comment was auto-generated by Graphite.

github-actions · 2023-01-20T21:50:44Z

🟢 CI successful 🟢

Thanks

alexkirsz · 2023-01-23T10:02:17Z

crates/turbopack-core/src/resolve/plugin.rs

+}
+
+#[turbo_tasks::value_trait]
+pub trait ResolvePlugin {


Should this be a ResolvePlugin or a straight up Resolver? I'm asking because in

turbo/crates/turbopack-core/src/resolve/mod.rs

Line 576 in 89ead9b

pub async fn resolve(

we already have three clear passes:

ImportMap(Resolver)

Request(Resolver)

FallbackImportMap(Resolver)

And so it seems like this would essentially add intermediate resolvers between 2. and 3.

Not sure if I understand the question, what's a Resolver? Would that be a new type?
To me, the resolve function is the Resolver

The ResolvePlugin right now is kinda part of pass 2, we already have the resolved_map in there.
And it would later on also become part of pass 1

Was thinking the import map could be implemented as a ResolvePlugin (not that we would necessarily want that)

Resolver is just another name for ResolvePlugin. I'm saying the current resolving logic can be separated into 3 ResolvePlugins.

crates/next-core/src/next_server/resolve.rs

crates/turbopack/src/resolve_options_context.rs

crates/turbopack-core/src/resolve/plugin.rs

crates/next-core/src/next_server/resolve.rs

sokra · 2023-01-24T16:00:21Z

crates/next-core/src/next_server/resolve.rs

+        // make sure we have a full package
+        let Some(package_name) = package["name"].as_str() else {
+            return Ok(ResolveResultOptionVc::none());
+        };
+
+        // check if we can resolve the package from the root dir (might be hidden by
+        // pnpm)
+        let FileJsonContent::Content(resolved_package) = &*self
+            .root
+            .join(&format!("node_modules/{}/package.json", package_name))
+            .read_json()
+            .await?
+        else {
+            return Ok(ResolveResultOptionVc::none());
+        };


That's a bit weird and also doesn't support to make packages external that have a inner package.json.

Could we try to resolve the request from the project_path and from the current context (need to passed to plugin) with some special ResolveOptions that match node.js behavior? And if both resolve to the same path and that isn't a ESM we can use that.

We don't get the original request here, so that's hard to do

Are there any cjs modules that have an inner package.json?

crates/turbopack-core/src/resolve/mod.rs

github-actions · 2023-01-31T17:46:27Z

Benchmark for `4b9e04b`

Test	Base	PR	%	Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules	7996.01µs ± 90.44µs	8933.10µs ± 110.25µs	+11.72%	+6.55%
bench_hmr_to_commit/Turbopack RCC/1000 modules	9704.15µs ± 88.64µs	9036.33µs ± 179.35µs	-6.88%	-1.38%
bench_hmr_to_commit/Turbopack RSC/1000 modules	469.11ms ± 2.94ms	508.66ms ± 10.01ms	+8.43%	+2.87%
bench_startup/Turbopack RSC/1000 modules	2429.66ms ± 21.70ms	2374.19ms ± 5.04ms	-2.28%	-0.08%
bench_startup/Turbopack SSR/1000 modules	2026.28ms ± 9.01ms	1992.49ms ± 6.80ms	-1.67%	-0.11%

Click to view full benchmark

Test	Base	PR	%	Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules	7996.01µs ± 90.44µs	8933.10µs ± 110.25µs	+11.72%	+6.55%
bench_hmr_to_commit/Turbopack RCC/1000 modules	9704.15µs ± 88.64µs	9036.33µs ± 179.35µs	-6.88%	-1.38%
bench_hmr_to_commit/Turbopack RSC/1000 modules	469.11ms ± 2.94ms	508.66ms ± 10.01ms	+8.43%	+2.87%
bench_hmr_to_commit/Turbopack SSR/1000 modules	8389.87µs ± 152.98µs	8351.78µs ± 94.65µs	-0.45%
bench_hmr_to_eval/Turbopack CSR/1000 modules	8091.97µs ± 83.18µs	8042.63µs ± 83.30µs	-0.61%
bench_hmr_to_eval/Turbopack RCC/1000 modules	8312.55µs ± 80.86µs	8174.45µs ± 73.11µs	-1.66%
bench_hmr_to_eval/Turbopack SSR/1000 modules	8158.19µs ± 121.67µs	8419.77µs ± 95.07µs	+3.21%
bench_hydration/Turbopack RCC/1000 modules	3698.28ms ± 21.08ms	3646.57ms ± 8.84ms	-1.40%
bench_hydration/Turbopack RSC/1000 modules	3202.06ms ± 5.49ms	3208.25ms ± 11.28ms	+0.19%
bench_hydration/Turbopack SSR/1000 modules	3034.85ms ± 16.24ms	3007.44ms ± 14.51ms	-0.90%
bench_startup/Turbopack CSR/1000 modules	2010.86ms ± 7.31ms	2017.66ms ± 12.75ms	+0.34%
bench_startup/Turbopack RCC/1000 modules	2526.44ms ± 10.51ms	2517.47ms ± 18.97ms	-0.36%
bench_startup/Turbopack RSC/1000 modules	2429.66ms ± 21.70ms	2374.19ms ± 5.04ms	-2.28%	-0.08%
bench_startup/Turbopack SSR/1000 modules	2026.28ms ± 9.01ms	1992.49ms ± 6.80ms	-1.67%	-0.11%

Adds resolve plugins (currently only running `after_resolve`, could add `resolve` later) Had to fix `ResolveResult::merge_alternatives` because it merged two external results into `Unresolvable`

…49736102+kodiakhq[bot]@users.noreply.github.com> # New Features - vercel/turbo#3540 - vercel/turbo#3549 - vercel/turbo#3465 - vercel/turbo#3550 - vercel/turbo#3495 - vercel/turbo#3624 - vercel/turbo#3600 - vercel/turbo#3676 - vercel/turbo#3689 # Fixes - vercel/turbo#3437 - vercel/turbo#3542 - vercel/turbo#3531 - vercel/turbo#3552 - vercel/turbo#3551 - vercel/turbo#3597 - vercel/turbo#3644 - vercel/turbo#3623 - vercel/turbo#3634 - vercel/turbo#3574 - vercel/turbo#3673 - vercel/turbo#3675 - vercel/turbo#3723 - vercel/turbo#3677 - vercel/turbo#3717 - vercel/turbo#3701 # Performance Improvements - vercel/turbo#3361 - vercel/turbo#3619 --------- Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>

Adds resolve plugins (currently only running `after_resolve`, could add `resolve` later) Had to fix `ResolveResult::merge_alternatives` because it merged two external results into `Unresolvable`

ForsakenHarmony mentioned this pull request Jan 18, 2023

add full next.js config #3343

Merged

github-actions bot added the area: turbopack label Jan 18, 2023

vercel bot deployed to Preview – turbo-site January 18, 2023 14:57 View deployment

ForsakenHarmony force-pushed the hrmny/web-395-external-node_modules branch 2 times, most recently from 778d693 to 1e3a19d Compare January 19, 2023 19:26

ForsakenHarmony force-pushed the hrmny/full-next-config branch from 0d30f91 to 7522b81 Compare January 19, 2023 19:32

ForsakenHarmony force-pushed the hrmny/web-395-external-node_modules branch from 1e3a19d to 262d934 Compare January 19, 2023 19:32

vercel bot deployed to Preview – turbo-site January 19, 2023 19:33 View deployment

Base automatically changed from hrmny/full-next-config to main January 19, 2023 20:54

ForsakenHarmony force-pushed the hrmny/web-395-external-node_modules branch from 262d934 to 9c34ee5 Compare January 20, 2023 17:18

vercel bot deployed to Preview – turbo-site January 20, 2023 17:19 View deployment

ForsakenHarmony marked this pull request as ready for review January 20, 2023 17:22

ForsakenHarmony requested a review from a team as a code owner January 20, 2023 17:22

ForsakenHarmony force-pushed the hrmny/web-395-external-node_modules branch 2 times, most recently from 5800784 to ba277c9 Compare January 20, 2023 18:54

alexkirsz reviewed Jan 23, 2023

View reviewed changes

ForsakenHarmony force-pushed the hrmny/web-395-external-node_modules branch from ba277c9 to 6dd5f55 Compare January 26, 2023 14:31

vercel bot deployed to Preview – turbo-site January 26, 2023 14:32 View deployment

alexkirsz approved these changes Jan 26, 2023

View reviewed changes

sokra reviewed Jan 27, 2023

View reviewed changes

ForsakenHarmony force-pushed the hrmny/web-395-external-node_modules branch from 6dd5f55 to 469b938 Compare January 30, 2023 15:48

ForsakenHarmony requested a review from a team as a code owner January 30, 2023 19:39

ForsakenHarmony requested review from nathanhammond and mehulkar January 30, 2023 19:39

github-actions bot added the area: ci label Jan 30, 2023

ForsakenHarmony force-pushed the hrmny/web-395-external-node_modules branch from 48ec02a to 8a99490 Compare January 30, 2023 19:40

vercel bot deployed to Preview – turbo-site January 30, 2023 19:41 View deployment

mehulkar removed their request for review January 31, 2023 01:04

ForsakenHarmony removed the request for review from nathanhammond January 31, 2023 16:24

ForsakenHarmony added 7 commits January 31, 2023 17:30

external node_modules for SSR

aa6c298

WIP resolve plugins

5ef8ac1

it works

071f776

check package.json match

1cec4fe

address comments

f40740c

address comments

122c596

fix AliasMap debug format

e0430b2

ForsakenHarmony force-pushed the hrmny/web-395-external-node_modules branch from 8a99490 to e0430b2 Compare January 31, 2023 16:31

vercel bot deployed to Preview – turbo-site January 31, 2023 16:31 View deployment

don't use stderr yet

1877ae2

ForsakenHarmony removed the request for review from a team January 31, 2023 17:25

ForsakenHarmony merged commit bc9e7c6 into main Feb 1, 2023

ForsakenHarmony deleted the hrmny/web-395-external-node_modules branch February 1, 2023 16:26

jridgewell mentioned this pull request Feb 10, 2023

Update turbopack to 230209.02 vercel/next.js#45776

Merged

DuCanhGH mentioned this pull request Mar 15, 2023

Fix(typedRoutes): fixed files from directories other than pages and app getting into Route vercel/next.js#47184

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

external node_modules for SSR #3361

external node_modules for SSR #3361

ForsakenHarmony commented Jan 18, 2023 •

edited

Loading

vercel bot commented Jan 18, 2023 •

edited

Loading

ForsakenHarmony commented Jan 18, 2023

github-actions bot commented Jan 20, 2023 •

edited

Loading

alexkirsz Jan 23, 2023

ForsakenHarmony Jan 26, 2023 •

edited

Loading

alexkirsz Jan 26, 2023

sokra Jan 24, 2023

ForsakenHarmony Jan 30, 2023

github-actions bot commented Jan 31, 2023

external node_modules for SSR #3361

external node_modules for SSR #3361

Conversation

ForsakenHarmony commented Jan 18, 2023 • edited Loading

vercel bot commented Jan 18, 2023 • edited Loading

ForsakenHarmony commented Jan 18, 2023

github-actions bot commented Jan 20, 2023 • edited Loading

🟢 CI successful 🟢

alexkirsz Jan 23, 2023

Choose a reason for hiding this comment

ForsakenHarmony Jan 26, 2023 • edited Loading

Choose a reason for hiding this comment

alexkirsz Jan 26, 2023

Choose a reason for hiding this comment

sokra Jan 24, 2023

Choose a reason for hiding this comment

ForsakenHarmony Jan 30, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 31, 2023

Benchmark for 4b9e04b

ForsakenHarmony commented Jan 18, 2023 •

edited

Loading

vercel bot commented Jan 18, 2023 •

edited

Loading

github-actions bot commented Jan 20, 2023 •

edited

Loading

ForsakenHarmony Jan 26, 2023 •

edited

Loading

Benchmark for `4b9e04b`