Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add entrypoint tracing #25538

Merged
merged 41 commits into from Aug 16, 2021
Merged

Add entrypoint tracing #25538

merged 41 commits into from Aug 16, 2021

Conversation

ijjk
Copy link
Member

@ijjk ijjk commented May 28, 2021

This adds tracing entrypoints directly after they have have been transpiled to allow us to trace before the webpack runtime has been added to the modules. This should allow for more accurate tracing of entrypoints and allow the trace step to be cached.

Bug

  • Related issues linked using fixes #number
  • Integration tests added

x-ref: #24700
x-ref: #26200
x-ref: #23894
x-ref: #25431
x-ref: #8251

@ijjk

This comment has been minimized.

@ijjk

This comment has been minimized.

@ijjk

This comment has been minimized.

@ijjk ijjk marked this pull request as ready for review Jun 4, 2021
@ijjk

This comment has been minimized.

@ijjk

This comment has been minimized.

}

try {
return compilation.inputFileSystem.readFileSync(path)
Copy link
Member

@sokra sokra Jul 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading the files directly from fs is probably not the best idea.

A loader could have modified the source code after filesystem. Or it could be transpiled from a language nft doesn't understand.

There is a NormalModule.originalSource() function which gives you the source code for a module after loader processing. That's in a language webpack understands, so either javascript, or something else you can ignore. Best check Module.type for javascript/* to only process JS code.

Note that some modules might not end up at all in the output, so best use the list of chunks from createTraceAssets and grab all modules from there, de-duplicate them, analyse them with nft for more references, merge and cache the results per chunk and add them to the analysis data while iterating over the chunks.

Copy link
Member Author

@ijjk ijjk Jul 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we can't use the list of files from createTraceAssets during this hook since we need it to be immediately after transpiling with babel and compilation.entrypoints.values() seems to be empty at this stage. I added some additional tests and gathering the files from the entry's module dependencies seems to gather everything we need currently and allows us to use the dependencies source if available as well.

}
const readlink = (path: string) => {
try {
return compilation.inputFileSystem.readlinkSync(path)
Copy link
Member

@sokra sokra Jul 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sync fs calls might have a performance influence...

Copy link
Member Author

@ijjk ijjk Jul 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems node-file-trace currently relies on sync fs calls, ideally we could get these from the webpack cache so that it's not actually hitting the filesystem for most of these calls

packages/next/taskfile.js Outdated Show resolved Hide resolved
@ijjk ijjk requested review from huozhi, padmaia and styfle as code owners Jul 7, 2021
styfle
styfle previously approved these changes Aug 16, 2021
@ijjk ijjk requested a review from styfle Aug 16, 2021
styfle
styfle approved these changes Aug 16, 2021
@ijjk

This comment has been minimized.

@ijjk ijjk dismissed sokra’s stale review Aug 16, 2021

changes applied

@ijjk
Copy link
Member Author

ijjk commented Aug 16, 2021

Stats from current PR

Default Build (Increase detected ⚠️)
General Overall increase ⚠️
vercel/next.js canary ijjk/next.js add/nft Change
buildDuration 13.6s 13.4s -160ms
buildDurationCached 3.4s 3.4s ⚠️ +59ms
nodeModulesSize 49.1 MB 49.7 MB ⚠️ +591 kB
Page Load Tests Overall increase ✓
vercel/next.js canary ijjk/next.js add/nft Change
/ failed reqs 0 0
/ total time (seconds) 2.676 2.625 -0.05
/ avg req/sec 934.19 952.54 +18.35
/error-in-render failed reqs 0 0
/error-in-render total time (seconds) 1.436 1.435 0
/error-in-render avg req/sec 1741.03 1742.56 +1.53
Client Bundles (main, webpack, commons)
vercel/next.js canary ijjk/next.js add/nft Change
745.HASH.js gzip 179 B 179 B
framework-HASH.js gzip 42.2 kB 42.2 kB
main-HASH.js gzip 23.1 kB 23.1 kB
webpack-HASH.js gzip 1.44 kB 1.44 kB
Overall change 67 kB 67 kB
Legacy Client Bundles (polyfills)
vercel/next.js canary ijjk/next.js add/nft Change
polyfills-a4..dd70.js gzip 31 kB 31 kB
Overall change 31 kB 31 kB
Client Pages
vercel/next.js canary ijjk/next.js add/nft Change
_app-HASH.js gzip 980 B 980 B
_error-HASH.js gzip 194 B 194 B
amp-HASH.js gzip 312 B 312 B
css-HASH.js gzip 329 B 329 B
dynamic-HASH.js gzip 2.64 kB 2.64 kB
head-HASH.js gzip 350 B 350 B
hooks-HASH.js gzip 904 B 904 B
image-HASH.js gzip 4.13 kB 4.13 kB
index-HASH.js gzip 261 B 261 B
link-HASH.js gzip 1.66 kB 1.66 kB
routerDirect..HASH.js gzip 319 B 319 B
script-HASH.js gzip 387 B 387 B
withRouter-HASH.js gzip 320 B 320 B
bb14e60e810b..30f.css gzip 125 B 125 B
Overall change 12.9 kB 12.9 kB
Client Build Manifests
vercel/next.js canary ijjk/next.js add/nft Change
_buildManifest.js gzip 491 B 491 B
Overall change 491 B 491 B
Rendered Page Sizes
vercel/next.js canary ijjk/next.js add/nft Change
index.html gzip 540 B 540 B
link.html gzip 551 B 551 B
withRouter.html gzip 532 B 532 B
Overall change 1.62 kB 1.62 kB

Webpack 4 Mode (Increase detected ⚠️)
General Overall increase ⚠️
vercel/next.js canary ijjk/next.js add/nft Change
buildDuration 10.9s 11s ⚠️ +49ms
buildDurationCached 4.5s 4.5s ⚠️ +19ms
nodeModulesSize 49.1 MB 49.7 MB ⚠️ +591 kB
Page Load Tests Overall increase ✓
vercel/next.js canary ijjk/next.js add/nft Change
/ failed reqs 0 0
/ total time (seconds) 2.657 2.669 ⚠️ +0.01
/ avg req/sec 940.79 936.76 ⚠️ -4.03
/error-in-render failed reqs 0 0
/error-in-render total time (seconds) 1.439 1.427 -0.01
/error-in-render avg req/sec 1737.53 1751.64 +14.11
Client Bundles (main, webpack, commons)
vercel/next.js canary ijjk/next.js add/nft Change
16.HASH.js gzip 186 B 186 B
677f882d2ed8..HASH.js gzip 14 kB 14 kB
framework.HASH.js gzip 41.9 kB 41.9 kB
main-HASH.js gzip 10.6 kB 10.6 kB
webpack-HASH.js gzip 1.19 kB 1.19 kB
Overall change 67.9 kB 67.9 kB
Legacy Client Bundles (polyfills)
vercel/next.js canary ijjk/next.js add/nft Change
polyfills-a4..dd70.js gzip 31 kB 31 kB
Overall change 31 kB 31 kB
Client Pages
vercel/next.js canary ijjk/next.js add/nft Change
_app-HASH.js gzip 965 B 965 B
_error-HASH.js gzip 3.71 kB 3.71 kB
amp-HASH.js gzip 552 B 552 B
css-HASH.js gzip 333 B 333 B
dynamic-HASH.js gzip 2.83 kB 2.83 kB
head-HASH.js gzip 2.97 kB 2.97 kB
hooks-HASH.js gzip 911 B 911 B
index-HASH.js gzip 231 B 231 B
link-HASH.js gzip 1.64 kB 1.64 kB
routerDirect..HASH.js gzip 298 B 298 B
script-HASH.js gzip 2.95 kB 2.95 kB
withRouter-HASH.js gzip 294 B 294 B
e025d2764813..52f.css gzip 125 B 125 B
Overall change 17.8 kB 17.8 kB
Client Build Manifests
vercel/next.js canary ijjk/next.js add/nft Change
_buildManifest.js gzip 498 B 498 B
Overall change 498 B 498 B
Rendered Page Sizes
vercel/next.js canary ijjk/next.js add/nft Change
index.html gzip 584 B 584 B
link.html gzip 596 B 596 B
withRouter.html gzip 578 B 578 B
Overall change 1.76 kB 1.76 kB
Commit: 2997ee2

@kodiakhq kodiakhq bot merged commit 24b09ad into vercel:canary Aug 16, 2021
32 checks passed
@ijjk ijjk deleted the add/nft branch Aug 16, 2021
@timneutkens timneutkens mentioned this pull request Aug 18, 2021
@pomber
Copy link

pomber commented Oct 5, 2021

When I turn nftTrancing on, I'm having this error while SSRing a Next.js page on Vercel: Error: Cannot find module './dist/client/router'

Full error
[GET] /simple-editor/material-palenight
13:59:05:35
2021-10-05T11:59:05.516Z	ea8ad694-cdcb-42b0-8039-525927f4c0ba	ERROR	Error: Cannot find module './dist/client/router'
Require stack:
- /var/task/node_modules/next/router.js
- /var/task/.next/server/pages/[demo]/[theme].js
- /var/task/node_modules/next/dist/server/require.js
- /var/task/node_modules/next/dist/server/load-components.js
- /var/task/node_modules/next/dist/server/api-utils.js
- /var/task/node_modules/next/dist/server/next-server.js
- /var/task/___next_launcher.js
- /var/runtime/UserFunction.js
- /var/runtime/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
    at Function.Module._load (internal/modules/cjs/loader.js:745:27)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at Object.<anonymous> (/var/task/node_modules/next/router.js:1:18)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
    at Module.load (internal/modules/cjs/loader.js:937:32)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Module.require (internal/modules/cjs/loader.js:961:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/var/task/node_modules/next/router.js',
    '/var/task/.next/server/pages/[demo]/[theme].js',
    '/var/task/node_modules/next/dist/server/require.js',
    '/var/task/node_modules/next/dist/server/load-components.js',
    '/var/task/node_modules/next/dist/server/api-utils.js',
    '/var/task/node_modules/next/dist/server/next-server.js',
    '/var/task/___next_launcher.js',
    '/var/runtime/UserFunction.js',
    '/var/runtime/index.js'
  ],
  page: '/simple-editor/material-palenight'
}
RequestId: ea8ad694-cdcb-42b0-8039-525927f4c0ba Error: Runtime exited with error: exit status 1
Runtime.ExitError

Here are the links to the Vercel project I'm testing with nftTrancing and without nftTrancing (not sure how to make those links public). I can try to create a minimal open source repro if you need it.

@ijjk
Copy link
Member Author

ijjk commented Oct 5, 2021

@pomber this has been updated in the latest version of Next.js v11.1.3-canary.43 please update and give it a try! Note: this config has also been renamed to experimental.outputFileTracing in the latest version.

@pomber
Copy link

pomber commented Oct 5, 2021

hmm, tried with "next": "11.1.3-canary.43" and same error.

config:

module.exports = {
  target: "serverless",
  experimental: {
    esmExternals: true,
    outputFileTracing: true,
  },
};

@ijjk
Copy link
Member Author

ijjk commented Oct 5, 2021

@pomber I see, after further investigation this does appear to be a bug in the latest version, I opened a PR here to fix this #29649

@ijjk
Copy link
Member Author

ijjk commented Oct 5, 2021

The above fix has been released in v11.1.3-canary.45 of Next.js, please update and give it a try!

@pomber
Copy link

pomber commented Oct 6, 2021

The above fix has been released in v11.1.3-canary.45 of Next.js, please update and give it a try!

New error: Cannot find module './utils/resolve-rewrites'

[GET] /section/material-palenight
14:00:11:90
2021-10-06T12:00:12.081Z	933c771f-4622-4cf1-b251-0029a0b07e5c	ERROR	Error: Cannot find module './utils/resolve-rewrites'
Require stack:
- /var/task/node_modules/next/dist/shared/lib/router/router.js
- /var/task/node_modules/next/dist/client/router.js
- /var/task/node_modules/next/router.js
- /var/task/.next/server/pages/[demo]/[theme].js
- /var/task/node_modules/next/dist/server/require.js
- /var/task/node_modules/next/dist/server/load-components.js
- /var/task/node_modules/next/dist/server/api-utils.js
- /var/task/node_modules/next/dist/server/next-server.js
- /var/task/___next_launcher.js
- /var/runtime/UserFunction.js
- /var/runtime/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
    at Function.Module._load (internal/modules/cjs/loader.js:745:27)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at Object.<anonymous> (/var/task/node_modules/next/dist/shared/lib/router/router.js:25:47)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
    at Module.load (internal/modules/cjs/loader.js:937:32)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Module.require (internal/modules/cjs/loader.js:961:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/var/task/node_modules/next/dist/shared/lib/router/router.js',
    '/var/task/node_modules/next/dist/client/router.js',
    '/var/task/node_modules/next/router.js',
    '/var/task/.next/server/pages/[demo]/[theme].js',
    '/var/task/node_modules/next/dist/server/require.js',
    '/var/task/node_modules/next/dist/server/load-components.js',
    '/var/task/node_modules/next/dist/server/api-utils.js',
    '/var/task/node_modules/next/dist/server/next-server.js',
    '/var/task/___next_launcher.js',
    '/var/runtime/UserFunction.js',
    '/var/runtime/index.js'
  ],
  page: '/section/material-palenight'
}
RequestId: 933c771f-4622-4cf1-b251-0029a0b07e5c Error: Runtime exited with error: exit status 1
Runtime.ExitError

@ijjk
Copy link
Member Author

ijjk commented Oct 6, 2021

@pomber Ah that error is from rewrites not being used causing us to not trace that file, I opened an additional fix for that case here #29673. In the mean time you can add a random rewrite e.g. { source: '/no-op', destination: '/no-op' } to fix that. Will update here when the above patch is landed.

kodiakhq bot pushed a commit that referenced this pull request Oct 6, 2021
This ensures we don't add the no-op resolve alias when rewrites aren't used for the server resolving since it is still needed while tracing server files. 

## Bug

- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`

Fixes: #25538 (comment)
@ijjk
Copy link
Member Author

ijjk commented Oct 6, 2021

The above is now available in v11.1.3-canary.48 of Next.js

@vercel vercel locked as resolved and limited conversation to collaborators Jan 27, 2022
natew pushed a commit to natew/next.js that referenced this pull request Feb 16, 2022
This ensures we don't add the no-op resolve alias when rewrites aren't used for the server resolving since it is still needed while tracing server files. 

## Bug

- [x] Related issues linked using `fixes #number`
- [x] Integration tests added
- [x] Errors have helpful link attached, see `contributing.md`

Fixes: vercel#25538 (comment)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants