Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

webpack 5 beta feedback #9802

Open
sokra opened this issue Oct 11, 2019 · 198 comments
Labels

Comments

@sokra
Copy link
Member

@sokra sokra commented Oct 11, 2019

馃帀 Thanks for being brave and donating your valuable time to testing unfinished beta software 馃帀

webpack 5 is in beta phase now. This means major changes and features are done. Major breaking changes have been added. Basic backward-compatibility has been added.

We want to use the beta phase to do the following:

  • Let a boarder range of users test the beta version.
  • Find bugs in new features.
  • Find regressions in old features.
  • Find places where a compat-layer can be added to avoid breaking changes.
  • Improve the migration guide and changelog.

We want to reach these goals after the beta phase:

  • Old features are very stable
  • New features are a bit stable
  • Backward-compatibility layer allows most existing plugins/loader to work unmodified (potentially with deprecation warnings)
  • There clear way how to migrate from webpack 4 to 5
  • There are experimental webpack 5 branches for higher-level tools (like angular-cli, vue-cli, create-react-app, ...) (at least 2)

To help with testing you can do the following:

  • Always test with the latest webpack beta version, as problems might already be fixed there
  • When using webpack directly:
    • Follow the migration guide
    • Report or add missing steps in the migration guide
    • Report problems during the migration
    • Report problems with the build after the migration
  • When using a higher-level tool:
    • Check if there is a branch/Pull Request for webpack 5
    • Follow guide there
    • Report problems in the Pull Request
  • Make sure to also report (positive) experience
    • Performance comparison
    • Size comparison
    • DX comparison
  • Try new features
    • Enable persistent caching -> guide
    • Enable Top Level Await
    • Enable the new asset module type
    • Enable Long Term Caching
    • Try to break filesystem watching
  • Trace back deprecation warnings with node --trace-deprecation
    • Help upgrading plugins/loaders
  • Make sure to follow beta releases
    • Retest newer version to avoid regressions in beta versions
  • Consider sponsoring webpack when this version
    • increases your productivity with better build performance
    • increases your productivity with better developer experience
    • increases your consumer happiness with better application performance
    • make you happy in some other way...

Known problems:

Planned breaking changes:

  • devtool options will be more restrictive
  • Internal HMR API for plugins will probably change
  • Disable some webpack-only syntax by default: require.ensure, require.include
  • terser-webpack-plugin will be upgraded
  • cache.store != "pack" will be removed
@sokra sokra pinned this issue Oct 11, 2019
@sokra sokra referenced this issue Oct 11, 2019
15 of 19 tasks complete
@hiroppy hiroppy added the webpack-5 label Oct 11, 2019
@miraage

This comment has been minimized.

Copy link

@miraage miraage commented Oct 11, 2019

Automatic Node.js Polyfills Removed
Please provide us with feedback whether you like or dislike the above mentioned change

Many thanks for that from my side. I can not understand how people could write frontend code without realizing what dependencies they actually use..

@Andarist

This comment has been minimized.

Copy link

@Andarist Andarist commented Oct 11, 2019

Do not edit files in node_modules directly unless you opt-out of this optimization with cache.managedPaths: []

This is generally fine, but it would be really great if we could opt-out of this with some flag passed into the webpack. There are cases when editing node_modules helps in debugging problems in dependencies and editing the config to opt-out of this for a "brief" moment is not really that ergonomic. For context e.g. Jest allows this with --no-cache

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Oct 11, 2019

For context e.g. Jest allows this with --no-cache

You can request that for webpack-cli, which can set cache: false.

@thasmo

This comment has been minimized.

Copy link

@thasmo thasmo commented Oct 11, 2019

Diving into updating our webpack-based tool and I read about setting these for HTTP2:
maxInitialRequests: 30, maxAsyncRequests: 30, maxSize: 100_000
Where do these numbers come from; has this been tested or something? Thanks!

@TchernyavskyDaniil

This comment has been minimized.

Copy link

@TchernyavskyDaniil TchernyavskyDaniil commented Oct 11, 2019

what about options like a --mode production, --env.file and etc. Is that dep?

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Oct 11, 2019

what about options like a --mode production, --env.file and etc. Is that dep?

This is currently unhandled. We need to find a solution for this in the CLI.

@7rulnik

This comment has been minimized.

Copy link

@7rulnik 7rulnik commented Oct 11, 2019

I have a node target and this optimization configuration

 optimization: {
    chunkIds: 'named',
    splitChunks: {
        chunks: 'all'
    },
}

and got this error:

89% chunk assets processingError: Conflict: Multiple chunks emit assets to the same filename server.js (chunks server and vendors-node_modules_alfabank_base-layer_index_js-node_modules_alfabank_container_index_js-no-8997df)
    at /Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Compilation.js:2317:12
    at /Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Cache.js:85:5
    at Hook.eval [as callAsync] (eval at create (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Cache.get (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Cache.js:70:18)
    at /Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Compilation.js:2277:18
    at arrayEach (/Users/7rulnik/projects/alfabank/assr/node_modules/neo-async/async.js:2405:9)
    at Object.each (/Users/7rulnik/projects/alfabank/assr/node_modules/neo-async/async.js:2846:9)
    at /Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Compilation.js:2270:14
    at symbolEach (/Users/7rulnik/projects/alfabank/assr/node_modules/neo-async/async.js:2444:9)
    at Object.each (/Users/7rulnik/projects/alfabank/assr/node_modules/neo-async/async.js:2849:16)
    at Compilation.createChunkAssets (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Compilation.js:2248:12)
    at /Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Compilation.js:1539:10
    at Hook.eval [as callAsync] (eval at create (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/node_modules/tapable/lib/Hook.js:18:14)
    at Compilation.seal (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Compilation.js:1433:27)
    at /Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Compiler.js:770:19
    at /Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Compilation.js:1357:4
    at _next1 (eval at create (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:21:1)
    at eval (eval at create (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:42:1)
    at /Users/7rulnik/projects/alfabank/assr/node_modules/neo-async/async.js:2830:7
    at Object.each (/Users/7rulnik/projects/alfabank/assr/node_modules/neo-async/async.js:2850:39)
    at /Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/FlagDependencyExportsPlugin.js:216:18
    at /Users/7rulnik/projects/alfabank/assr/node_modules/neo-async/async.js:2830:7
    at Object.each (/Users/7rulnik/projects/alfabank/assr/node_modules/neo-async/async.js:2850:39)
    at /Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/FlagDependencyExportsPlugin.js:40:16
    at Hook.eval [as callAsync] (eval at create (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:38:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/node_modules/tapable/lib/Hook.js:18:14)
    at Compilation.finish (/Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Compilation.js:1328:28)
    at /Users/7rulnik/projects/alfabank/assr/node_modules/webpack/lib/Compiler.js:765:18
    at processTicksAndRejections (internal/process/task_queues.js:75:11)
/Users/7rulnik/projects/alfabank/assr/node_modules/.bin/webpack process exited with code 1

Without splitChunks build works as expected

 optimization: {
    chunkIds: 'named',
}
@TchernyavskyDaniil

This comment has been minimized.

Copy link

@TchernyavskyDaniil TchernyavskyDaniil commented Oct 11, 2019

what about options like a --mode production, --env.file and etc. Is that dep?

This is currently unhandled. We need to find a solution for this in the CLI.

Could you explain me another way, add full path to config prod/dev and about env.file ...?

For example:

Build client

./node_modules/webpack/bin/webpack.js --mode production --env.file=production --env.folder=$BUILD_FOLDER

Build server

./node_modules/webpack/bin/webpack.js --env.file=ssr.production --env.folder=$BUILD_FOLDER

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Oct 11, 2019

Where do these numbers come from; has this been tested or something? Thanks!

@thasmo It's an estimated guess. Probably depends on the application.

maxInitialRequests: 30, maxAsyncRequests: 30

These are pretty high limits. I think no normal application will run into these limits at all. I uses 30 instead of Infinity to have some kind of parachute for extreme edgecases. (Infinity has a little performance benefit, so that's probably also fine if you are not at these edgecases)

maxSize: 100_000

The minSize defaults to 30kb. With this config chunks will be between 30kb and 100kb. The algorithm work the way that chunks bigger than maxSize are split into two parts. The splitting point is chosen in a deterministic way based on module names. The split point can't be in the first 30kb and can't be in the last 30kb, so it has about 40kb to move. Smaller maxSizes give it less room to move, which makes it less deterministic, which hurts long term caching. Bigger maxSize makes the chunks bigger, which hurts long term caching. Difficult to find a perfect value, depends on the app and which changes you do. As a rule of thumb I thought maxSize = 3 * minSize could be a good idea.

If someone want to do deeper investigations here, that would be interesting.

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Oct 11, 2019

Multiple chunks emit assets to the same filename server.js

@7rulnik Did you set output.filename: "server.js" or used the default output.filename: "[name].js"

@thasmo

This comment has been minimized.

Copy link

@thasmo thasmo commented Oct 11, 2019

@sokra Thanks a lot for the explanation.

@7rulnik

This comment has been minimized.

Copy link

@7rulnik 7rulnik commented Oct 11, 2019

@sokra yep, it was server.js and entry was server.
I tried [name].js 鈥 now it works. Is it the recommended approach?

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Oct 11, 2019

Could you explain me another way, add full path to config prod/dev and about env.file ...?

@TchernyavskyDaniil I'm not sure if I understood the question correctly. Was this related to Persistent Caching?

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Oct 11, 2019

I tried [name].js 鈥 now it works. Is it the recommended approach?

@7rulnik When chunks are splitted (or multiple entrypoints are used) output.filename affects multiple chunk. You can give multiple chunks the same filename. A constant output.filename won't work in these cases.

@thasmo

This comment has been minimized.

Copy link

@thasmo thasmo commented Oct 11, 2019

About the persistent cache configuration; how would it look like if I add custom/other files?

cache: {
  // 1. Set cache type to filesystem
  type: "filesystem",
  
  buildDependencies: {
    // 2. Add your config as buildDependency to get cache invalidation on config change
    config: [__filename],
  
    // 3. If you have other things the build depends on you can add them here
    // Note that webpack, loaders and all modules referenced from your config are automatically added
    anyKeyHere: ['/absolute/path/'], // ?
  }
}

Would it take a regex/glob too?

In that sense; is there documentation for v5 already somewhere?

@TchernyavskyDaniil

This comment has been minimized.

Copy link

@TchernyavskyDaniil TchernyavskyDaniil commented Oct 11, 2019

Could you explain me another way, add full path to config prod/dev and about env.file ...?

@TchernyavskyDaniil I'm not sure if I understood the question correctly. Was this related to Persistent Caching?

Just a another way to build like that :)

Build client

./node_modules/webpack/bin/webpack.js --mode production --env.file=production --env.folder=$BUILD_FOLDER

Build server

./node_modules/webpack/bin/webpack.js --env.file=ssr.production --env.folder=$BUILD_FOLDER

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Oct 11, 2019

Would it take a regex/glob too?

No, only files: something, or folders: something/. It will be resolved so it could be a node_modules or file without extension.

In that sense; is there documentation for v5 already somewhere?

Not that much. The Changelog has much info. There is a work in progress branch here: https://github.com/webpack/webpack.js.org/tree/next

@7rulnik

This comment has been minimized.

Copy link

@7rulnik 7rulnik commented Oct 11, 2019

@sokra also build time increased by ~25%. I can provide the graphs from ProfilingPlugin. Should I?

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Oct 11, 2019

Should I?

@7rulnik yep. Could you create a new issue with your case + config?

@thasmo

This comment has been minimized.

Copy link

@thasmo thasmo commented Oct 11, 2019

Noticed when using eslint-loader it prints this to the terminal:

  assets\base\scripts\application\components\component.ts:14:18
  鈥   5:24  Unexpected any. Specify a different type.  @typescript-eslint/no-explicit-any
  鈥   6:27  Unexpected any. Specify a different type.  @typescript-eslint/no-explicit-any
  鈥   7:25  Unexpected any. Specify a different type.  @typescript-eslint/no-explicit-any
  鈥  16:47  Unexpected any. Specify a different type.  @typescript-eslint/no-explicit-any
  脳  14:18  Unexpected empty method run.               @typescript-eslint/no-empty-function

  4 warnings
  1 error

    at emitError (C:\Users\user\Projects\project\node_modules\webpack\lib\NormalModule.js:273:6)
    at Linter.printOutput (C:\Users\user\Projects\project\node_modules\eslint-loader\dist\Linter.js:95:5)
    at cache (C:\Users\user\Projects\project\node_modules\eslint-loader\dist\cacheLoader.js:46:14)
    at C:\Users\user\Projects\project\node_modules\loader-fs-cache\index.js:122:24
    at Gunzip.cb (C:\Users\user\Projects\project\node_modules\loader-fs-cache\index.js:47:14)
    at Gunzip.zlibBufferOnEnd (zlib.js:131:10)
    at Gunzip.emit (events.js:203:15)
    at Gunzip.EventEmitter.emit (domain.js:448:20)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

Any idea why the stack is printed as well? The call stack should not be printed actually, I guess - only the linting errors should be printed.

@7rulnik

This comment has been minimized.

Copy link

@7rulnik 7rulnik commented Oct 11, 2019

@sokra issue about build time #9807

@thasmo

This comment has been minimized.

Copy link

@thasmo thasmo commented Oct 12, 2019

Got lots of these logs on the console when using the new filesystem caching:

<w> [webpack.cache.PackFileCacheStrategy] Caching failed for /module/C:\Users\thasmo\Projects\project\node_modules\mini-css-extract-plugin\dist\loader.js!C:\Users\thasmo\Projects\project\node_modules\css-loader\dist\cjs.js??ruleSet[1].rules[0].use[1]!C:\Users\thasmo\Projects\project\node_modules\postcss-loader\src\index.js??ruleSet[1].rules[0].use[2]!C:\Users\thasmo\Projects\project\node_modules\resolve-url-loader\index.js??ruleSet[1].rules[0].use[3]!C:\Users\thasmo\Projects\project\node_modules\sass-loader\dist\cjs.js??ruleSet[1].rules[0].use[4]!C:\Users\thasmo\Projects\project\assets\components\sitemap\styles\base.scss: Error: No serializer registered for CssDependency
<w> We will try to cache this entry again once in a while or when the cache file is deleted.

What do they mean; can I fix/disable them?

@thasmo

This comment has been minimized.

Copy link

@thasmo thasmo commented Oct 12, 2019

Also encountering an issue described in webpack-contrib/stylelint-webpack-plugin#187.

@dreyks

This comment has been minimized.

Copy link

@dreyks dreyks commented Oct 12, 2019

Tried caching option and looks like it works only every other time. Continuously running the build without changing anything: one build is cached another one is not

@thasmo

This comment has been minimized.

Copy link

@thasmo thasmo commented Oct 13, 2019

About persistent caching; can it be used in production and development mode and even with watch, or are there limitations? If not, I'm looking forward enabling it in general.

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

slow-run.zip -> seems like the difference between 9s and 23s is that the slower build serialize the cache file at the end and the fast ones doesn't. The cache is serialized when the cache has been changed.

You can enable the debug logging for the persistent cache with infrastructureLogging: { debug: /PackFileCache/ } and this should result in a message starting with Pack got invalid because of . But this generates a lot of debug output so better pipe it to a file (at least for the initial build).

@lukeapage

This comment has been minimized.

Copy link
Contributor

@lukeapage lukeapage commented Nov 14, 2019

@sokra

<t> [webpack.cache.PackFileCacheStrategy] restore pack: 3183.380133ms
    [webpack.cache.PackFileCacheStrategy] Cached /resolve/normal/path=C:\git\xyz/request=./frontend/libs/polyfills.js to pack.
[webpack.cache.PackFileCacheStrategy] Pack got invalid because of /resolve/normal/path=C:\git\xyz/request=./frontend/libs/polyfills.js

This is our first entry file and definitely is not changed between runs ?!

The file is unremarkable - it is commonjs, has a few requires and uses a defined variable BUILD_IS_DEBUG

@marksamman

This comment has been minimized.

Copy link

@marksamman marksamman commented Nov 14, 2019

I switched from [hash] to [contenthash] for output filenames when migrating to webpack 5 and just noticed that the content hash doesn't change when you use dynamic imports and the content of a dynamically imported file has changed. Is that working as intended? I've switched to [chunkhash] now.

@lukeapage

This comment has been minimized.

Copy link
Contributor

@lukeapage lukeapage commented Nov 14, 2019

I get different failures - sometimes its this - with my enhanced logging:

    [webpack.cache.PackFileCacheStrategy] Cached /asset/chunkentry to pack.
    [webpack.cache.PackFileCacheStrategy] Pack got invalid because of /asset/chunkentry content same false etag same false
    [webpack.cache.PackFileCacheStrategy] 697a1509ef3a5915306b98b0ef3ac430 !== ad752b0da2b2bb49157e6ba54b9aa7ff
    [webpack.cache.PackFileCacheStrategy] string !== string
    [webpack.cache.PackFileCacheStrategy] '() => {
    		if (memorized) {
    			return result;
    		} else {
    			result = fn();
    			memorized = true;
    			// Allow to clean up memory for fn
    			// and all dependent resources
    			fn = undefined;
    			return result;
    		}
    	}' !== '[object Object]'

I'm not sure why its comparing contents but one is an object and one is a memoize function result...

and when it fails onthe first file:

    [webpack.cache.PackFileCacheStrategy] Pack got invalid because of /resolve/normal/path=C:\git\xyz/request=./frontend/libs/polyfills.js content same false etag same true
    [webpack.cache.PackFileCacheStrategy] '[object Object]' !== '[object Object]'

@lukeapage

This comment has been minimized.

Copy link
Contributor

@lukeapage lukeapage commented Nov 14, 2019

comparing the objects using json.stringify they differ by a _deopt property?

image

It seems also that the timings don't depend on the cold run, but randomly occur - sometimes I get a miss because function != object, sometimes I get the entry file contents object not being the same and sometimes I get nothing and a fast build.

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

just noticed that the content hash doesn't change when you use dynamic imports and the content of a dynamically imported file has changed.

This is indeed a bug. Will fix it.

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

We get lots of warnings while using filesystem caching

@AlonMiz Update to beta.5 which fixes the warnings

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

[webpack.cache.PackFileCacheStrategy] Pack got invalid because of /resolve/normal/path=C:\git\xyz/request=./frontend/libs/polyfills.js

@lukeapage That the cache entry of the resolving of these file. You may noticed it carries file/context/missingDependencies and a snapshot of these files. When one of these file timestamps changes or is greater that read start time - 1s the cache entry is invalidated and will re-resolve. The new result and snapshot is stored to the cache.

You should see in the debug logging of webpack.FileSystemInfo when a snapshot is considered invalid:

DEBUG LOG from webpack.FileSystemInfo
C:\git\xyz invalidated because it may have changed

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

[webpack.cache.PackFileCacheStrategy] '() => {
		if (memorized) {
			return result;
		} else {
			result = fn();
			memorized = true;
			// Allow to clean up memory for fn
			// and all dependent resources
			fn = undefined;
			return result;
		}
	}' !== '[object Object]'

That doesn't matter. This is a difference between lazy serialisierend value and eager value.

[webpack.cache.PackFileCacheStrategy] 697a1509ef3a5915306b98b0ef3ac430 !== ad752b0da2b2bb49157e6ba54b9aa7ff

This matters. hash shouldn't differ when files are equal...

The hash used here is the chunk.hash. Not sure what exactly is the problem, but I found one place where something isn't in deterministic order. Do you have an array as entrypoint for this entry entry: { entry: ["a", "b"] }? If yes, could you try to replace it with a single file importing all modules.

(You mentioned the ./frontend/libs/polyfills.js is your entrypoint, so you probably have an array as entrypoint)

@lukeapage

This comment has been minimized.

Copy link
Contributor

@lukeapage lukeapage commented Nov 14, 2019

@sokra in the 2 conditions it writes a new cache

when the chunk is different:

  [webpack.cache.PackFileCacheStrategy] Pack got invalid because of /asset/chunkentry content same false etag same false

I have no FileSystemInfo logs

when the entry is different

    [webpack.cache.PackFileCacheStrategy] Pack got invalid because of /resolve/normal/path=C:\git\xyz/request=./frontend/libs/polyfills.js content same false etag same true
    [webpack.cache.PackFileCacheStrategy] '[object Object]' !== '[object Object]'

I have only:

        C:\git\xyz invalidated because it may have changed (1573732851707) after the start time of the snapshot (1573732142508)

aha could this be because of writing profile info and log files into the project directory? why doesnt this happen every time? or is it if it finishes 1s after writing cache?

I'll try your suggestions and disabling the profiler plugin

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

aha could this be because of writing profile info and log files into the project directory? why doesnt this happen every time? or is it if it finishes 1s after writing cache?

I think I got the bug. We take modified timestamps from directories for the safeTime, but as fileDependencies on directory means a dependency on existance, we should use create time instead. Otherwise cache entry invalidates when a new file/directory is written to the directory.

@lukeapage

This comment has been minimized.

Copy link
Contributor

@lukeapage lukeapage commented Nov 14, 2019

Otherwise cache entry invalidates when a new file/directory is written to the directory.

Could it not be a new file causes a different build? e.g.
if module resolution looks for a.js file first then a/index.js file, if you add a file you can change what it would resolve to?
But I'd definitely like to say project/src directory and node_modules should be the only one monitored and not the whole project

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

These files are separately tracked in missingDependencies. You see that e. g. in your screenshot above: #9802 (comment) where missing package.jsons are tracked. Similar for missing files that have higher precedence.

@lukeapage

This comment has been minimized.

Copy link
Contributor

@lukeapage lukeapage commented Nov 14, 2019

your right, removing profiler output fixes it.

updated metrics with persitent cache:
wp 5 - 35s / 7.5s
wp 4 - 18s / 18s

but wp5, warm with 1 file change - 23s

So currently in normal use the persistent cache doesnt add much

without persistent cache I see an improvement wp4 -> wp5 of 18s -> 16s

@artem-malko

This comment has been minimized.

Copy link

@artem-malko artem-malko commented Nov 14, 2019

Hi @sokra
We are added cache to our CI, but it is not working(

<t> [webpack.cache.PackFileCacheStrategy] restore pack container: 82.242391ms
    [webpack.cache.PackFileCacheStrategy] Restored pack from /app/packages/desktop/.cache/webpack/client.pack, but build dependencies have changed.
<t> [webpack.cache.PackFileCacheStrategy] check build dependencies: 352.571957ms

What does "build dependencies" mean?

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

So currently in normal use the persistent cache doesnt add much

Note that technically cache serialization is not part of the build. So cache is serializated after the build is complete. For a single run this doesn't matter that much, but assuming normal development uses watch mode or webpack-dev-server you can open the application when the build is complete before the cache serialization starts. There are also options to delay cache serialization until a certain idle timeout has been passed. The default serialize the cache instantly after the initial run (if changed happened), but only serialized the cache after 60s idle time after subsequent builds.

Assuming you only make changes to your application while webpack is in watch mode, you should never feel the delay of cache serialization.

We also need to make cache serialization cancel-able to completely avoid any delay because of cache serialization.

On the other side we need to improve cache serialization performance. It takes 2x 10s for your build. I have the feeling this can be faster.

Also note that the serialization time should decrease when cache has be deserialized before, as some cache entries can be kept in serialized form.

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

What does "build dependencies" mean?

The source code that where used to generate the build. Usually webpack + dependencies, all used loaders and you config file + dependencies (like plugins etc.).

All these files are hashed and compared with the cached hash. On mismatch cache is thrown away and you get this message.

You could set infrastructureLogging: { debug: /FileSystemInfo/ } to get more info why.

@artem-malko

This comment has been minimized.

Copy link

@artem-malko artem-malko commented Nov 14, 2019

@sokra thx

@artem-malko

This comment has been minimized.

Copy link

@artem-malko artem-malko commented Nov 14, 2019

@sokra by the way, there can I read about new cache? How it works?

@artem-malko

This comment has been minimized.

Copy link

@artem-malko artem-malko commented Nov 14, 2019

<t> [webpack.cache.PackFileCacheStrategy] restore pack container: 67.05819ms
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/packages/common/src/lib/lazyLoader/shared.ts invalidated because timestamps differ (1573744081000 != 1573742947000)
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/packages/desktop/webpack/universal.ts invalidated because timestamps differ (1573744082000 != 1573742948000)
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/packages/desktop/webpack/client.ts invalidated because timestamps differ (1573744082000 != 1573742948000)
    [webpack.cache.PackFileCacheStrategy] Restored pack from /app/packages/desktop/.cache/webpack/client.pack, but build dependencies have changed.

We build our app in docker container in CI-runner. So, we clone our git repo in runner, copy file to docker-container and then build our app in docker-container.
Is it possible to change cache invalidation mechanism? As I understand, our cache will be invalidated every time, cause of file copying.

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

Is it possible to change cache invalidation mechanism?

Actually it shouldn't work this way. That's why it should use hashes instead of timestamps for build dependencies. But there seem to be a bug here. I'll look into that. Copying files should be fine.

@artem-malko

This comment has been minimized.

Copy link

@artem-malko artem-malko commented Nov 14, 2019

@sokra thx one more time) Will be waiting for results

@MLoughry

This comment has been minimized.

Copy link
Contributor

@MLoughry MLoughry commented Nov 14, 2019

I didn't get a chance to prifle/benchmark beta.4, but here are the results for beta.5:

  • initial build: 385-438s (mean 412s)
  • rebuild: 18.9-35.1s (mean 24.6s)

webpack-5-beta-5-rebuild-no-eager.zip

@AlonMiz

This comment has been minimized.

Copy link

@AlonMiz AlonMiz commented Nov 14, 2019

webpackChunkName

webpackChunkName naming

https://github.com/webpack/changelog-v5#named-chunk-ids
according to that description, i can still use this syntax to specify that bundle name (prod)

const getStripe = () => import(/* webpackChunkName: "react-stripe-elements" */ 'react-stripe-elements);

but, with deterministic modules i get: 206.bundle.c79008efb3bae3e06ff4.js
and with named i get vendors-node_modules_react-stripe-elements_es_index_js.bundle.aaed43553c15f7ac637b.js
the expected outcome from my perspective is react-stripe-elements.app.bundle.c79008efb3bae3e06ff4.js

webpackChunkName being swallowed by cacheGroups - SOLVED

I used to define webpackChunkName and it was never swallowed by a cacheGroup,
now it seems that i cannot control entirely which lazy module will be swallowed by a cacheGroup.
EDIT: I solved the swallowing by reducing all groups priority under 0, and setting minSize to 0. but this config worked prior migrating to webpack 5. so it might be a bug that webpackChunkName don't get high priority for naming and splitting. it feels that if i want to lazy load a module it should NEVER merge it with a sync cacheGroup. @sokra WDYT?

cacheGroup naming

it's a bit cumbersome that I need to specify the name of a cacheGroup twice, one as the cacheGroup, and one as the name field:

this webpack 4 config used to produce: frameworks.app.bundle.d1973d679853f1eeb6fb.js

        frameworks: {
          test: /\/node_modules\/(angular|react|react-dom)\//,
          enforce: true,
          priority: 10,
        },

now in order to get the same output, i have to specify the name, and add the .app suffix :/ so the app suffix is not dynamic as itt used to be (used to name it according to the entry file )

        frameworks: {
         name: 'framework.app'
          test: /\/node_modules\/(angular|react|react-dom)\//,
          enforce: true,
          priority: 10,
        },

LICENSE additional file

I'm getting LICENSE files for modules in cacheGroup and lazy load. haven't seen docs about that, and i wonder how to opt-out of this
eg. vendors-node_modules_lodash_lodash_js.bundle.c0bd0033e3c9d32e432e.js.LICENSE

config

module.exports = {
  context,
  entry: { app: appEntry },
  output: {
    pathinfo: false,
    path: path.join(context, '../dist'),
    filename: '[name].bundle.[contenthash].js',
    chunkFilename: '[name].bundle.[contenthash].js',
  },
  module: {
    rules: [tsRule({ configFileName: tsConfigFilePath, include: [appEntry] }), cssRule, htmlRule, lessRule, staticRule],
  },
  optimization: {
    chunkIds: 'named',
    moduleIds: 'named',
    splitChunks: {
      chunks: 'all',
      minSize: 0,
      automaticNameDelimiter: '.',
      cacheGroups: {
        frameworks: {
         name: 'framework.app'
          test: /\/node_modules\/(angular|react|react-dom)\//,
          enforce: true,
          priority: -5,
        },
        vendor: {
          name: 'vendor.app',
          test: /\/node_modules|3rdparty\//,
          enforce: true,
          priority: -10,
        },
      },
    },
  },
};
@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 14, 2019

so released a 5.0.0-beta.6 which should (hopefully) fix some of your problems

@artem-malko

  • fix logging of FileSystemInfo for multiple different invalidation reasons

Your log messages were incomplete. Please repeat with this version to get the correct logging.

@lukeapage

  • fixes non-deterministic order of entry modules in hashing

Maybe fixes the different etag of /asset/chunkentry

  • fixed snapshotting of directories

Should fix the invalidation of snapshots when files are added to directories and should no longer cause serialization at the end of the build when no change has made

  • move restored data for provided exports in new class

Should made it easier to locate the cost of optimization.providedExports serialization in profiles.

@marksamman

  • include runtime modules in chunk hash

Should fix the [contenthash] issue. I think this has affected [chunkhash] too.

@artem-malko

This comment has been minimized.

Copy link

@artem-malko artem-malko commented Nov 15, 2019

@sokra I've checked beta.6 in our CI.

<t> [webpack.cache.PackFileCacheStrategy] restore pack container: 69.299859ms
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/packages/common/src/lib/lazyLoader/shared.ts invalidated because timestamps differ (1573788706000 != 1573788131000)
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/packages/desktop/webpack/universal.ts invalidated because timestamps differ (1573788707000 != 1573788132000)
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/packages/desktop/webpack/client.ts invalidated because timestamps differ (1573788707000 != 1573788132000)
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/webpack/po-loader.ts invalidated because timestamps differ (1573788707000 != 1573786943000)
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/package.json invalidated because timestamps differ (1573788706000 != 1573786942000)
<t> [webpack.cache.PackFileCacheStrategy] check build dependencies: 290.135245ms
<t> [webpack.cache.PackFileCacheStrategy] restore pack: 2431.625165ms

I have quite strange result, cause file with changes was not invalidated. Other files without changes were invalidated cause of different timestamps.
And it's strange, only some files invalidated cause of timestamp, not all files from my app.

@sokra

This comment has been minimized.

Copy link
Member Author

@sokra sokra commented Nov 15, 2019

I've checked beta.6 in our CI.

<t> [webpack.cache.PackFileCacheStrategy] restore pack container: 69.299859ms
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/packages/common/src/lib/lazyLoader/shared.ts invalidated because timestamps differ (1573788706000 != 1573788131000)
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/packages/desktop/webpack/universal.ts invalidated because timestamps differ (1573788707000 != 1573788132000)
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/packages/desktop/webpack/client.ts invalidated because timestamps differ (1573788707000 != 1573788132000)
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/webpack/po-loader.ts invalidated because timestamps differ (1573788707000 != 1573786943000)
    [webpack.cache.PackFileCacheStrategy/webpack.FileSystemInfo] /app/package.json invalidated because timestamps differ (1573788706000 != 1573786942000)
<t> [webpack.cache.PackFileCacheStrategy] check build dependencies: 290.135245ms
<t> [webpack.cache.PackFileCacheStrategy] restore pack: 2431.625165ms

That seem to be fine. Build dependencies were not considered as changed and cache was reused. There would be a fast path when timestamp match, but it's ok if not. Only hash mismatches are bad for the build dependencies.

But that's only for build dependencies.


The fact that /app/package.json has a different timestamp probably invalidates a lot of resolve cache entries. You can enable the same debug logging for stats: { loggingDebug: /FileSystemInfo/ } to see why cache entries are invalidated. If this is only because files are copied, you could consider to copy files with original timestamps (mtime).

@artem-malko

This comment has been minimized.

Copy link

@artem-malko artem-malko commented Nov 15, 2019

@sokra thx.
I'll check stats logs too.
And I'll try to mount dir with sources to docker container, instead of copying.

@AlonMiz

This comment has been minimized.

Copy link

@AlonMiz AlonMiz commented Nov 16, 2019

Yarn workspace - use case for persistence caching

TL;DR - i want to force watch packages originated from yarn workspaces as they are inside node_modules folder
We are using yarn workspaces for monorepo.
our frontend is under app-frontend/ folder. and our "shared packages" under packages/
together with hoisting, some of my node_modules are in app-frontend/ and some of them are in / (root).
so i've changed managedPaths to look for both node_modules folder.
but, now it treats my yarn packages as node_module and will not apply their changes (somehow, it does recompile, but not applying the changes)

i am looking for a way that both:

  1. will cause node_modules to be cached (without heavy Hashing and timestamping)
  2. will watch for changes in my packages. (eg. everything that starts with @my-repo/)

suggestions:

  1. accept function in managedPaths.
managedPaths: (path) => {
   if (path.includes('@my_repo_prefix/')) return false;
   return path.includes('node_modules';
}
  1. forceInclude - force hashing and watch changes over files although catched by managedPaths
forceInclude: `\@my_repo_prefix\/\`

my persistence caching config looks like that:

  cache: {
    type: 'filesystem',
    name: env.ENV_NAME,
    managedPaths: [
      `${path.resolve(__dirname, '../node_modules')}/`,
      `${path.resolve(__dirname, '../../node_modules')}/`,
    ],
    buildDependencies: {
      config: [`${path.resolve(__dirname)}/`],
    },
  },
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can鈥檛 perform that action at this time.