Duplicate modules - NOT solvable by `npm dedupe` #5593
Comments
|
Why is this a bug? webpack behaves like spec'ed. You installed the package twice, it's included twice. |
|
Thing is, I didn't choose to install the package twice, and there is no way for me to "fix" this at the package manager level, as you've suggested in other issue threads. Wepback used to solve this problem in v1 with |
|
Furthermore, this has the potential to cause sudden bumps in the package size, without much warning. Let's say that I have an app using a popular lib, like Next day, a version Whilst that may not be a bug, and sure, webpack might be behaving like specced, it's not very helpful from a tool that is supposed to help with building optimised versions of application bundles for the web. |
|
You can try |
|
Oh, we're using npm... Plus, looking at the docs for So |
|
@tomhicks-bsf Do you have a suggestion on how we are to handle this with your information in mind? DedupePlugin cannot work with HarmonyModules. Plus it was a pretty hacky plugin to begin with and we will not resurface that kind of work back into webpack. For the time being I'm personally happy to assist you in working with the module author to help PR the sub dependency so that it is at a similar stable version. Otherwise, we are happy to listen to ideas for how we might accomplish handling this. |
|
Would |
|
Thanks very much for the response.
Good to hear that you won’t allow hacks back in :)
I can personally hack around the problem on the consumer end but this feels like it’s something webpack could help with. An IgnorePlugin will not help if we are across major versions, as the APIs are stated to be incompatible.
As for suggestions, I’ll get back to you on this.
Thanks for opening up the discussion!
|
|
It's clear from a few other threads I've read that you don't see vanilla webpack as "the npm bundler" which is fair enough. I don't know much beyond npm in the JS world so I don't know exactly what other problems exist. As far as npm/yarn are concerned, however, incompatible versions of modules should be treated as totally different modules, as though they had different names. This is why the So, my suggestion for this would be to use a plugin or loader that is package-manager aware. NPM and yarn will do their best to choose a single version of a dependency and reference that, but in this case it simply isn't possible. It would require some big changes to that system for it to allow multiple places for "shared" dependencies. One "top-level" place for shared dependencies isn't enough. This means that in order for webpack to map from I can imagine some kind of |
|
I will put together a repo that demonstrates this issue in its simplest form. Give me a day to do this. |
|
@tomhicks-bsf did you ever put together a demo repo, definitely interested in this issue. |
|
Has there been any additional movement on this discussion in another ticket? |
|
Has anyone got anywhere from this? Looking at my webpack stats I have the below stats (truncated to only show > 50kb id, name & size). As you can see, I'm importing immutable js 3 times into webpack. This seems really unnecessary. Does this mean my webpack minimized bundle has 3 immutable.js' in it? Note that all of them are using the same version, hence the identical file size. How can I fix this? Will forking those repos and making them 'peer-dependencies' result in deduplication?
|
|
@terencechow in your case when you have the same version of dependency this can be fixed by removing node_modules directory and installing modules again with npm or yarn command. Both commands will place immutable to /node_modules and this package will not be duplicated in sub-directories. For other cases when we have 3 copies of dependency v1 and 3 copies of v2 I wrote a small plugin |
|
It doesn't look like it was ever provided, so I created a repo that shows a simple example of this issue: https://github.com/franjohn21/dedupe-example FWIW the code being webpacked in that example does not use harmony modules, so ideally webpack 2/3 could maintain parity with webpack 1. We consistently see higher (>10%) bundle sizes in production apps at NerdWallet when migrating from webpack 1 -> webpack 3 mostly due to this issue. |
|
Wouldn't solution the to this issue be rather trivial in nature? In the directory structure @tomhicks-bsf provided, the same library version is installed twice, and while you may disagree whether this should be done, there should be nothing complex about handling this scenario and bundling in the duplicated library only once. All it would take for this to work is for webpack to check for modules' contents to verify whether two (or more) imported modules are in fact the same. My blind guess is that path at which the module is being resolved in currently taken into account, and this is why the same version of a library would be bundled in twice instead of once. Looks like it indeed used to work as indicated by the docs at https://github.com/webpack/docs/wiki/optimization#deduplication |
|
I'd also love some sort of solution for this - I've got several packages with the same dependency and it's being loaded twice.
|
|
I was able to achieve the desired effect described in this issue with some creative use of the My project depends on new NormalModuleReplacementPlugin(/^immutable$/, resource => {
if (resource.context.includes('node_modules/draft-js-plugins-editor')) {
resource.request = 'draft-js/node_modules/immutable';
}
});I'm basically instructing webpack that any call to |
|
We've run in to this issue also, where we need to have two major versions of a library included in our bundle but don't want to have multiple copies beyond those two. In our case, our app was using version 1.x of a large library (let's call it @benthemonkey's solution looks clever, I'll have to give it a shot, but in the end it's a somewhat brittle hack that introduces complexity to the build. Some sort of officially supported solution to this issue (or a well-supported plugin) could be very beneficial. Thanks for sharing your plugin work @RoboBurned, curious if other people have had any luck with it so far? |
|
I've solved it by using yarn workspaces. Does exactly what's expected - dedupe where versions match, but bundle different versions when they don't. |
|
@maciej-gurban that's reasonable in some cases, but in other cases yarn workspaces doesn't resolve this issue. Say you have nine dependencies, three each depending on version 1.x, 2.x, and 3.x of @benthemonkey tried out your solution and it worked for me, though I'm a little scared what will happen when paths/versions of things change or someone else has to discover what's going on haha |
|
I seem to have this issue when using |
|
Was any official resolution to this issue ever suggested? |
|
we've been using this in our react-native project: https://github.com/tradle/dedupe-deps |
|
The least disruptive solution for me has been to take advantage of Webpack's I understand that Webpack's What makes the problem even worse, though, is the difficulty in its identification. So to really know this is occurring, you need to periodically inspect your stats.json with a tool like https://webpack.github.io/analyse/, sort by file size, and manually inspect. Since |
|
Any update on this one? We're seeing the same behavior and it's resulting in 600kb of duplicated code which is too much to be acceptable in our case |
|
Having the same problem, got a mono repo with multiple packages, all packages depend on one core package which contains most of the dependencies. However after building every entry point as well as the vendor chunk contains the chunk that was supposed to be split, rendering splitChunks useless. I would say this is a huge bug since the only real reason to split chunks is caching... I want to split a dependency from the rest of my bundle so it doesn't have to be redownloaded every time my bundle changes. However instead of reducing this problem webpack is now actually increasing it... Result should be: Reality is: Then there are 2 projects, project 1 contains app1 and app2, project 2 contains app3. Now when I'm download project 1 in the browser I am actually including (and therefore downloading) dep.js FOUR times instead of one. Twice in vendor.js, once in app1.js, once in app2.js. When I download project 2 it includes dep.js THREE times, twice in vendor.js, once in app3.js. Seems to me that this is a bug; in no way or form should webpack ever increase bundle size compared to the source (aside from boilerplating and such which is negligible). With this kind of behavior it actually makes no sense to use splitChunks because it's actually increasing my bundle and chunk sizes. I am inclined to remove it entirely and find some other solution. The only possibility I can still imagine is by splitting every single package from node_modules to a seperate file with the name and version of the package.. FYI: In my case these are the exact same module (only one version exists in the filesystem), only difference is that they are shared by multiple packages and therefore symlinked to those packages. |
|
It actually seems to be NPM bug, not Webpack's. |
Webpack relies on package managers to do module hoisting and doesn't have any deduping logic since version 4. However relaying on package manager has a number of short comings, such as when having the same library with the same version laid out in different parts of the node_modules tree. Example: ``` /node_modules/tslib@2.0.0 /node_modules/library-1/node_modules/tslib@1.0.0 /node_modules/library-2/node_modules/tslib@1.0.0 ``` In the above case, in the final bundle we'll end up with 3 versions of tslib instead of 2, even though 2 of the modules are identical. Webpack has an open issue for this webpack/webpack#5593 (Duplicate modules - NOT solvable by `npm dedupe`) With this change we add a custom resolve plugin that dedupes modules with the same name and versions that are laid out in different parts of the node_modules tree.
Webpack relies on package managers to do module hoisting and doesn't have any deduping logic since version 4. However relaying on package manager has a number of short comings, such as when having the same library with the same version laid out in different parts of the node_modules tree. Example: ``` /node_modules/tslib@2.0.0 /node_modules/library-1/node_modules/tslib@1.0.0 /node_modules/library-2/node_modules/tslib@1.0.0 ``` In the above case, in the final bundle we'll end up with 3 versions of tslib instead of 2, even though 2 of the modules are identical. Webpack has an open issue for this webpack/webpack#5593 (Duplicate modules - NOT solvable by `npm dedupe`) With this change we add a custom resolve plugin that dedupes modules with the same name and versions that are laid out in different parts of the node_modules tree.
Webpack relies on package managers to do module hoisting and doesn't have any deduping logic since version 4. However relaying on package manager has a number of short comings, such as when having the same library with the same version laid out in different parts of the node_modules tree. Example: ``` /node_modules/tslib@2.0.0 /node_modules/library-1/node_modules/tslib@1.0.0 /node_modules/library-2/node_modules/tslib@1.0.0 ``` In the above case, in the final bundle we'll end up with 3 versions of tslib instead of 2, even though 2 of the modules are identical. Webpack has an open issue for this webpack/webpack#5593 (Duplicate modules - NOT solvable by `npm dedupe`) With this change we add a custom resolve plugin that dedupes modules with the same name and versions that are laid out in different parts of the node_modules tree.
Webpack relies on package managers to do module hoisting and doesn't have any deduping logic since version 4. However relaying on package manager has a number of short comings, such as when having the same library with the same version laid out in different parts of the node_modules tree. Example: ``` /node_modules/tslib@2.0.0 /node_modules/library-1/node_modules/tslib@1.0.0 /node_modules/library-2/node_modules/tslib@1.0.0 ``` In the above case, in the final bundle we'll end up with 3 versions of tslib instead of 2, even though 2 of the modules are identical. Webpack has an open issue for this webpack/webpack#5593 (Duplicate modules - NOT solvable by `npm dedupe`) With this change we add a custom resolve plugin that dedupes modules with the same name and versions that are laid out in different parts of the node_modules tree.
Webpack relies on package managers to do module hoisting and doesn't have any deduping logic since version 4. However relaying on package manager has a number of short comings, such as when having the same library with the same version laid out in different parts of the node_modules tree. Example: ``` /node_modules/tslib@2.0.0 /node_modules/library-1/node_modules/tslib@1.0.0 /node_modules/library-2/node_modules/tslib@1.0.0 ``` In the above case, in the final bundle we'll end up with 3 versions of tslib instead of 2, even though 2 of the modules are identical. Webpack has an open issue for this webpack/webpack#5593 (Duplicate modules - NOT solvable by `npm dedupe`) With this change we add a custom resolve plugin that dedupes modules with the same name and versions that are laid out in different parts of the node_modules tree.
|
I had to use |
|
Are there any updates on this issue? Given webpack is about building optimized bundles it boggles the mind removing duplicate code is not part of that process. We are suffering the same problems expressed here. The hashing approach seems like a nice way to resolve this but would need to be performant. @TheLarkInn does webpack do any other dedupe process? |
|
duplicate of #985 |
and others
Do you want to request a feature or report a bug?
Bug
What is the current behavior?
Two identical sub-dependencies are being included twice in the build.
If the current behavior is a bug, please provide the steps to reproduce.
projectAdepends onlibA@^2.0.0,depAanddepB.depAanddepBboth depend onlibA@^1.0.0This results in a directory structure like this:
When the project is built, included in the bundle are:
What is the expected behavior?
The included modules should be:
Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.
webpack@3.5.5I have run
npm dedupebut it can't fix the problem. This is understandable, because npm doesn't have anywhere to "put" this lower-version dependency that is shared by bothdepAanddepB. Previously I believe this was solved with theDedupePluginwhich has since been turned into a no-op.The text was updated successfully, but these errors were encountered: