Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Webpack 4 chunkhash/contenthash can vary between builds #7179

Closed
mikeyoon opened this issue May 2, 2018 · 28 comments
Closed

Webpack 4 chunkhash/contenthash can vary between builds #7179

mikeyoon opened this issue May 2, 2018 · 28 comments

Comments

@mikeyoon
Copy link

@mikeyoon mikeyoon commented May 2, 2018

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
I'm finding that the chunkhash varies between builds in v4. Using the diff utility between the the different files shows no differences. Attempting to narrow the codebase down reduces the frequency of different hashes.

If the current behavior is a bug, please provide the steps to reproduce.
I've been unsuccessful in creating a small test case as it's just too inconsistent and would take such an enormous amount of time for me to figure out by slicing and dicing our codebase.

I'm happy to help as much as I can to find a repro, but I'd need some information about diagnostics I can turn on to figure out what's happening.

What is the expected behavior?
The chunkhash should be consistent, as does in version 3.

If this is a feature request, what is motivation or use case for changing the behavior?

Please mention other relevant information such as the browser version, Node.js version, webpack version, and Operating System.

Webpack 4.6.0, Node 8.9.4, OS X 10.13.4

Here's our webpack.config.js

const path = require('path');
const webpack = require('webpack');
const ProgressBarPlugin = require('progress-bar-webpack-plugin');
const childProcess = require('child_process');
const UglifyJsPlugin = require('uglifyjs-webpack-plugin')
const AssetsPlugin = require('assets-webpack-plugin')
const CleanWebpackPlugin = require('clean-webpack-plugin')
const WebpackBuildNotifierPlugin = require('webpack-build-notifier');

const root = path.resolve(__dirname, './src');
const webappPath = path.resolve(__dirname, '..');
const deployPath = path.resolve(webappPath, './public/assets');

const dev = !process.env.NODE_ENV;
const ci = process.env.NODE_ENV === 'ci';
const prod = process.env.NODE_ENV === 'production';

if (prod) {
  console.log('PRODUCTION MODE ENABLED');
}

// don't hash the filename unless we're building for prod or staging. Hashed filenames interfere with karma tests.
const outputFilename = dev || ci ? '[name].bundle.js' : '[name].[chunkhash].bundle.js';
const excludedMapSources = dev ? [/vendor.*bundle.js/, /manifest.*bundle.js/] : [/manifest.*bundle.js/];

function getPlugins() {
  let plugins = [
    new CleanWebpackPlugin(['public/assets/*'], {
      allowExternal: true,
      root: webappPath
    }),

    new webpack.SourceMapDevToolPlugin({
      filename: "[file].map",
      exclude: excludedMapSources
    }),

    new AssetsPlugin({
      includeManifest: true,
      path: deployPath,
      prettyPrint: true
    })
  ];

  plugins.push(new webpack.NamedModulesPlugin());
  plugins.push(new webpack.optimize.ModuleConcatenationPlugin());

  return plugins;
}

module.exports = {
  context: root,

  mode: prod ? 'production' : 'development',
  devtool: false, // rely on SourceMapDevToolPlugin
  cache: dev,

  stats: {
    chunks: false,
    chunkModules: false,
    entrypoints: false,
    modules: false,
    version: false,
    hash: false,
    timings: false,
  },

  entry: {
    app: './core-app/bootstrap.ts',
    embed: './core-embed/bootstrap.ts'
  },

  watchOptions: {
    aggregateTimeout: 300,
    ignored: /node_modules/
  },

  resolve: {
    extensions: ['.ts', '.tsx', '.js', '.jsx'],
    alias: {
      handsontable: 'handsontable/dist/handsontable.full.js'
    }
  },

  output: {
    filename: outputFilename,
    path: path.resolve(__dirname, '../public/assets'),
  },

  optimization: {
    runtimeChunk: {
      name: "manifest",
    },
    splitChunks: {
      cacheGroups: {
        // default: false,
        commons: {
          test: /[\\/]node_modules[\\/]/,
          name: "vendor",
          chunks: "initial",
          enforce: true
        }
      },
    },
  },

  externals: {
    nvd3: 'window.nv',
    highcharts: "window.Highcharts",
    angular: 'window.angular',
    jquery: 'window.jQuery',
    lodash: 'window._',
    moment: 'window.moment',
    ace: 'window.ace',
    d3: 'window.d3',
    chiliPiper: 'window.ChiliPiper',
  },

  plugins: getPlugins(),

  module: {
    rules: [
      {
        test: /\.ts(x?)$/,
        exclude: [/\.(spec|e2e)\.ts$/],
        use: [{
          loader: 'ts-loader',
          options: {
            experimentalWatchApi: true // seems to work ok atm, but performance improvement varies
          }
        }]
      }, {
        test: /\.(html|css)$/,
        loader: 'raw-loader',
        exclude: /\.async\.(html|css)$/
      }, {
        test: /\.svg$/,
        loader: 'url-loader?' + JSON.stringify({
          name: '[name]_[hash]'
        })
      }
    ]
  }
};
@michael-ciniawsky
Copy link
Member

@michael-ciniawsky michael-ciniawsky commented May 2, 2018

- const outputFilename = dev || ci ? '[name].bundle.js' : '[name].[chunkhash].bundle.js';
+ const outputFilename = dev || ci ? '[name].bundle.js' : '[name].[contenthash].bundle.js';

TLDR: Use [contenthash] instead of [chunkhash] to ensure that every asset (output file) has a stable hash based on it's contents (only).

[chunkhash] isn't very 'stable' as it can change when e.g a module of a chunk is moved for some reason etc etc (It's always hard to tell why and will always be based on how [chunkhash] works (in terms of browser caching)). For predictable long term (browser) caching the contents of the asset are the only reliable hashing source since browsers cache files based on Map { url => content } relations without any knowledge of webpack chunks or the like...

@michael-ciniawsky
Copy link
Member

@michael-ciniawsky michael-ciniawsky commented May 2, 2018

Also don't forget to set the chunkFilename for Code Splitting (Async Chunks)

const output = {
  filename: dev || ci ? '[name].bundle.js' : '[name].[contenthash].bundle.js',
  chunkFilename: dev || ci ? '[name].chunk.js' : '[name].[contenthash].chunk.js'
}
@mikeyoon
Copy link
Author

@mikeyoon mikeyoon commented May 2, 2018

I changed it to contenthash, but I still get different hashes and they have the same content. I even took out the sourcemap output to make sure they are exactly the same.

@mikeyoon mikeyoon changed the title Webpack 4 chunkhash can vary between builds Webpack 4 chunkhash/contenthash can vary between builds May 3, 2018
@viktorlarsson
Copy link

@viktorlarsson viktorlarsson commented May 18, 2018

Experiencing the same issue, did you resolve it?

EDIT: This plugin fixed my issue:
https://www.npmjs.com/package/webpack-plugin-hash-output

@alexander-akait
Copy link
Member

@alexander-akait alexander-akait commented May 18, 2018

@mikeyoon can you create minimum reproducible test repo?

@mikeyoon
Copy link
Author

@mikeyoon mikeyoon commented May 18, 2018

@evilebottnawi I tried, but wasn't able to narrow it down because I don't know what causes the issue or how the logic works.

@alexander-akait
Copy link
Member

@alexander-akait alexander-akait commented May 18, 2018

@mikeyoon can we close issue? if problem again appear we can reopen

@mikeyoon
Copy link
Author

@mikeyoon mikeyoon commented May 18, 2018

The issue has already been closed earlier, although it definitely still happens to me and I have chosen not to upgrade to Webpack 4 because of it. I don't really care too much at this point as v3 still works for us.

@MQuy
Copy link

@MQuy MQuy commented May 18, 2018

it happens for me too and I do use contenthash but they are changed per my build 😢 (it seems that chunk's content is different per build)

@jacobwindsor
Copy link

@jacobwindsor jacobwindsor commented Jun 14, 2018

I also have this issue with chunkhash but I am seeing a pattern:

  • Only the chunkhash of one "vendor" splitChunk is affected
  • Only the hash is changed and not the file
    • I am using sourcemaps so the file does change but only on the line referencing the sourcemap
  • The hash remains consistent through builds started in the same bash
    • I.e. if I run the build several times from git bash it's all good. Only if I run the build from VS integrated terminal and then from git bash is there a difference.
@ArtemZag
Copy link

@ArtemZag ArtemZag commented Aug 6, 2018

Probably the hash is different because it's based on module IDs that are generated during the build.

@tkh44
Copy link

@tkh44 tkh44 commented Aug 20, 2018

This section of the docs might be relevant.

... we can see that all three have. This is because each module.id is incremented based on resolving order by
default. Meaning when the order of resolving is changed, the IDs will be changed as well. So, to recap:

The main bundle changed because of its new content.
The vendor bundle changed because its module.id was changed.
And, the manifest bundle changed because it now contains a reference to a new module.
The first and last are expected -- it's the vendor hash we want to fix. Luckily, there are two plugins we can use
to resolve this issue. The first is the NamedModulesPlugin, which will use the path to the module rather than a
numerical identifier. While this plugin is useful during development for more readable output, it does take a bit
longer to run. The second option is the HashedModuleIdsPlugin, which is recommended for production builds:

Now, despite any new local dependencies, our vendor hash should stay consistent between builds:

https://webpack.js.org/guides/caching/#module-identifiers

@mikeyoon
Copy link
Author

@mikeyoon mikeyoon commented Aug 21, 2018

I have HashedModuleIdsPlugin enabled, so there is probably some other factor.

@ArtemZag
Copy link

@ArtemZag ArtemZag commented Aug 22, 2018

@mikeyoon , try also to enable NamedChunksPlugin (otherwise your chunks will get incremental ids as well as the modules)

@mikeyoon
Copy link
Author

@mikeyoon mikeyoon commented Aug 24, 2018

Didn't work with [contenthash] or [chunkhash] unfortunately. There's definitely something with our codebase that's triggering the issue. Not all of our chunks have inconsistent hashes.

@jakub-g
Copy link

@jakub-g jakub-g commented Sep 28, 2018

FYI
We had the same issue in our builds (though with webpack 3) when using chunkhash and I tracked it down to script-loader.
webpack-contrib/script-loader#56

I think plugins and loaders (raw-loader) are most likely to contribute to the issue.
Right now I'm in the process of migration to webpack 4 and contenthash, will report my findings in the coming weeks.

@jakub-g
Copy link

@jakub-g jakub-g commented Dec 22, 2018

Update: from what I've noticed in the recent weeks, the issue actually happens also for some new chunks in our project that are not loaded via script-loader. The issue is not script-loader then (probably).

On my machine, for a given git revision, contenthashes are deterministic, but when running a different git revision, some output chunks are often identical, but contenthash is different than a few days before.

It seems that the source of this is some internal webpack global build state, and that contenthash is not a reliable tool for the job, as of webpack4. I will try webpack5 in the coming weeks to see if it improves anything. If not, an alternative would be some hashing tool independent of webpack (after-build), as contenthash is also not taking into account the minification process (webpack-contrib/terser-webpack-plugin#18 (comment)), and I suspect many other webpack plugins/loaders may have the same issue.

@onlybye
Copy link

@onlybye onlybye commented Mar 28, 2019

Didn't work with [contenthash] or [chunkhash] unfortunately. There's definitely something with our codebase that's triggering the issue. Not all of our chunks have inconsistent hashes.

@mikeyoon have you solved this problem ?

@alexander-akait
Copy link
Member

@alexander-akait alexander-akait commented Mar 28, 2019

Please create minimum reproducible test repo

@mikeyoon
Copy link
Author

@mikeyoon mikeyoon commented Mar 30, 2019

I have not looked at the issue since I posted about it last time. I also haven't had a chance to try out the latest major release either. We're still just using v3.x for now.

@advl
Copy link

@advl advl commented May 13, 2019

Having the same issue on webpack 4, both with contenthash and chunkhash. If I find some time I will make a minimum repo. Shall the ticket be reopened ?

tatsuyafw added a commit to tatsuyafw/webpacker that referenced this issue May 16, 2019
As mentioned in the below link, [chunkhash] is unstable.

webpack/webpack#7179 (comment)

To ensure that every bundled files have consistent hash names based on it's contents,
use [contenthash] instead of [chunkhash].
gauravtiwari added a commit to rails/webpacker that referenced this issue May 22, 2019
As mentioned in the below link, [chunkhash] is unstable.

webpack/webpack#7179 (comment)

To ensure that every bundled files have consistent hash names based on it's contents,
use [contenthash] instead of [chunkhash].
@halfnibble
Copy link

@halfnibble halfnibble commented Jun 27, 2019

We are seeing this problem too. Unfortunately, it's inconsistent and the codebase is private. Could it have something to do with running 10+ webpack commands simultaneously in different threads? (We are building a mono repo managed by rush.

My assigned task is to determine why this is inconsistent, and to fix it, so any help you can provide would be much appreciated. Thanks.

@vcraescu
Copy link

@vcraescu vcraescu commented Oct 29, 2019

Same issue here. We do async imports and because of this we can't run the app on multiple machines because the code is built separately on each machine so generated files names are not the same on all the machines.

@halfnibble
Copy link

@halfnibble halfnibble commented Oct 29, 2019

I should have come back and posted an update.

Short answer, Webpack's hashing algorithm is deterministic. But the content being sent to it may not be. I discovered a whole slew of issues:

  1. Some npm packages create absolute paths in postInstall scripts.
  2. Some packages' error messages get sent to the hashing algorithm containing an absolute path.
  3. And we had one webpack loader generating content containing absolute paths.

I found the easiest way to debug this issue was to override the webpack hashing algorithm and add a little script to detect for my local absolute path. Here is approximately what I used:
https://gist.github.com/halfnibble/c9f0c3b8d61e7002c799b5d24ae05643

And then in webpack config:

{
  ...
  output: {
    hashFunction: HashFunction,
    ...
  }
}
@alexander-akait
Copy link
Member

@alexander-akait alexander-akait commented Nov 5, 2019

@halfnibble can you create minimum reproducible test repo, maybe we can fix some cases on webpack side

@halfnibble
Copy link

@halfnibble halfnibble commented Nov 5, 2019

@evilebottnawi As far as I can tell, there is absolutely nothing wrong with webpack's hashing algorithm. The problem lies in certain npm packages and webpack loaders. None of the problems I encountered were in code from the webpack organization.

I don't know if it would be worthwhile to create a content hash debug feature like the one we developed to pin point the packages and loaders causing problems.

@alexander-akait
Copy link
Member

@alexander-akait alexander-akait commented Nov 5, 2019

@halfnibble thanks for feedback, yep, some loaders/plugins emit assets with absolute path, better open issues in repos with loaders/plugins

@jakub-g
Copy link

@jakub-g jakub-g commented Nov 6, 2019

@halfnibble re: #7179 (comment)

fantastic comment, thanks to it I revisited the topic and found that it was indeed script-loader that was breaking hashing in our project (due to the usage of path.join and require.resolve which both output absolute paths). I migrated away from it to raw-loader instead, and submitted the PR to deprecate script-loader.

The info about not using absolute paths is documented in "writing a loader" doc:

https://webpack.js.org/contribute/writing-a-loader/#absolute-paths

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet