New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Webpack 4 chunkhash/contenthash can vary between builds #7179

Closed
mikeyoon opened this Issue May 2, 2018 · 17 comments

Comments

Projects
None yet
9 participants
@mikeyoon
Copy link

mikeyoon commented May 2, 2018

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
I'm finding that the chunkhash varies between builds in v4. Using the diff utility between the the different files shows no differences. Attempting to narrow the codebase down reduces the frequency of different hashes.

If the current behavior is a bug, please provide the steps to reproduce.
I've been unsuccessful in creating a small test case as it's just too inconsistent and would take such an enormous amount of time for me to figure out by slicing and dicing our codebase.

I'm happy to help as much as I can to find a repro, but I'd need some information about diagnostics I can turn on to figure out what's happening.

What is the expected behavior?
The chunkhash should be consistent, as does in version 3.

If this is a feature request, what is motivation or use case for changing the behavior?

Please mention other relevant information such as the browser version, Node.js version, webpack version, and Operating System.

Webpack 4.6.0, Node 8.9.4, OS X 10.13.4

Here's our webpack.config.js

const path = require('path');
const webpack = require('webpack');
const ProgressBarPlugin = require('progress-bar-webpack-plugin');
const childProcess = require('child_process');
const UglifyJsPlugin = require('uglifyjs-webpack-plugin')
const AssetsPlugin = require('assets-webpack-plugin')
const CleanWebpackPlugin = require('clean-webpack-plugin')
const WebpackBuildNotifierPlugin = require('webpack-build-notifier');

const root = path.resolve(__dirname, './src');
const webappPath = path.resolve(__dirname, '..');
const deployPath = path.resolve(webappPath, './public/assets');

const dev = !process.env.NODE_ENV;
const ci = process.env.NODE_ENV === 'ci';
const prod = process.env.NODE_ENV === 'production';

if (prod) {
  console.log('PRODUCTION MODE ENABLED');
}

// don't hash the filename unless we're building for prod or staging. Hashed filenames interfere with karma tests.
const outputFilename = dev || ci ? '[name].bundle.js' : '[name].[chunkhash].bundle.js';
const excludedMapSources = dev ? [/vendor.*bundle.js/, /manifest.*bundle.js/] : [/manifest.*bundle.js/];

function getPlugins() {
  let plugins = [
    new CleanWebpackPlugin(['public/assets/*'], {
      allowExternal: true,
      root: webappPath
    }),

    new webpack.SourceMapDevToolPlugin({
      filename: "[file].map",
      exclude: excludedMapSources
    }),

    new AssetsPlugin({
      includeManifest: true,
      path: deployPath,
      prettyPrint: true
    })
  ];

  plugins.push(new webpack.NamedModulesPlugin());
  plugins.push(new webpack.optimize.ModuleConcatenationPlugin());

  return plugins;
}

module.exports = {
  context: root,

  mode: prod ? 'production' : 'development',
  devtool: false, // rely on SourceMapDevToolPlugin
  cache: dev,

  stats: {
    chunks: false,
    chunkModules: false,
    entrypoints: false,
    modules: false,
    version: false,
    hash: false,
    timings: false,
  },

  entry: {
    app: './core-app/bootstrap.ts',
    embed: './core-embed/bootstrap.ts'
  },

  watchOptions: {
    aggregateTimeout: 300,
    ignored: /node_modules/
  },

  resolve: {
    extensions: ['.ts', '.tsx', '.js', '.jsx'],
    alias: {
      handsontable: 'handsontable/dist/handsontable.full.js'
    }
  },

  output: {
    filename: outputFilename,
    path: path.resolve(__dirname, '../public/assets'),
  },

  optimization: {
    runtimeChunk: {
      name: "manifest",
    },
    splitChunks: {
      cacheGroups: {
        // default: false,
        commons: {
          test: /[\\/]node_modules[\\/]/,
          name: "vendor",
          chunks: "initial",
          enforce: true
        }
      },
    },
  },

  externals: {
    nvd3: 'window.nv',
    highcharts: "window.Highcharts",
    angular: 'window.angular',
    jquery: 'window.jQuery',
    lodash: 'window._',
    moment: 'window.moment',
    ace: 'window.ace',
    d3: 'window.d3',
    chiliPiper: 'window.ChiliPiper',
  },

  plugins: getPlugins(),

  module: {
    rules: [
      {
        test: /\.ts(x?)$/,
        exclude: [/\.(spec|e2e)\.ts$/],
        use: [{
          loader: 'ts-loader',
          options: {
            experimentalWatchApi: true // seems to work ok atm, but performance improvement varies
          }
        }]
      }, {
        test: /\.(html|css)$/,
        loader: 'raw-loader',
        exclude: /\.async\.(html|css)$/
      }, {
        test: /\.svg$/,
        loader: 'url-loader?' + JSON.stringify({
          name: '[name]_[hash]'
        })
      }
    ]
  }
};
@michael-ciniawsky

This comment has been minimized.

Copy link
Member

michael-ciniawsky commented May 2, 2018

- const outputFilename = dev || ci ? '[name].bundle.js' : '[name].[chunkhash].bundle.js';
+ const outputFilename = dev || ci ? '[name].bundle.js' : '[name].[contenthash].bundle.js';

TLDR: Use [contenthash] instead of [chunkhash] to ensure that every asset (output file) has a stable hash based on it's contents (only).

[chunkhash] isn't very 'stable' as it can change when e.g a module of a chunk is moved for some reason etc etc (It's always hard to tell why and will always be based on how [chunkhash] works (in terms of browser caching)). For predictable long term (browser) caching the contents of the asset are the only reliable hashing source since browsers cache files based on Map { url => content } relations without any knowledge of webpack chunks or the like...

@michael-ciniawsky

This comment has been minimized.

Copy link
Member

michael-ciniawsky commented May 2, 2018

Also don't forget to set the chunkFilename for Code Splitting (Async Chunks)

const output = {
  filename: dev || ci ? '[name].bundle.js' : '[name].[contenthash].bundle.js',
  chunkFilename: dev || ci ? '[name].chunk.js' : '[name].[contenthash].chunk.js'
}
@mikeyoon

This comment has been minimized.

Copy link

mikeyoon commented May 2, 2018

I changed it to contenthash, but I still get different hashes and they have the same content. I even took out the sourcemap output to make sure they are exactly the same.

@mikeyoon mikeyoon changed the title Webpack 4 chunkhash can vary between builds Webpack 4 chunkhash/contenthash can vary between builds May 3, 2018

@viktorlarsson

This comment has been minimized.

Copy link

viktorlarsson commented May 18, 2018

Experiencing the same issue, did you resolve it?

EDIT: This plugin fixed my issue:
https://www.npmjs.com/package/webpack-plugin-hash-output

@evilebottnawi

This comment has been minimized.

Copy link
Member

evilebottnawi commented May 18, 2018

@mikeyoon can you create minimum reproducible test repo?

@mikeyoon

This comment has been minimized.

Copy link

mikeyoon commented May 18, 2018

@evilebottnawi I tried, but wasn't able to narrow it down because I don't know what causes the issue or how the logic works.

@evilebottnawi

This comment has been minimized.

Copy link
Member

evilebottnawi commented May 18, 2018

@mikeyoon can we close issue? if problem again appear we can reopen

@mikeyoon

This comment has been minimized.

Copy link

mikeyoon commented May 18, 2018

The issue has already been closed earlier, although it definitely still happens to me and I have chosen not to upgrade to Webpack 4 because of it. I don't really care too much at this point as v3 still works for us.

@MQuy

This comment has been minimized.

Copy link

MQuy commented May 18, 2018

it happens for me too and I do use contenthash but they are changed per my build 😢 (it seems that chunk's content is different per build)

@jacobwindsor

This comment has been minimized.

Copy link

jacobwindsor commented Jun 14, 2018

I also have this issue with chunkhash but I am seeing a pattern:

  • Only the chunkhash of one "vendor" splitChunk is affected
  • Only the hash is changed and not the file
    • I am using sourcemaps so the file does change but only on the line referencing the sourcemap
  • The hash remains consistent through builds started in the same bash
    • I.e. if I run the build several times from git bash it's all good. Only if I run the build from VS integrated terminal and then from git bash is there a difference.
@ArtemZag

This comment has been minimized.

Copy link

ArtemZag commented Aug 6, 2018

Probably the hash is different because it's based on module IDs that are generated during the build.

@tkh44

This comment has been minimized.

Copy link

tkh44 commented Aug 20, 2018

This section of the docs might be relevant.

... we can see that all three have. This is because each module.id is incremented based on resolving order by
default. Meaning when the order of resolving is changed, the IDs will be changed as well. So, to recap:

The main bundle changed because of its new content.
The vendor bundle changed because its module.id was changed.
And, the manifest bundle changed because it now contains a reference to a new module.
The first and last are expected -- it's the vendor hash we want to fix. Luckily, there are two plugins we can use
to resolve this issue. The first is the NamedModulesPlugin, which will use the path to the module rather than a
numerical identifier. While this plugin is useful during development for more readable output, it does take a bit
longer to run. The second option is the HashedModuleIdsPlugin, which is recommended for production builds:

Now, despite any new local dependencies, our vendor hash should stay consistent between builds:

https://webpack.js.org/guides/caching/#module-identifiers

@mikeyoon

This comment has been minimized.

Copy link

mikeyoon commented Aug 21, 2018

I have HashedModuleIdsPlugin enabled, so there is probably some other factor.

@ArtemZag

This comment has been minimized.

Copy link

ArtemZag commented Aug 22, 2018

@mikeyoon , try also to enable NamedChunksPlugin (otherwise your chunks will get incremental ids as well as the modules)

@mikeyoon

This comment has been minimized.

Copy link

mikeyoon commented Aug 24, 2018

Didn't work with [contenthash] or [chunkhash] unfortunately. There's definitely something with our codebase that's triggering the issue. Not all of our chunks have inconsistent hashes.

@jakub-g

This comment has been minimized.

Copy link

jakub-g commented Sep 28, 2018

FYI
We had the same issue in our builds (though with webpack 3) when using chunkhash and I tracked it down to script-loader.
webpack-contrib/script-loader#56

I think plugins and loaders (raw-loader) are most likely to contribute to the issue.
Right now I'm in the process of migration to webpack 4 and contenthash, will report my findings in the coming weeks.

@jakub-g

This comment has been minimized.

Copy link

jakub-g commented Dec 22, 2018

Update: from what I've noticed in the recent weeks, the issue actually happens also for some new chunks in our project that are not loaded via script-loader. The issue is not script-loader then (probably).

On my machine, for a given git revision, contenthashes are deterministic, but when running a different git revision, some output chunks are often identical, but contenthash is different than a few days before.

It seems that the source of this is some internal webpack global build state, and that contenthash is not a reliable tool for the job, as of webpack4. I will try webpack5 in the coming weeks to see if it improves anything. If not, an alternative would be some hashing tool independent of webpack (after-build), as contenthash is also not taking into account the minification process (webpack-contrib/terser-webpack-plugin#18 (comment)), and I suspect many other webpack plugins/loaders may have the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment