Webpack 5 persistent cache is not usable in CI environments

[trappar]

Bug report

What is the current behavior?

In CI, a common paradigm is to store cachable data between runs to cut down on execution time. The webpack 5 persistent cache seems like a prime target for such caching. Currently, webpack fails after recalling the persistent cache when run this way.

The configuration used to produce the following error output works both outside CI AND within CI as long as the cache is not restored from a previous run. I've already found issues which claim to fix the errors which appear in the following output such as:

• The "path" argument must be of type string. Received type undefined #12616
• Webpack cache not compatible with mini-css-extract-plugin #12458
• lost [path] in key when boot with webpack 5 persistent cache webdeveric/webpack-assets-manifest#131

I do not believe these issues to be relevant because this issue happens exclusively when a persistent cache is moved from one CI run to another.

Here are the different scenarios where my configuration works/fails:

• Running a build locally with no cache - Works
• Running a build locally with a persistent cache - Works
• Running a build in CI with no cache - Works
• Running a build in CI with a persistent cache generated on the same CI node - Works
• Running a build in CI with a persistent cache generated on a different CI execution - Fails with the following output:

$ NODE_OPTIONS=--max_old_space_size=3072 NODE_ENV=production webpack
<w> [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/home/********/********/client/node_modules/mini-css-extract-plugin/dist/loader.js!/home/********/********/client/node_modules/css-loader/dist/cjs.js!/home/********/********/client/node_modules/postcss-loader/dist/cjs.js??ruleSet[1].rules[5].use[2]!/home/********/********/client/node_modules/sass-loader/dist/cjs.js??ruleSet[1].rules[5].use[3]!/home/********/********/client/src/theme/base.scss': No serializer registered for HookWebpackError
<w> while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> HookWebpackError
<w> [webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/home/********/********/client/node_modules/mini-css-extract-plugin/dist/loader.js!/home/********/********/client/node_modules/css-loader/dist/cjs.js!/home/********/********/client/node_modules/postcss-loader/dist/cjs.js??ruleSet[1].rules[5].use[2]!/home/********/********/client/node_modules/sass-loader/dist/cjs.js??ruleSet[1].rules[5].use[3]!/home/********/********/client/src/features/help/HelpView/HelpView.scss': No serializer registered for HookWebpackError
<w> while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> webpack/lib/ModuleBuildError -> HookWebpackError
[webpack-cli] HookWebpackError: The "path" argument must be of type string. Received undefined
    at makeWebpackError (/home/********/********/client/node_modules/webpack/lib/HookWebpackError.js:48:9)
    at /home/********/********/client/node_modules/webpack/lib/Compilation.js:2515:12
    at eval (eval at create (/home/********/********/client/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:13:1)
    at fn (/home/********/********/client/node_modules/webpack/lib/Compilation.js:452:17)
    at _next0 (eval at create (/home/********/********/client/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:11:1)
    at eval (eval at create (/home/********/********/client/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:26:1)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
-- inner error --
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at validateString (internal/validators.js:124:11)
    at Object.dirname (path.js:1248:5)
    at WebpackAssetsManifest.handleProcessAssetsAnalyse (/home/********/********/client/node_modules/webpack-assets-manifest/src/WebpackAssetsManifest.js:477:31)
    at fn (/home/********/********/client/node_modules/webpack/lib/Compilation.js:450:10)
    at _next0 (eval at create (/home/********/********/client/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:11:1)
    at eval (eval at create (/home/********/********/client/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:26:1)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
caused by plugins in Compilation.hooks.processAssets
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at validateString (internal/validators.js:124:11)
    at Object.dirname (path.js:1248:5)
    at WebpackAssetsManifest.handleProcessAssetsAnalyse (/home/********/********/client/node_modules/webpack-assets-manifest/src/WebpackAssetsManifest.js:477:31)
    at fn (/home/********/********/client/node_modules/webpack/lib/Compilation.js:450:10)
    at _next0 (eval at create (/home/********/********/client/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:11:1)
    at eval (eval at create (/home/********/********/client/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:26:1)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)

Reproduction instructions

We are using a CircleCI configuration like the following with the Docker executor:

- restore_cache:
    keys:
      - webpack-persistent-v1-{{ .Branch }}
      - webpack-persistent-v1-
- run:
    command: webpack
- save_cache:
    key: webpack-persistent-v1-{{ .Branch }}
    paths:
      - node_modules/.cache

I suspect that it might be possible to replicate this issue by creating a docker container, running a build, and then copying the cache files out of that build and into a new one. I haven't got the time to create such a replication repo.

What is the expected behavior?

I would expect that if a persistent cache were generated in an environment then copied/moved to an identical environment, that would not introduce errors into subsequent webpack builds.

Other relevant information:
webpack version: 5.50.0
Node.js version: v14.17.4
Operating System: Locally Pop!_OS, CI is using node:14-buster as its base docker image

[alexander-akait]

Based on your output bug in webpack-assets-manifest, don't forget we store absolute paths in cache (#10400), so moving/copied can doesn't work, anyway please provide example how we can reproduce it, there can be many problems here, and without additional information we cannot help

[trappar]

Does webpack cache work fine without webpack-assets-manifest?

Without the webpack-assets-manifest plugin, I do not receive an error.

There is no CI log that would be helpful, I'm only able to tell that it's not utilizing the webpack cache because in this project I know the amount of time it takes in different situations. I've seen:

• With no caches: 1m30s
• With babel cache: 1m
• With webpack filesystem cache: 12s

So when I removed webpack-assets-manifest the build didn't crash, but it took 1 minute to run, so the webpack cache was not being used. I tried re-running the job with SSH and found that both the webpack and babel cache files were present.

I then tried running the build again, and that time it only took 12 seconds, since the cache at that point was one that had been generated by that particular CI container.

So it seems like webpack considered the cache to be stale. Maybe something like the cache.buildDepdencies.config files had different modification times or something?

[alexander-akait]

Do you do other changes in node_modules/configuration files/etc? It's very difficult to say what is prevent usage cache, with the source code, I probably answered very quickly, as minimum we need CI log

[sokra]

You can use infrastructureLogging.debug: /PackFile/ in configuration to get more insight.

Note that only the production mode default allow to cache on CI, so make sure to build with mode: "production"

[trappar]

We were using mode: "development" for running CI tests. I switched to production and all issues disappeared.

[sokra]

If you want to run in development mode you need to change the snapshotting configuration to use file hashes instead of only timestamps. See snapshot option for details on that.