Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with pnpm and the way loaders are resolved #5087

Closed
donaldpipowitch opened this issue Jun 19, 2017 · 48 comments
Closed

Problems with pnpm and the way loaders are resolved #5087

donaldpipowitch opened this issue Jun 19, 2017 · 48 comments

Comments

@donaldpipowitch
Copy link

Do you want to request a feature or report a bug?

bug?

What is the current behavior?

I can't create libs which use webpack and loaders, install these libs with pnpm and try to use them, because the modules can't be found.

It looks like pnpm installs my lib here: /Users/foo/test/ws-pnpm-test/node_modules/.registry.npmjs.org/@mercateo/ws/1.0.2/node_modules/@mercateo/ws.

The loader is here /Users/foo/test/ws-pnpm-test/node_modules/.registry.npmjs.org/string-replace-loader and here /Users/foo/test/ws-pnpm-test/node_modules/.registry.npmjs.org/@mercateo/ws/1.0.2/node_modules/string-replace-loader.

I think webpack only looks here for loaders: /Users/foo/test/ws-pnpm-test/node_modules/string-replace-loader, but this does not exist.

I think webpack resolves loaders in node_modules relative to process.cwd(), but not relative to my lib.

I can't just rewrite my config to something like this:

{
  resolveLoader: {
    modules: [
       // goes up '@mercateo/ws/dist' to match `'/Users/foo/test/ws-pnpm-test/node_modules/.registry.npmjs.org/@mercateo/ws/1.0.2/node_modules/string-replace-loader'`
       // this could lead to different directories, if my lib isn't installed with pnpm
       join(__dirname, '..', '..', '..'),
       // this would match `'/Users/foo/test/ws-pnpm-test/node_modules/.registry.npmjs.org/string-replace-loader'`
       // but I can't be sure, that my users use the official registry and not a private mirror or something like that
       join(process.cwd(), 'node_modules/.registry.npmjs.org'),
      'node_modules'  // default
    ]
  }
}

What is the most robust way to solve this?
Can I specify something like that? resolveLoaders.basedir: __dirname, so all loaders can be resolved relative to my lib?

// cc @zkochan

If the current behavior is a bug, please provide the steps to reproduce.

package.json:

{
  "name": "ws-pnpm-test",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "ws": "ws"
  },
  "ws": {
    "type": "browser"
  },
  "dependencies": {
    "@mercateo/ws": "^1.0.2"
  }
}

tsconfig.json:

{}

src/index.ts:

console.log('hello');
$ pnpm install
$ npm run -s ws -- build
ERROR in Entry module not found: Error: Can't resolve 'string-replace-loader' in '/Users/foo/test/ws-pnpm-test'

What is the expected behavior?

Build without error.

If this is a feature request, what is motivation or use case for changing the behavior?

Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.

Node LTS, Webpack 2.6, pnpm 0.71, Mac OS X.

@donaldpipowitch
Copy link
Author

Maybe I can just use resolveLoaders.modules: [ join(__dirname, '..', '..', '..'), 'node_modules' ]?

If my lib is installed with pnpm the path matches '/Users/foo/test/ws-pnpm-test/node_modules/.registry.npmjs.org/@mercateo/ws/1.0.2/node_modules. For npm and yarn it would match '/Users/foo/test/ws-pnpm-test/node_modules'. Looks okay I guess?

Feel free to close this issue, if this is the intended way.

@zkochan
Copy link

zkochan commented Jun 19, 2017

cc @yegor-sytnyk, @max-devjs

Ref pnpm/pnpm#801

We have issues in lots of applications because webpack can't resolve packages in the node_modules created by pnpm. I think it is because webpack doesn't resolve dependencies the way Node does. Node resolves requires from the real path of a module. That is why the symlink approach pnpm uses works with all Node apps. However, it seems to confuse webpack and some other tools like browserify and eslint.

@zkochan
Copy link

zkochan commented Jun 19, 2017

Also, I believe @andreypopp has looked into these issues in scope of CRA pnpm/pnpm#581

And @vjpr has investigated it a bit

@webpack-bot
Copy link
Contributor

This issue had no activity for at least half a year.

It's subject to automatic issue closing if there is no activity in the next 15 days.

@andreineculau
Copy link

bump

@alexander-akait
Copy link
Member

Problem still exists? Can you create minimum reproducible test repo?

@markerdmann
Copy link

I think I'm hitting this issue or something similar in this React project:

https://github.com/dirtprotocol/dirt/tree/master/sampleprojects/sample-web-react

If I install with npm or yarn, everything is fine when I run npm start. If I install with pnpm, I get this error:

/Users/sendrecv/sproj/dirt/packages/lib/dist/services/StaticContractProvider.js
Module not found: Can't resolve '@dirt/contracts' in '/Users/sendrecv/sproj/dirt/packages/lib/dist/services'

@mariotee
Copy link

i also get a similar issue with pnpm

this webpack.config.js works when i have no node_modules and run npm install
however it does not work when i have no node_modules and run pnpm install

const HtmlWebPackPlugin = require("html-webpack-plugin")
const path = require("path")

const htmlWebpackPlugin = new HtmlWebPackPlugin(
{
  template: "./public/index.html",
  filename: "index.html",
})

module.exports =
{
  entry: ["./src/index.js"],
  devServer: {
    historyApiFallback: true,
    overlay: true,
  },
  resolve: {
    modules: [
      path.resolve(__dirname, "node_modules"),
      path.resolve(__dirname, "src")
    ],
  },
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
        use: {
          loader: "babel-loader",
        },
      },
      {
        test: /\.css$/,
        use: [
          { loader: "style-loader" },
          {
            loader: "css-loader",
            options:
            {
              modules: true,
              importLoaders: 1,
              localIdentName: "[name]_[local]_[hash:base64]",              
            },
          },
        ],
      },
    ],
  },
  plugins: [htmlWebpackPlugin],
}

also my package.json

{
  "name": "mydashboard",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "start": "webpack-dev-server --mode development --hot --port 3000",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@babel/cli": "^7.2.3",
    "@babel/core": "^7.3.4",
    "@babel/preset-env": "^7.3.4",
    "@babel/preset-react": "^7.0.0",
    "babel-loader": "^8.0.5",
    "css-loader": "^2.1.0",
    "html-webpack-plugin": "^3.2.0",
    "style-loader": "^0.23.1",
    "webpack": "^4.29.5",
    "webpack-cli": "^3.2.3",
    "webpack-dev-server": "^3.2.1"
  },
  "dependencies": {
    "react": "^16.8.3",
    "react-dom": "^16.8.3"
  }
}

@nmccready
Copy link
Contributor

@sokra any ideas? This issue is still prevalent and old. pnpm is very promising and this is blocker.

@viceice
Copy link

viceice commented Apr 30, 2019

webpack should have a peeDependency to events, because hot module loading requires it

var EventEmitter = require("events");

@NikitaIT
Copy link

failed in pnpm 3.8, too

@alexander-akait
Copy link
Member

/cc @sokra i think we need add to peeDependency optional dep events

@alexander-akait
Copy link
Member

@NikitaIT what is problem?

@NikitaIT
Copy link

@evilebottnawi

  1. it all started with this error Error: Cannot find module 'typescript/package.json' with pnpm  vuejs/vue-cli#4571
  2. i update pnpm to 3.8.1 and solve it
  3. after build I found an error Module not found: Error: Can't resolve 'ts-loader' and same errors for tslib and @types/node.
  4. after forse install tslib and @types/node in package.json my app started
  5. but failed in runtime with Module not found: Error: Can't resolve 'ts-loader'
  6. after install ts-loader same as in vue-cli deps, i found Module build failed (from ../common/temp/node_modules/.registry.npmjs.org/babel-loader/8.0.6_@babel+core@7.6.0+webpack@4.12.0/node_modules/babel-loader/lib/index.js):
  7. I gave up and switched to rush + npm.

@alexander-akait
Copy link
Member

@NikitaIT i think problem in vue-cli, webpack doesn't provide ts-loader out of box

@NikitaIT
Copy link

NikitaIT commented Sep 13, 2019

@evilebottnawi i know, but i followed pnpm/pnpm#1678 and this problem mark as related.
I have Module not found: Error: Can't resolve 'ts-loader' when loader was already installed as a dependency for cli-plugin-typescript.

@wbern
Copy link

wbern commented Oct 16, 2019

It's a shame this hasn't been resolved yet. I love using pnpm because it treats dependencies in a sane way and it's fast. I also like the simplicity of vue-cli and the power of webpack.

Could we attempt to fix this somehow? I don't even know who should change.

This caught my eye in pnpm/pnpm#1678

I think the best/cleanest/most portable solution is to use resolveLoader.alias and require.resolve to map loaders directly to absolute paths. (i.e. config.resolveLoader.alias.set('babel-loader$', require.resolve('babel-loader')))

Any help is appreciated here @evilebottnawi .

@fengerzh
Copy link

Have you tried to put a file .npmrc in your project root folder? The content is just like this:

shamefully-hoist=true

@wbern
Copy link

wbern commented Oct 17, 2019

@fengerzh in my case I'm using rush, which will break if I try to do that I think. So all the shameful options are not an option for me. In my monorepo I would prefer not to break the boundaries between the packages anyway.

@alexander-akait
Copy link
Member

@wbern What is problem? webpack should works (some plugins/loaders/tools can don't work), what is error or example of error?

@ExE-Boss
Copy link

ExE-Boss commented Oct 28, 2019

@fengerzh
It still occurs even with shamefully-hoist=true.

@evilebottnawi

ERROR in Entry module not found: Error: Can't resolve 'babel-loader' in '…\Mozilla\kuma'

Regardless of shamefully-hoist

@alexander-akait
Copy link
Member

Should be yes, feel free to test it and report if you have problems

Doginal pushed a commit to Doginal/nx that referenced this issue Nov 25, 2020
With strict package managers such as pnpm or Yarn PnP, transitive
dependencies are *not* hoisted to the root node_modules folder. This
means that a webpack config defined within a package like
'@nrwl/cypress' cannot resolve loaders like 'ts-loader', unless
'ts-loader' is declared in the workspace's own package.json.

This is a problem because the workspace might define a different version
of 'ts-loader', incompatible with the version declared by
'@nrwl/cypress/package.json'. The workspace should not need to declare
a dependency on 'ts-loader' anyway.

See also:
* pnpm/pnpm#801
* webpack/webpack#5087
Doginal pushed a commit to Doginal/nx that referenced this issue Nov 25, 2020
* fix(core): resolve webpack loaders with `require.resolve()`

With strict package managers such as pnpm or Yarn PnP, transitive
dependencies are *not* hoisted to the root node_modules folder. This
means that a webpack config defined within a package like
'@nrwl/cypress' cannot resolve loaders like 'ts-loader', unless
'ts-loader' is declared in the workspace's own package.json.

This is a problem because the workspace might define a different version
of 'ts-loader', incompatible with the version declared by
'@nrwl/cypress/package.json'. The workspace should not need to declare
a dependency on 'ts-loader' anyway.

See also:
* pnpm/pnpm#801
* webpack/webpack#5087

* fix(core): resolve absolute 'raw-loader' path

When replacing the 'raw-loader' rule in the `getStylesPartial` function,
check for the absolute path of 'raw-loader' rather than just the name.
@MondoGao
Copy link

beep

@MondoGao
Copy link

it there a workaround in webpack@1?

@sprockow
Copy link

sprockow commented Apr 4, 2021

If it's unclear from the above concersation, the resolution problems with webpack loaders were fixed when I upgraded to webpack@5 (while using node@ˆ14 and pnpm@5.18) . I'm not familiar enough with webpack 5 to know why this is the case, but I was able to repro the issue by reverting the change and then reconfirmed the fix by applying the change again.

@ckken
Copy link

ckken commented Apr 22, 2021

same problem

@wbern
Copy link

wbern commented Apr 22, 2021

I think the way to solve this, if you intend to use a config package that provides a webpack config for you, is to use require.resolve for all of your loader references, like for example require.resolve('babel-loader') instead of plain babel-loader as a string in the webpack config.

I don't have these issues anymore, as long as I use the above.

If you don't have control over the package you get the configuration from, you should probably open an issue with them, because require.resolve makes more sense to do from their side rather than webpack's side. Webpack only reads loaders from working directory, changing that would be quite a breaking change, I believe.

@ckken
Copy link

ckken commented Apr 22, 2021

has workspace demo or use with lerna demo

broerjuang pushed a commit to broerjuang/nx-next that referenced this issue Jun 9, 2021
With strict package managers such as pnpm or Yarn PnP, transitive
dependencies are *not* hoisted to the root node_modules folder. This
means that a webpack config defined within a package like
'@nrwl/cypress' cannot resolve loaders like 'ts-loader', unless
'ts-loader' is declared in the workspace's own package.json.

This is a problem because the workspace might define a different version
of 'ts-loader', incompatible with the version declared by
'@nrwl/cypress/package.json'. The workspace should not need to declare
a dependency on 'ts-loader' anyway.

See also:
* pnpm/pnpm#801
* webpack/webpack#5087
broerjuang pushed a commit to broerjuang/nx-next that referenced this issue Jun 9, 2021
* fix(core): resolve webpack loaders with `require.resolve()`

With strict package managers such as pnpm or Yarn PnP, transitive
dependencies are *not* hoisted to the root node_modules folder. This
means that a webpack config defined within a package like
'@nrwl/cypress' cannot resolve loaders like 'ts-loader', unless
'ts-loader' is declared in the workspace's own package.json.

This is a problem because the workspace might define a different version
of 'ts-loader', incompatible with the version declared by
'@nrwl/cypress/package.json'. The workspace should not need to declare
a dependency on 'ts-loader' anyway.

See also:
* pnpm/pnpm#801
* webpack/webpack#5087

* fix(core): resolve absolute 'raw-loader' path

When replacing the 'raw-loader' rule in the `getStylesPartial` function,
check for the absolute path of 'raw-loader' rather than just the name.
@vankop
Copy link
Member

vankop commented Sep 15, 2021

Should work with webpack@5. Feel free to report new issue with reproducible repo.

@vankop vankop closed this as completed Sep 15, 2021
@icecream17
Copy link

https://github.com/icecream17/solver/runs/3694840400
I'm getting this error in create-react-app, react-scripts version ^5.0.0-next.37 and webpack 5.41.1 according to pnpm-lock.yaml.

@icecream17
Copy link

icecream17 commented Sep 24, 2021

Repro that's easy to do yourself (but not necessarily minimal):

  1. npx create-react-app@next --scripts-version=@next --template=typescript@next my-ts-app
  2. cd my-ts-app
  3. Remove package-lock.json and node_modules
    • The command is different based on os, but if it matters I'm on windows vscode so rm -Force node_modules in the terminal and Backspace on the package-lock file.
  4. pnpm install
  5. pnpm run build
Terminal output
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Try the new cross-platform PowerShell https://aka.ms/pscore6

PS C:\Users\Steven\Documents\code\js\test> npx create-react-app@next --scripts-version=@next --template=typescript@next my-ts-app
Need to install the following packages:
  create-react-app@next
Ok to proceed? (y) y
npm WARN deprecated tar@2.2.2: This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.

Creating a new React app in C:\Users\Steven\Documents\code\js\test\my-ts-app.

Installing packages. This might take a couple of minutes.
Installing react, react-dom, and react-scripts with cra-template-typescript...


added 1675 packages in 5m

163 packages are looking for funding
  run `npm fund` for details

Initialized a git repository.

Installing template dependencies using npm...

added 45 packages, and changed 1 package in 12s

164 packages are looking for funding
  run `npm fund` for details

We detected TypeScript in your project (src\App.test.tsx) and created a tsconfig.json file for you.

Your tsconfig.json has been populated with default values.

Removing template package using npm...


removed 1 package, and audited 1720 packages in 4s

164 packages are looking for funding
  run `npm fund` for details

3 moderate severity vulnerabilities

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.

Created git commit.

Success! Created my-ts-app at C:\Users\Steven\Documents\code\js\test\my-ts-app
Inside that directory, you can run several commands:

  npm start
    Starts the development server.

  npm run build
    Bundles the app into static files for production.

  npm test
    Starts the test runner.
    Removes this tool and copies build dependencies, configuration files
    and scripts into the app directory. If you do this, you cant go back!

We suggest that you begin by typing:

  cd my-ts-app
  npm start
Happy hacking!
PS C:\Users\Steven\Documents\code\js\test> cd my-ts-app
PS C:\Users\Steven\Documents\code\js\test\my-ts-app> rm -F node_modules
Remove-Item : Parameter cannot be processed because the parameter name 'F' is ambiguous. Possible matches include: -Filter -Force.
At line:1 char:4
+    ~~
    + CategoryInfo          : InvalidArgument: (:) [Remove-Item], ParameterBindingException
    + FullyQualifiedErrorId : AmbiguousParameter,Microsoft.PowerShell.Commands.RemoveItemCommand
 
PS C:\Users\Steven\Documents\code\js\test\my-ts-app> rm -Force node_modules

Confirm
The item at C:\Users\Steven\Documents\code\js\test\my-ts-app\node_modules has children and the Recurse parameter was not specified. If you continue, all children will be removed with the item. Are you sure you want to 
 continue?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):
PS C:\Users\Steven\Documents\code\js\test\my-ts-app> pnpm install
 WARN  @testing-library/user-event@12.8.3 requires a peer of @testing-library/dom@>=7.21.4 but none was installed.
Packages: +1340
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: C:\Users\Steven\.pnpm-store\v3
  Virtual store is at:             node_modules/.pnpm
node_modules/.pnpm/core-js-pure@3.18.0/node_modules/core-js-pure: Running postinstall script, done in 199ms
node_modules/.pnpm/core-js@3.18.0/node_modules/core-js: Running postinstall script, done in 198ms
node_modules/.pnpm/ejs@2.7.4/node_modules/ejs: Running postinstall script, done in 241ms
Cannot link binary 'jest' of 'jest-cli' to 'C:\Users\Steven\Documents\code\js\test\my-ts-app\node_modules\.pnpm\jest@26.6.0\node_modules\jest\node_modules\.bin': binary of 'jest' is already linked

dependencies:
+ @testing-library/jest-dom 5.14.1
+ @testing-library/react 11.2.7 (12.1.0 is available)
+ @testing-library/user-event 12.8.3 (13.2.1 is available)
+ @types/jest 26.0.24 (27.0.2 is available)
+ @types/node 12.20.26 (16.9.6 is available)
+ @types/react 17.0.24
+ @types/react-dom 17.0.9
+ react 17.0.2
+ react-dom 17.0.2
+ react-scripts 5.0.0-next.37
+ typescript 4.4.3
+ web-vitals 1.1.2 (2.1.0 is available)
PS C:\Users\Steven\Documents\code\js\test\my-ts-app> pnpm run build

> my-ts-app@0.1.0 build C:\Users\Steven\Documents\code\js\test\my-ts-app
> react-scripts build

Creating an optimized production build...
Failed to compile.

Module not found: Error: Can't resolve 'source-map-loader' in 'C:\Users\Steven\Documents\code\js\test\my-ts-app'


 ELIFECYCLE  Command failed with exit code 1.
PS C:\Users\Steven\Documents\code\js\test\my-ts-app>

Screenshot
image

Repo: https://github.com/icecream17/my-ts-app

Technically, icecream17/solver@a09eefb is also a repo showing this.
So you can compare solver's pnpm-lock to my-ts-app's pnpm-lock here: https://www.diffchecker.com/kcu6McIU

@vjpr
Copy link

vjpr commented Sep 24, 2021

Easy fix is to install source-map-loader at the top of your monorepo/project.

@icecream17
Copy link

icecream17 commented Sep 24, 2021

I kinda fixed it in package.json, but now react-scripts gets an error. I will try your idea

{
  "pnpm": {
    "overrides": {
      "webpack": "^5.51.0"
    }
  }
}

EDIT: Weirdly this package.json solution doesn't work, there's still the source-map-loader problem

@icecream17
Copy link

icecream17 commented Sep 24, 2021

Unfortunately when source-map-loader is installed, there's another error
image

@alexander-akait
Copy link
Member

alexander-akait commented Sep 24, 2021

something wrong with postcss-normalize

@mikob
Copy link

mikob commented Apr 29, 2022

make sure you have resolve.symlinks set to true

@misha-erm
Copy link

I think the way to solve this, if you intend to use a config package that provides a webpack config for you, is to use require.resolve for all of your loader references, like for example require.resolve('babel-loader') instead of plain babel-loader as a string in the webpack config.

I don't have these issues anymore, as long as I use the above.

Wow, you saved my day... Was very frustrated by that issue but require.resolve made the thing for me
Thank you @wbern

In my scenario I have a monorepo with separate packages for babel-preset and frontend so I was able to easily wrap all babel-loaders with require.resolve

@wrightwriter
Copy link

wrightwriter commented Jun 13, 2023

Fixed with this in .npmrc in current version of webpack (+ svelte and typescript):

node-linker=hoisted
shamefully-hoist=true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests