yarn workspaces do not create correct symlinks when node_modules is symlinked as well

[samuela]

Do you want to request a feature or report a bug? bug

What is the current behavior?
I set up workspaces according to the docs here: https://yarnpkg.com/lang/en/docs/workspaces/. I have a root package.json (shown in yarn-error.log below) and package.json's in each of these subdirs. There are no dependencies between any of the workspaces. Here's what happens when I run yarn install in the root directory:

$ yarn install
yarn install v1.13.0
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
warning "workspace-aggregator-3ae5604c-da7d-41f8-8a75-ce118cf782ca > api > ts-jest@23.10.5" has incorrect peer dependency "jest@>=22 <24".
error An unexpected error occurred: "EEXIST: file already exists, mkdir '/nu/skainswo/nuvemfs/node_modules/api'".
info If you think this is a bug, please open a bug report with the information provided in "/nu/skainswo/nuvemfs/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

And here's yarn-error.log:

Arguments: 
  /usr/local/Cellar/node/11.8.0/bin/node /usr/local/Cellar/yarn/1.13.0/libexec/bin/yarn.js install

PATH: 
  /Users/skainswo/.opam/default/bin:/usr/local/sbin:/Users/skainswo/dev/google-cloud-sdk/bin:/Users/skainswo/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:/opt/X11/bin:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Xamarin Workbooks.app/Contents/SharedSupport/path-bin

Yarn version: 
  1.13.0

Node version: 
  11.8.0

Platform: 
  darwin x64

Trace: 
  Error: EEXIST: file already exists, mkdir '/nu/skainswo/nuvemfs/node_modules/api'

npm manifest: 
  {
    "private": true,
    "workspaces": [
      "api",
      "nuvemfs-cli",
      "shared"
    ]
  }

yarn manifest: 
  No manifest

Lockfile: 
  No lockfile

If the current behavior is a bug, please provide the steps to reproduce. described above

What is the expected behavior? yarn install to work in workspaces

Please mention your node.js, yarn and operating system version.

[samuela]

Also,

$ ls -al node_modules/api
lrwxr-xr-x  1 skainswo  wheel  6 Jan 29 09:15 node_modules/api -> ../api

[samuela]

I thought this might be a naming conflict between my workspace "api" and an external dependency named "api". Seems not to be case. I renamed that package to something unique, but now I see

$ yarn install
yarn install v1.13.0
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
error An unexpected error occurred: "EEXIST: file already exists, mkdir '/nu/skainswo/nuvemfs/node_modules/nuvemfs-cli'".
info If you think this is a bug, please open a bug report with the information provided in "/nu/skainswo/nuvemfs/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

[samuela]

This shouldn't really matter but FWIW my node_modules directory is symlinked somewhere else.

[samuela]

Ok, well it seems as though yarn workspaces don't work when node_modules is symlinked, because the symlinks that it creates within node_modules are relative and therefore will not be pointing to the right things when node_modules is not in the root of the project directory. In other words, yarn makes assumptions about where node_modules actually lives that are not correct. Editing the issue to reflect this...

My hacky workaround for the moment is to use --modules-folder.

[samuela]

Where is this logic implemented? I can try taking a stab at it...

[samuela]

I just deleted all of the node_modules folders and tried again... somehow it's magically working now. Very strange... I did however change the package names from "api", "cli" to "@myproj/api" and "@myproj/cli", etc.

EDIT: I forgot I had --modules-folder /absolute/path/to/node_modules in my .yarnrc. Install first with that and then adding a symlink to it manually in the root of the project is working for me now. Of course that doesn't solve the issue per se, but it's a workaround for anyone else in my shoes for the time being.

[mmahut]

Seeing this issue as well.

[DenyVeyten]

Same for us for 1.22.0, it works fine for 1.6.0

[shayc]

This is so awful:

Upgrading to yarn v1.22.0 broke 23 tests out of ~400 in our workspaces repo, (something about missing modules in our monorepo).

Downgrading to v1.9.2 fixed the issue.
Edit:
v1.17.3 also works great.

If this saves you some time I want some beer.

[donmccurdy]

I'm having the same issue. Yarn workspaces aren't creating symlinks correctly when used with Lerna, and my builds fail because dependencies among workspaces aren't resolved.

Moreover, I can't install 1.9.2:

❯ yarn set version '1.9.2'

Resolving 1.9.2 to a url...
error An unexpected error occurred: "Release not found: 1.9.2".

Trying a looser version gets the broken 1.22 release:

❯ yarn set version '^1.9.2'

Resolving ^1.9.2 to a url...
Downloading https://github.com/yarnpkg/yarn/releases/download/v1.22.4/yarn-1.22.4.js...

[donmccurdy]

Looks like this was my mistake. I had references to my subpackages in package.json, left over from migrating the project from npm+Lerna:

"devDependencies": {
  "@my-scope/a": "file:packages/a",
  "@my-scope/b": "file:packages/b",
...

With those lines removed, Yarn appears to be linking packages as expected.

[pitops]

downgrading from v1.22.0 to something like v1.17.3 as referred to above worked.

[lisbethw1130]

A similar issue here #8079 , and the author said don't make a symlink and close the issue. so it's a breaking change?

[diasrodinei]

If that directory (node_modules) is mapped by Google Drive, it use to create something like a simlink and that happens. Just move it to a directory that is not observed by any sync tool and it's sorted out.

[holstvoogd]

For anyone else stumbeling in here after trying to symlink the node_modules dir:

I've worked around yarn not expanding links by using bind mounts. So I have something like this stucture:

apps/foo/node_modules
apps/bar/node_modules
lib/baz/node_modules
node_modules

But I want all these dirs to be in a shared path, lets say shared_modules, something like this:

shared_modules/foo/node_modules
shared_modules/bar/node_modules
shared_modules/baz/node_modules
shared_modules/node_modules

(This is so that I can cache them in one go in my CI)

So what worked is to do sudo --bind shared_modules/foo/node_modules apps/foo/node_modules for each path.
This only works on linux afaik, macos has a weird mount command.