Linking dependencies across drives is slow

[fabiancook]

Do you want to request a feature or report a bug?

bug

What is the current behavior?

When installing any package with yarn, my entire node_modules files are cloned, this happens in what I think is the "Linking Dependencies" phase

I think it is related to this:

#6177

I usually run of an SSD, but whenever I use the HDD the process is super super slow, to the point where I have to go out and come back 10 minutes later...

When running in verbose mode I see this:

verbose 3.913306808 Copying "/Users/fabian/Library/Caches/Yarn/v4/npm-@storybook-podda-1.2.3-53c4a1a3f8c7bbd5755dff5c34576fd1af9d38ba/node_modules/@storybook/podda/yarn.lock" to "/Volumes/External Drive/Files/Development/src/project/node_modules/@storybook/podda/yarn.lock"

So the problem happens when copying from one drive to another, which is super frustrating because now yarn forces me to use only one drive.

If the current behavior is a bug, please provide the steps to reproduce.

yarn add --dev react-testing-library

What is the expected behavior?

For yarn add to be fast

Please mention your node.js, yarn and operating system version.

$ npx envinfo

  System:
    OS: macOS 10.14
    CPU: (12) x64 Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz
    Memory: 5.47 GB / 32.00 GB
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 11.13.0 - /usr/local/bin/node
    Yarn: 1.15.2 - /usr/local/bin/yarn

[fabiancook]

I would expect the patterns passed here should just match the pattern for the modules I am installing. https://github.com/yarnpkg/yarn/blob/master/src/package-linker.js#L723

[fabiancook]

Within this function I can log, and I see this...

Change:

console.log('preparePatternsForLinking', { cwdIsRoot, patterns })

Output:

preparePatternsForLinking { cwdIsRoot: true,
  patterns:
   [ '@fortawesome/fontawesome@^1.1.8',
     '@fortawesome/fontawesome-pro-light@^5.0.13',
     '@fortawesome/fontawesome-svg-core@^1.2.0',
     '@fortawesome/pro-light-svg-icons@^5.4.1',
     '@fortawesome/pro-solid-svg-icons@^5.4.1',
     '@fortawesome/react-fontawesome@^0.1.0',
     '@turf/bbox-polygon@^6.0.1',
     '@turf/destination@^6.0.1',
     '@turf/difference@^6.0.2',
     '@turf/distance@^6.0.1',
     '@turf/helpers@^6.1.4',
     '@turf/transform-translate@^5.1.5',
     '@turf/turf@^5.1.6',
     'ag-grid-community@^19.1.2',
     'ag-grid-react@^20',
     'aws-sdk@^2.277.1',
     'bootstrap@^4.1.1',
     'i18next@^11.9.1',
     'js-base64@^2.4.8',
     'js-sha256@^0.9.0',
     'jsencrypt@^3.0.0-rc.1',
     'jss@^9.8.7',
     'jss-preset-default@^4.5.0',
     'local-storage-fallback@^4.1.1',
     'luxon@^1.4.3',
     'mapbox-gl@^0.47.0',
     'mitt@^1.1.3',
     'moment@^2.22.2',
     'nanographql@^2.0.0',
     'prop-types@^15.6.1',
     'qs@^6.6.0',
     'react@^16.8.6',
     'react-dom@^16.4.1',
     'react-i18next@^8.0.7',
     'react-router@^4.3.1',
     'react-router-dom@^4.3.1',
     'react-sidebar@^3.0.2',
     'react-waterfall@^4.0.0',
     'reactstrap@^6.1.0',
     'uuid@^3.2.1',
     'validator@^10.8.0',
     '@storybook/addon-actions@^3.4.11',
     '@storybook/addon-links@^3.4.11',
     '@storybook/addons@^3.4.11',
     '@storybook/react@^3.4.11',
     'babel-core@^6.26.3',
     'babel-runtime@^6.26.0',
     'nwb@0.21.x',
     'react-testing-library@^6.0.4' ] }

Why are all these files being included for checking? In this case I am running yarn add --dev react-testing-library and only expect react-testing-library@^6.0.4 to be included for this point in time.

[fabiancook]

By replacing this function with the following:

async fetchRequestFromCwd(): Promise<InstallCwdRequest> {
    const original = await Install.prototype.fetchRequestFromCwd.call(this, this.args);
    return {
      ...original,
      patterns: []
    }
  }

Things speed up a bunch, and the module I want gets installed as I would expect, but then all the other modules are removed... I feel like it should be that only modules matching the currently installing dependency should be touched, anything else should stay static, unless requested to be changed

[fabiancook]

Hi, I'm unsure how to make progress here, is there some way I can push this forward? If there are any hints on this I can do a pull request.

In its current form I can't use yarn at all if utilizing the additional drives.

[noahkiss]

I'm seeing this same issue. Took 15+ minutes on network share vs. 10 seconds locally.

[sibelius]

I've made a repro for this https://github.com/sibelius/yarn-workspaces-slow

[prakashsvmx]

I am also seeing it to be very slow.