New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The global store should always be on the same disk on which installation is done #712

Closed
zkochan opened this Issue Apr 21, 2017 · 33 comments

Comments

Projects
None yet
7 participants
@zkochan
Member

zkochan commented Apr 21, 2017

pnpm version:

0.64.8

Code to reproduce the issue:

Run pnpm install no a secondary disk (lets say it is disk D), not the one on which your system lives.

Expected behavior:

Installation is successful and a store on disk D is created/used

Actual behavior:

A store on disk C is used and pnpm fails because hardlinks cannot be cross-disk.

Additional information:

  • node -v prints: any
  • Windows, OS X, or Linux?: Windows

Temporary fix

If you want to do installations on a different disk till this is not fixed, just change the location of your global store to a directory on the disk you wanna use.

npm config set store-path <path to the store>
@ghost

This comment has been minimized.

ghost commented Apr 21, 2017

in my case MacOS (X), i'll try it out on Ubuntu later (might work, not sure)

ERROR EXDEV: cross-device link not permitted,
link '/Users/dym/.pnpm-store/1/registry.npmjs.org/gm/1.23.0/.npmignore'
-> '/Volumes/Data/work/site/node_modules/.registry.npmjs.org/gm/1.23.0/node_modules/gm+stage/.npmignore'

@zkochan

This comment has been minimized.

Member

zkochan commented Apr 22, 2017

Seems like this package can help us implement this https://www.npmjs.com/package/drivelist

@pstoev

This comment has been minimized.

pstoev commented Apr 24, 2017

@zkochan how do you think that check would work? Checking against only physical drives? I think the issue is reproducible on logical drives as well...

@zkochan

This comment has been minimized.

Member

zkochan commented Apr 24, 2017

@zkochan

This comment has been minimized.

Member

zkochan commented Apr 24, 2017

@noformnocontent could you please check on your environment whether path.parse correctly detects the root on your different disks?

@ghost

This comment has been minimized.

ghost commented Apr 25, 2017

tried on Ubuntu,

ERROR EXDEV: cross-device link not permitted,
link '/home/dym/.pnpm-store/1/registry.npmjs.org/morgan/1.8.1/HISTORY.md' ->
'/media/dym/Data/work/site/node_modules/.registry.npmjs.org/morgan/1.8.1/node_modules/morgan+stage/HISTORY.md'  

path.parse:

> path.parse('/media/dym/Data/work/site/package.json')
{ root: '/',
  dir: '/media/dym/Data/work/site',
  base: 'package.json',
  ext: '.json',
  name: 'package' }
@zkochan

This comment has been minimized.

Member

zkochan commented Apr 25, 2017

thanks, I hoped it would return /media/dym as root. In this case we'll have to use the drivelist package. I haven't found anything else

@ghost

This comment has been minimized.

ghost commented Apr 25, 2017

pattern is /media/<user>/<mounted_volume>, so in that case /media/dym/Data
i'll try the path.parse on mac later today

what happens if you run pnpm in a Docker/Kubernetes enviroment -- should be no problem, right?

@zkochan

This comment has been minimized.

Member

zkochan commented Apr 25, 2017

I guess if it has one filesystem then it should be fine

@zkochan

This comment has been minimized.

Member

zkochan commented Apr 25, 2017

OK, so on Windows it will be <disk letter>:\

on Linux /media/<user>/<mounted_volume>

on OSX /Volumes/<mounted_volume>

Seems easy.

@minecrawler

This comment has been minimized.

minecrawler commented Apr 25, 2017

on Linux /media//<mounted_volume>

what if I do

$ sudo mount /somedir && cd /somedir && pnpm install

on Linux, with /somedir defined in /etc/fstab? What about

$ sudo mount /dev/sdc1 /mnt/tmp && cd /mnt/tmp && pnpm install

?

On Linux, any directory might be on a different disk! Not everyone uses Ubuntu and its way of handling some things...

@pstoev

This comment has been minimized.

pstoev commented Apr 25, 2017

thanks, I hoped it would return /media/dym as root. In this case we'll have to use the drivelist package. I haven't found anything else

@zkochan wouldn't it be enough if we "extract" (e.g. dir.substring second slash, regex, etc) info from the dir property. I'm talking the path.parse approach?

@minecrawler

This comment has been minimized.

minecrawler commented Apr 25, 2017

Here is a way to get the filesystems on Linux: https://www.cyberciti.biz/faq/linux-command-list-mounted-devices-in-terminal/

tl;dr: read the file /proc/mounts. On each line, the first bit is the partition (for example /dev/sda3) and the second bit is the mount-point (for example /). You will have to check the given paths against the mount-points and find the closest match (be careful to not only always find the root-fs)

@pstoev

This comment has been minimized.

pstoev commented Apr 25, 2017

@minecrawler that's only for linux. path.parse seems more universal to me across different OS's

@minecrawler

This comment has been minimized.

minecrawler commented Apr 25, 2017

@pstoev yeah, true, but you cannot parse a path on linux in order to get the mount point. Any directory could be located on any disk and partition. There is no way to tell just by looking at the path, as is the case for Windows (I don't know about macOS, however it is *NIX at heart, so it might have the same problem)

@zkochan

This comment has been minimized.

Member

zkochan commented Apr 25, 2017

We should also come up with a way to configure the path to the store on file systems. I think we can use a similar format to the one that is used for storing registries and auth info.

store = D:\data\pnpm-store
D:\subdir:store = D:\subdir\foo\pnpm-store
D:\:store = D:\store
@tunnckoCore

This comment has been minimized.

tunnckoCore commented Apr 28, 2017

Actually, does fixing that issue means that it will be allowed cross-device linking? If yes, ping me, because don't want to work and store on same device ;d I just have totally separated partition for caches, backups and such things.

@zkochan

This comment has been minimized.

Member

zkochan commented Apr 28, 2017

@tunnckoCore

This comment has been minimized.

tunnckoCore commented Apr 29, 2017

@zkochan that's very sad ;/ One more thing why won't use it. Maybe one day when I dont have separate partitions, but.. It's not the only problem. More important kinda problem for me is that non-flat tree, because it messes with my planned package development workflow. Yea, otherwise, it is awesome and super fast exactly because the hardlinks and non-flat architecture.

I just kinda like the trick that i found using flat mode. Let me explain the plan, it's a bit offtopic, but shows the cases/scenario why some like me won't use it at el.

  1. I'm using StandardJS, Prettier and Atom with Linter-ESLint and Prettier-Atom packages
  2. I have .eslintrc.json file in my packages
  3. My packages just includes standard as dependency
  4. the eslintrc file (notice that it isn't even eslintConfig field) is intentional, it gives great IDE feeling
  • first, because the Linter ESLint recognizes it
  • second, allows contributors to just use their global eslint, instead forcing them to have standard globally or forcing them to run npm scripts of the package (no matter that i have script that executes standard)
  • third, extensibility - for things like enforcing standard/object-curly-even-spacing: error, always, just because hate to not have parens which is allowed in standard by default
  • fourth, it resolves the eslint-config-standard that is installed by the standard package
  1. My npm scripts are fantastic, they just not allow even to be commited something that is wrong
  • and actually i even don't need them, exactly because that integration that i'm trying to accomplish
  • everything is realtime in the Atom.. even don't need to run any scripts, because linting and formating is done on save

So, hope that clear the situation a bit.

@zkochan

This comment has been minimized.

Member

zkochan commented Apr 29, 2017

I don't understand what is the issue. It will still use not more space per partition than npm or yarn.

Also I don't see in your workflow where you leverage the flattened node_modules structure. It seems you just use global packages

@tunnckoCore

This comment has been minimized.

tunnckoCore commented Apr 29, 2017

Also I don't see in your workflow where you leverage the flattened node_modules structure.

Atom Editor's ESLint resolves eslintrc which in turn will try to resolve eslint-config-standard and if I use Pnpm the Linter-ESLint will throw an error that it can't resolve the StandardJS config because the tree isn't flat. To fix that, one solution is to add eslint-config-standard to my devDeps too, but I don't. If i add it, I'll just add all the eslint-* stuff instead of standard. But I don't want to add them, because the GreenKeeper bumping spam, so i always try to have as low deps as possible.

It seems you just use global packages

No. Almost never, just have rollup, mocha (hence, didn't write mocha for years ;d) and standard just for my fast fingers.

It will still use not more space per partition than npm or yarn.

yea, in any way, in my case it won't be so much, and so i can reconsider using Pnpm one day when i'm tired of yarn.

In any way, keep doing the good job! 🎉

@zkochan

This comment has been minimized.

Member

zkochan commented May 5, 2017

Should we rename "global store" to something else to reflect that it is per filesystem/partition?

Maybe "partition store", or "disk store"... any ideas?

@Meeeeow

This comment has been minimized.

Meeeeow commented May 9, 2017

@zkochan

This comment has been minimized.

Member

zkochan commented May 9, 2017

Well yarn does not use hard links pointing outside of the node_modules folder, so they don't have this issue

@minecrawler

This comment has been minimized.

minecrawler commented May 9, 2017

@Meeeeow Don't use commands/3rd party programs from within your source code. You never know if they exist (even though the GNU utils are popular, there are other options available!). Read the source of truth yourself (/proc/mounts). I already posted a guide earlier.

@zkochan

This comment has been minimized.

Member

zkochan commented May 24, 2017

I suggest a quick fix before the smart, harder one.

If there is a "cross-device link not permitted" error when creating the hard link from the global store to the project's node_modules then pnpm prints a warning and falls back to copying.

zkochan added a commit that referenced this issue Jun 4, 2017

zkochan added a commit that referenced this issue Jun 4, 2017

@zkochan

This comment has been minimized.

Member

zkochan commented Jun 4, 2017

A temporary fix published with v0.69.0-beta.3. It can be installed via npm i -g pnpm@next

The fix is: if linking from store fails, the package is copied.

Tested on Linux, let me know if there are issues on other systems

@TigersWay

This comment has been minimized.

TigersWay commented Jul 27, 2017

As this does not seem to be fixable quickly, the "fix" should be added and explained in the first page. Quite a lot of dev. already gave up and it's a pity!

zkochan added a commit that referenced this issue Jul 27, 2017

@zkochan

This comment has been minimized.

Member

zkochan commented Jul 27, 2017

Mentioned these issues in the README

@zkochan zkochan self-assigned this Sep 27, 2017

zkochan added a commit to pnpm/package-store that referenced this issue Sep 30, 2017

zkochan added a commit to pnpm/package-store that referenced this issue Sep 30, 2017

zkochan added a commit to pnpm/package-store that referenced this issue Sep 30, 2017

@zkochan zkochan removed the help wanted label Sep 30, 2017

@zkochan zkochan closed this in 9672df8 Sep 30, 2017

@zkochan

This comment has been minimized.

Member

zkochan commented Sep 30, 2017

Published in v1.15.0. Each disk has a dedicated store now.

@dandv

This comment has been minimized.

Contributor

dandv commented Feb 6, 2018

Admittedly I haven't fully read each comment on this issue, but "Each disk has a dedicated store now." doesn't seem to be the case with Truecrypt drives mounted on Linux.

I've just installed pnpm and haven't configured a store. The README states

If there is no homedir on the disk, then the store is created in the root. For example, if installation is happening on disk D then the store will be created in D:\.pnpm-store.

I have some packages on a Truecrypt drive mounted at /t/, and if I run pnpm install there, I get the EXDEV: cross-device link not permitted warning, and a node_modules/.registry.npmjs.org directory is created in each package directory, containing directories and files (not links) for each package. I don't have a home directory on that disk, but /t/.pnpm-store was not created in the root of the Truecrypt disk. Instead, ~/.pnpm-store was created.

Presumably can't tell that the /t/ path is not the same disk as /home/dandv?

I tried as a workaround to set npm config set store /t/.pnpm-store and delete /home/dandv/.pnpm-store, but that didn't help - node_modules/.registry.npmjs.org was still created in each package directory, and it contained directories and files (no symlinks), not saving any space.

@zkochan

This comment has been minimized.

Member

zkochan commented Feb 6, 2018

@dandv the second time, if you specified store to be on the same disk, you did not get the EXDEV: cross-device link not permitted warning, right? In that case files in node_modules/.registry.npmjs.org are hard links and symlinks. They are not copies of files from the store

@dandv

This comment has been minimized.

Contributor

dandv commented Feb 8, 2018

Thanks @zkochan. indeed, I didn't get any warnings the second time.

As an experiment, I installed the dependencies from the same package.json twice, in two different directories on the Truecrypto drive. Still, 9MB of space were consumed. This is much better than yarn, but I'm curious why 9 extra MB.

/t/prg $ pnpm config get store
/t/.pnpm-store

/t/prg $ rm -rf $(pnpm config get store)

/t/prg $ df -h
Filesystem              Size  Used Avail Use% Mounted on
[...]
/dev/mapper/truecrypt1  1.5G  368M  986M  28% /t
[...]

/t/prg $ (cd monitors ; pnpm install)
Packages: +372
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Resolving: total 372, reused 0, downloaded 372, done
Running install for registry.npmjs.org/utf-8-validate/4.0.0, done
/t/prg/monitors/node_modules/influx-utils -> ../influx-utils
[...]
/t/prg/monitors/node_modules/utils -> ../utils
dependencies:
+ big.js 5.0.3
[...]
+ simple-xmpp 1.3.0

devDependencies:
+ eslint 4.17.0
+ eslint-config-airbnb-base 12.1.0
+ eslint-plugin-import 2.8.0

/t/prg $ df -h
Filesystem              Size  Used Avail Use% Mounted on
[...]
/dev/mapper/truecrypt1  1.5G  475M  880M  36% /z
[...]

# At this point, /t/.pnpm-store is ~65MB and monitors/node_modules is ~47MB.

/t/prg $ mkdir monitors2
/t/prg $ cp monitors/package.json monitors2/
/t/prg $ cd monitors2; pnpm install ; df -h
Packages: +372
Running install for registry.npmjs.org/utf-8-validate/4.0.0, done
/t/prg/monitors2/node_modules/influx-utils -> ../influx-utils
[...]
/t/prg/monitors2/node_modules/utils -> ../utils
dependencies:
+ big.js 5.0.3
[...]
+ simple-xmpp 1.3.0

devDependencies:
+ eslint 4.17.0
+ eslint-config-airbnb-base 12.1.0
+ eslint-plugin-import 2.8.0

Filesystem              Size  Used Avail Use% Mounted on
[...]
/dev/mapper/truecrypt1  1.5G  484M  871M  36% /z
[...]

# Only 9MB extra were consumed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment