Skip to content
This repository has been archived by the owner. It is now read-only.

npm does not work with Virtualbox shared folders. #992

Closed
edmellum opened this issue May 30, 2011 · 46 comments

Comments

@edmellum
Copy link

commented May 30, 2011

I'm running a Virtualbox with Linux in it and I'm using a shared folder so I have all my actual files on my Windows host. That way I can use my text editor in Windows.

When I use an older version of NPM everything works fine, probably because it stays out of my shared folder.

Problem is that Virtualbox shared folders don't support symlinks in a shared folder, so I'm getting errors. I can't quite understand why NPM would need to place symlinks in the local folder when installing either.

npm ERR! tar "-mvxpf" "-" "--no-same-owner" "-C" "/media/sf_Projects/Personal/chatty/node_modules/___jade.npm" tar: package/lib/index.js: Cannot create symlink to `jade.js': Protocol error
npm ERR! tar "-mvxpf" "-" "--no-same-owner" "-C" "/media/sf_Projects/Personal/chatty/node_modules/___jade.npm" tar: Exiting with failure status due to previous errors
npm ERR! tar "-mvxpf" "-" "--no-same-owner" "-C" "/media/sf_Projects/Personal/chatty/node_modules/___jade.npm"
npm ERR! Failed unpacking /home/david/.npm/jade/0.11.0/package.tgz
npm ERR! error installing jade@0.11.0 Error: Failed tar "-mvxpf" "-" "--no-same-owner" "-C" "/media/sf_Projects/Personal/chatty/node_modules/___jade.npm"
npm ERR! error installing jade@0.11.0 exited with 2
npm ERR! error installing jade@0.11.0     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/lib/utils/exec.js:85:8)
npm ERR! error installing jade@0.11.0     at ChildProcess.emit (events.js:67:17)
npm ERR! error installing jade@0.11.0     at ChildProcess.onexit (child_process.js:192:12)
npm ERR! Error: Failed tar "-mvxpf" "-" "--no-same-owner" "-C" "/media/sf_Projects/Personal/chatty/node_modules/___jade.npm"
npm ERR! exited with 2
npm ERR!     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/lib/utils/exec.js:85:8)
npm ERR!     at ChildProcess.emit (events.js:67:17)
npm ERR!     at ChildProcess.onexit (child_process.js:192:12)
npm ERR! Report this *entire* log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <npm-@googlegroups.com>
npm ERR!
npm ERR! System Linux 2.6.32-31-generic
npm ERR! command "node" "/usr/local/bin/npm" "install" "jade"
@isaacs

This comment has been minimized.

Copy link
Member

commented May 30, 2011

That's correct, because the vbox shared filesystem doesn't support symbolic links, it won't work with local installs.

npm will likely never work with this setup -- symlinks aren't supported, but the OS is telling us that symlinks are supported, so there's really no way to win.

@isaacs isaacs closed this May 30, 2011

@clintberry

This comment has been minimized.

Copy link

commented Apr 5, 2012

Anyone have any suggestions for this? I started using Vagrant recently and this is a hurdle for node.js projects. Is there a way to force npm not to use symlinks?

@edmellum

This comment has been minimized.

Copy link
Author

commented Apr 6, 2012

I'm using Vagrant too and have settled on using my editor to upload my project files into the VM as I save them using SFTP instead of sharing in folders. My solution isn't optimal for me, but as Isaac says when the OS says it supports symlinks but doesn't there's really no winning.

@geedew

This comment has been minimized.

Copy link

commented Oct 8, 2012

I found a workaround is to install the packages one directory up on the guest. You can also make the sym-link to the node_modules folder from another location yourself.

@kud

This comment has been minimized.

Copy link

commented Oct 10, 2012

So what's the best way? Installing modules in global, and copy it into node_modules in your local project?

@kud

This comment has been minimized.

@geedew

This comment has been minimized.

Copy link

commented Oct 16, 2012

I found the best way is to put a node_modules one directory up from the
shared directory
On Oct 10, 2012 4:55 PM, "Erwann MEST" notifications@github.com wrote:

So what's the best way? Installing modules in global, and copy it into
node_modules in your local project?


Reply to this email directly or view it on GitHubhttps://github.com/isaacs/npm/issues/992#issuecomment-9321166.

@kud

This comment has been minimized.

Copy link

commented Oct 16, 2012

It's a solution yes, but I prefer mine. ;)

@geedew

This comment has been minimized.

Copy link

commented Oct 17, 2012

Your way only works if you have few projects that don't collide in the
requirements , like different versions of underscore being needed.
On Oct 16, 2012 6:15 PM, "Erwann MEST" notifications@github.com wrote:

It's a solution yes, but I prefer mine. ;)


Reply to this email directly or view it on GitHubhttps://github.com/isaacs/npm/issues/992#issuecomment-9510643.

@kud

This comment has been minimized.

Copy link

commented Oct 17, 2012

Did you read my latest solution (https://github.com/isaacs/npm/issues/992#issuecomment-9510227) ? I just enabled symlinks on shared folders. In that way, I can use npm without any "hack". I can do $ npm install myModule now.

@ebertx

This comment has been minimized.

Copy link

commented Jul 3, 2013

I second the solution kud mentioned. Enabling symlinks seems like the simplest solution for a dev environment.

@droope

This comment has been minimized.

Copy link

commented Aug 16, 2013

This happened to me, and I firmly believe that it should be more widely known. :@

@matteius

This comment has been minimized.

Copy link

commented May 5, 2014

This affected me too, tried to enable symlinks on the shared folder, but I must not have defined the command quite correctly. It shows up in enumeration, yet I still can't create the symlinks.

@kud

This comment has been minimized.

Copy link

commented May 5, 2014

I've stopped any tries to develop on Windows, even with a VM. This OS sucks so much.

@matteius

This comment has been minimized.

Copy link

commented May 5, 2014

I'll disagree and take my time somewhere else Kud.

@khromov

This comment has been minimized.

Copy link

commented Jul 17, 2014

Got burned by this when using Grunt, so I copied the project directory (with package.json) to another folder outside the shared folder, ran npm install there and then copied node_modules back to the shared folder. After that grunt commands seem to works fine.

@othiym23 othiym23 changed the title NPM does not work with Virtualbox shared folders. npm does not work with Virtualbox shared folders. Sep 22, 2014

@othiym23 othiym23 added the support label Sep 22, 2014

@BenLiyanage

This comment has been minimized.

Copy link

commented Dec 1, 2015

@kud that link is dead, can you post your solution to this thread?

@kud

This comment has been minimized.

Copy link

commented Dec 1, 2015

Sorry it wasn't my solution, it was a solution of someone else and I don't remember what it was. :/

@rafaelpirolla

This comment has been minimized.

Copy link

commented Dec 5, 2015

@BenLiyanage @kud Probably something like: http://www.ahtik.com/blog/fixing-your-virtualbox-shared-folder-symlink-error/
If you automate it, please share.

@adamlwgriffiths

This comment has been minimized.

Copy link

commented Jan 23, 2016

Since no one will paste the actual fix, and these pages are slowly 404'ing, here's the code to fix it

# if running inside VirtualBox on a shared folder
# you must enable symlinks on the shared folder
$ VBoxManage setextradata "<vm name>" VBoxInternal2/SharedFoldersEnableSymlinksCreate/<shared folder> 1

# verify with
$ VBoxManage getextradata "<vm name>" enumerate
@matteius

This comment has been minimized.

Copy link

commented Feb 9, 2016

I use Virtual Box with shared folders on both projects that do and do not require npm package installs. I suggest trying npm install --no-bin-links to see if you can work around most of your problems that way.

Another option may be to mount the share as a different file system type such as samba and try there. I am hesitant to suggest that because at least the shared file system protects the interactions of the disk safely, in other words less chance of corruption.

Finally, use an IDE like PyCharm to push files out to your deployed VM and just run the npm install there natively and that is surely going to be the best experience besides say the first option of no bin links.

@BenLiyanage

This comment has been minimized.

Copy link

commented Feb 10, 2016

It seems like npm > 3.1 fixes this issue also by generating a flatter folder tree. I haven't confirmed this yet, but hope to do it over the next couple weeks.

@sparkida

This comment has been minimized.

Copy link

commented Feb 20, 2016

@BenLiyanage you will be sorely disappointed, still exists in 3.6.0

@BenLiyanage

This comment has been minimized.

Copy link

commented Feb 22, 2016

Someone else on our team has upgraded us to npm 3.7.x--I haven't had an opportunity to verify if it works better on that version yet. I hope to do so over the next couple days.

@demonkoryu

This comment has been minimized.

Copy link

commented Mar 21, 2016

Related to this, I had several errors of the form

ETXTBSY: text file is busy, rename '/vagrant/.npm/json-loader/0.5.4/package.tgz.b3c1e249ab668c318b2a6a8c0786db3b' -> '/vagrant/.npm/json-loader/0.5.4/package.tgz'

Since moving the npm cache directory to somewhere outside the shared folder, it now works:
npm config set cache /tmp/.npm-cache

@slashsteiner

This comment has been minimized.

Copy link

commented Jun 3, 2016

Encountered the same error and doing

npm install --no-bin-links

as what @matteius suggested works.

@samayo

This comment has been minimized.

Copy link

commented Jan 26, 2017

I could be wrong, or doing something entirely differnt, but doing:

npm install --no-bin-links

seems to strip you of the ability to require module, at-least globally. Doesn't it?

@BenLiyanage

This comment has been minimized.

Copy link

commented Feb 6, 2017

Just as an FYI--3.7+ fixed this for me. I believe we are on node 4.x at this point where everything is also all peachy.

@jonnyasmar

This comment has been minimized.

Copy link

commented Mar 28, 2017

This issue still exists in 4.1.1 :(

@matteius suggestion to use --no-bin-links does indeed seem to work, though I'm not sure I know what potential side-effects this has? I assume it does something that would differentiate the install from a standard one.

I can also confirm that @demonkoryu suggestion to move the npm cache directory also works (npm config set cache /tmp/.npm-cache), though I imagine this may have unexpected behavior if you try to use npm from the host machine?

EDIT:
I've actually come across an instance (trying to install nodemon) where modifying the npm cache does not work. This is also an instance where --no-bin-links causes an issue as the library cannot be executed if it's not linked in the .bin directory of node_modules.

As it stands, I'm of the impression this issue is not resolved and has no universally acceptable workaround :(

EDIT [SOLVED!]:
I was able to finally solve this issue by digging through the post located here: https://www.virtualbox.org/ticket/10085

To summarize:

Symlinking works in VirtualBox but is disabled by default "for security reasons". As such, you need to enable it manually and run VirtualBox as administrator. The simple steps are as follows:

  1. On the host machine (assuming Windows), but some variation of this will work in any OS: cd C:\Program Files\Oracle\VirtualBox

  2. Run the following command:
    VBoxManage setextradata VM_NAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/SHARE_NAME 1
    replacing VM_NAME with your Virtual Machine's name (if you don't know this, in VBox go to Machine > Settings > General > Basic > Name --- also replace SHARE_NAME with the name of your shared folder, if you don't remember this, go to Machine > Settings > Shared Folders.

  3. Restart your VM AND VirtualBox, but run it as administrator. On Windows hosts, just right click the icon for VirtualBox and select Run as Administrator.

Hope this helps!

luisnaranjo733 added a commit to ddmiller1995/InForm that referenced this issue Mar 29, 2017

tweak npm so it works on virtualbox vm
required to get past acorn npm install crash
was happening because vm doesn't support symlinks
reference: npm/npm#992
@jpmckearin

This comment has been minimized.

Copy link

commented Mar 31, 2017

@jonnyasmar you could not have been sent at a better time! Works great!

@BlueAccords

This comment has been minimized.

Copy link

commented Jun 22, 2017

@jonnyasmar Can also confirm this worked for me using windows 10 as the host os and ubuntu 16.04 as the guest os with virtualbox. Only difference was I had to prepend ./ to the command so it became ./VBoxManage setextradata VM_NAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/SHARE_NAME 1

@Cornpop456

This comment has been minimized.

Copy link

commented Jul 22, 2017

@jonnyasmar Thank you, everything works fine, host - windows 10, guest ubuntu 17.

@Lillebo

This comment has been minimized.

Copy link

commented Sep 19, 2017

@jonnyasmar Thank you! Can confirm it works with Windows 8.1 host, guest Debian 9.

@MehmetKaplan

This comment has been minimized.

Copy link

commented Oct 18, 2017

This seems closed by a workaround though the root cause of the problem is still at npm side.

As pointed out in the link that @jonnyasmar (precisely in this comment), VirtualBox does not allow sym links for security reasons. And the virtual machine should reside as a sandbox. Apparently the workaround seems as a security breach where an external software can place a link that can cause the host system to impact other directories that have no relation with the guest virtual machine.

Does anybody know why npm relies on symbolic links and why --no-bin-links does not function properly?

@samayo

This comment has been minimized.

Copy link

commented Oct 19, 2017

It needs it to load libraries from node_modules globaly. And this should run on you local machine so the security risk you mention is minimal. Also, if you install node you'll never need vbox, you can just run your node app without starting vbox itself

@MehmetKaplan

This comment has been minimized.

Copy link

commented Oct 19, 2017

@samayo, thank you for the answer.

Need for VirtualBox may rise from some other necessities.

On the other hand the major issue can be said, "why --no-bin-links does not solve the issue". Because it is the supposed to be solution where soft links are not appropriate.

@matteius

This comment has been minimized.

Copy link

commented Oct 19, 2017

I need VirtualBox because I enjoy using windows to move around my web browser tabs and Python IDE. I used to be able to use node on the shared file system more easily but now even with --no-bin-links it fails on the npm install and then I am forced to push files out to the local file system of the VM and run from within there before I can get the npm depdenencies working on the loaded page. Annoying for sure, and I would never want to run a Python app native on windows until they get the Ubuntu Linux subsystem for Windows on a more modern set of packages and python.

@samayo

This comment has been minimized.

Copy link

commented Oct 19, 2017

@matteius I don't understand. What is preventing you from using node on windows without Vbox?
I only use vbox when I am developing with PHP because I don't want to use wamp/xamp. If I am build a nodejs app as I am doing now (vue, webpack, babel, express) I don't need vbox at all.

@matteius

This comment has been minimized.

Copy link

commented Oct 19, 2017

@samayo Mainly large system differences ... the code I am developing will be run on Linux which supports things like POSIX threads. For a more concrete example of why it won't work, I use the async task framework celery and its not supported to run on windows anymore http://docs.celeryproject.org/en/latest/faq.html#does-celery-support-windows.

I am sure some would argue that not to use Windows, but as I point out I like the driver support and GUI support way better, or others would claim that anything could be worked around. The reality is using an Ubuntu VM allows me to test the code I am writing in the GUIs of my choice in an environment that the code will actually end up running in ... without workarounds.

The thing I end up working around these days is --no-bin-links not being as great as it used to be.

@samayo

This comment has been minimized.

Copy link

commented Oct 19, 2017

Well, then I guess you have to dual-boot windows or use docker

@matteius

This comment has been minimized.

Copy link

commented Oct 19, 2017

@samayo I think you are missing the point. Rebooting into a different OS is even less convenient than pushing files out the VM ... the issue is using a shared file system such as vboxfs which allows you to operate on files on the windows host system inside the VM ... so you edit on Windows it immediately shows on the VM. This is a good workflow for any non-npm based project but every UI with npm I end up having to push the files out to the local file system of the VM. The is equivalently the same as running docker which is an alternate virtualization technology. Its not like docker is going to solve the shared file system problem.

@samayo

This comment has been minimized.

Copy link

commented Oct 19, 2017

Yes, I know what vboxsf means or does. I use it everyday when working on php-related projects. But I have given up trying to enable symlinks with --no-bin-links for node project, but it was probably for the best since I now use node without starting any VM. Having said that, @jonnyasmar's solution worked for me once albeit with some minor issues. That is your last resort I think

@jonnyasmar

This comment has been minimized.

Copy link

commented Oct 19, 2017

@samayo --no-bin-links does resolve this issue in many instances. It only breaks down when your Node project references packages that are linked in node_modules/.bin. --no-bin-links is not meant to "enable" symlinks, its actually quite the opposite. According to npm docs:

The --no-bin-links argument will prevent npm from creating symlinks for any binaries the package might contain.

I could be totally off base, but I have to assume that the reason Node uses symlinks for the .bin is because recursively rifling through all of the directories in node_modules probably impacts performance, so it just creates an index as necessary during the install/update process. Telling npm to not create those symlinks is definitely not an inconsequential instruction.

As this thread has been made aware of now, due to the security "feature" implemented in VBox, your Guest OS will not follow the symlinks in .bin unless you explicitly enable this feature from the Host OS using the steps I outlined above (#992 (comment))

Anyways, glad to hear this worked for you @samayo, despite the issues. Good luck with the rest of your project!

@matteius

This comment has been minimized.

Copy link

commented Oct 19, 2017

Thanks @samayo and @jonnyasmar ... I totally missed the bit in Jons comment about setting the npm cache directory to be somewhere on the local VM. Example: npm config set cache ~/.npm-cache this helped me get past 90% of the npm install failures. My additional followup included:

  • installing latest node version
  • upgrading to latest npm version
  • removing npm from the packages.json in the project that was still failing (not sure why it was in there when npm installs things from that file)
  • finally I was bitten by a checked in hard link (that didn't work) pointing some static directory back up to the top level node_modules directory. I think most people's project would not have something like this, but once I did that I could actually compile the less files that depended on the node install that worked after changing the default npm cache directory.

Getting this working again has totally been worth having the discussion points on this thread. Thanks everyone!

@alex-left

This comment has been minimized.

Copy link

commented Oct 24, 2017

I solved creating a mounted folder in local storage (with guest linux):
sudo mount --bind /home/ubuntu/local-folder /vagrant

You can make it permanent adding in /etc/fstab of your VM:
/vagrant /home/ubuntu/local-folder none bind 0 0

@maxblax

This comment has been minimized.

Copy link

commented Dec 12, 2017

Take care, the fix that enable symlink seems to work, but is a bit confusing:
You have to actually setup the shared directory GUEST Name, aka. prefix

sf_SharedWithUbuntu

Only setting SharedWithUbuntu name folders ( HOST Folder ) Did not work for me.
Example for my case:

@DELL_TTM MINGW64 /d/Program Files

./VBoxManage.exe setextradata "Ubuntu 16 LTS" VBoxInternal2/SharedFoldersEnableSymlinksCreate/sf_SharedWithUbuntu 1

And don't forget to check that this ** VBoxManager program does not silent fail ;)

./VBoxManage.exe getextradata "Ubuntu 16 LTS" enumerate

which return

Key: GUI/LastCloseAction, Value: PowerOff
Key: GUI/LastGuestSizeHint, Value: 1920,954
Key: GUI/LastNormalWindowPosition, Value: 3840,27,1920,1053,max
Key: GUI/RestrictedRuntimeDevicesMenuActions, Value: HardDrives
Key: GUI/RestrictedRuntimeMachineMenuActions, Value: SaveState,PowerOff
Key: GUI/ScaleFactor, Value: 1
Key: GUI/StatusBar/IndicatorOrder, Value: HardDisks,OpticalDisks,FloppyDisks,Network,USB,SharedFolders,Display,VideoCapture,Feat
Key: VBoxInternal2/SharedFlodersEnableSymlinksCreate/SharedWithUbuntu, Value: 1
Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/SharedWithUbuntu, Value: 1
Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/sf_SharedWithUbuntu, Value: 1

Host: Windows 7
Guest Ubuntu 16 LTS
VirtualBoxManager: Version 5.1.30 r118389 (Qt5.6.2)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
You can’t perform that action at this time.