long paths in the vagrant ubuntu disk on win7 causes problems with npm, etc. #1953

Closed
Pomax opened this Issue Jul 19, 2013 · 64 comments

Projects

None yet
@Pomax
Pomax commented Jul 19, 2013

From npm/npm#3670, if I run Vagrant on windows 7, the host max path length is 260 characters at the utter most without special instructions/commands, so when running something that has to make deep directories, like checking out a large software project or doing an npm install, causes "protocol errors".

For npm in particular this is a problem; on native windows npm can make use of UNC paths to get around the path limit, and it works brilliant, but inside a vagrant VM the "protocol error" means projects can't be instantiated, and consequently not worked with, effectively negating the benefit that vagrant offers.

I don't know if this is something that can be fixed in Vagrant itself (maybe?) or whether it's a virtualbox issue (I have no other virtualisation software to test that with, unfortunately), but if this can be fixed, that would be amazingly useful for software project virtualisation/testing.

@mitchellh
Owner

Unfortunately this is very likely an upstream issue. Can you check with VirtualBox?

@mitchellh mitchellh closed this Jul 19, 2013
@Pomax
Pomax commented Jul 19, 2013

Filed https://www.virtualbox.org/ticket/11976 against VirtualBox, hopefully this can get fixed =)

@murilobr
murilobr commented Nov 5, 2013

@Pomax I'm with the same problem. Windows is for users not for developers.

@Pomax
Pomax commented Nov 5, 2013

? Not sure what you meant, but if you're not testing your stuff on OSX, Windows and at least Ubuntu, you're not doing your job as a serious developer =)

@murilobr
murilobr commented Nov 6, 2013

@Pomax this problem occurs just with Windows. Im using Windows at work (no chance to change). I want to say with last comment is that Windows is not good.

@Pomax
Pomax commented Nov 6, 2013

still not sure whether you're saying whether this is a problem only for windows (which is true, it's caused by windows path length limitations), and that that's a bad thing, implying it should be fixed, or whether you're saying that windows itself is just plain bad and implying we shouldn't bother fixing things for it because it's just plain bad =)

(I'd agree with the first, I'd strongly disagree with the second)

@lynnaloo
lynnaloo commented Feb 3, 2014

@Pomax How were you able to get past this issue? I was able to complete my npm install by running Vagrant as Admin on Windows, but some of my submodules have deep paths that are now causing issues.

@Pomax
Pomax commented Feb 3, 2014

I wasn't. I gave up after filing the bug on VirtualBox and Oracle ignoring it ever since then. It might at some point get fixed, but I'm not holding my breath. Instead out Vagrant VM now doesn't set up synced folders until you set up the local dir <-> /vagrant mapping yourself after running the box install. That way we don't insta-break things in windows, and on OSes that support it you can set up syncing. It's about as far from ideal as possible, but having something that works for some is preferable to something that doesn't work for all.

@jhnns
jhnns commented May 20, 2014

Having the same trouble ... long paths are a PITA on windows.

@lynnaloo

If I do the submodule update the first time on the Windows host then I seem to be fine after that.

@henricook

Also struggling with this, such a PITA that Oracle are just ignoring this!

@lynnaloo

I actually haven't run across this issue recently. I was having a similar issue and it turned out that a reboot of the Windows machine fixed the npm install. Fingers crossed.

@henricook

I had some relief when i started working out of C:\R (a one letter directory) on my host - but i have suitable nested node_modules folders on npm install that it eventually became a problem again

I think it's fundamentally a windows limitation, i hope i'm wrong, but i don't think Win7 supports > 256 characters in a path - i'd love some official virtualbox confirmation

@Pomax
Pomax commented Jul 19, 2014

Windows 7 very much supports paths over 256 (up to 32767, in fact), you just need to tell windows you live in unicode land, not in ascii land. The windows API has separate calls for creating huge paths, and I'm pretty sure virtualbox doesn't bother calling the unicode-enabled rather than ancient-ascii functions. see here ("Maximum Path Length Limitation" section)

@henricook

Great knowledge Mike, thanks for the links - here's hoping someone at
Virtualbox wakes up to this :-(

On 19 July 2014 23:46, Mike Kamermans notifications@github.com wrote:

Windows 7 very much supports paths over 256, you just need to tell windows
you live in unicode land, not in ascii land. The windows API has separate
calls for creating huge paths, and I'm pretty sure virtualbox doesn't
bother calling those. see here
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#short_vs._long_names
("maximum pathlength limitations"


Reply to this email directly or view it on GitHub
#1953 (comment).

--------------------------------------
Henri Cook
*Senior Software Developer *
*Technical Lead - Content Services *

http://www.rightster.com

Rightster Plc | Third Floor, 1 Neal Street, London, WC2H 9QL

@jhnns
jhnns commented Jul 21, 2014

In the meantime there is a workaround. It's a script which symlinks node_modules to the unix home folder where long pathnames are no problem.

See also this related discussion.

@Indemnity83

There is a comment added to the Virtualbox ticket a few days ago that provides a solution that Vagrant may be able to implement:

A workaround is to remove the original share and then use VBoxManage to add it back while prepending \?\ to the host path, e.g.

  VBoxManage.exe sharedfolder add "C:\Users\<username>\VirtualBox VMs\<vm-name>\<vm-name>.vbox" --name "MyStuff" --hostpath "\\?\C:\MyStuff" 

-- ticket/11976#comment:2

@jhnns
jhnns commented Oct 28, 2014

Nice!
Switched to OS X in the meantime and everything is running smooth 😝

@svengerlach

+1 for integrating the workaround mentioned in the virtualbox ticket into vagrant

@Pomax
Pomax commented Nov 11, 2014

if only switching to OSX didn't come with a $1000+ pricetag for the mac you need to legally run it =P

And yeah, if the share rebinding with UNC path prefixing could be done by Vagrant as part of the vagrant creation process, that would be super useful.

@chernetsov0

@svengerlach go support us at #4815 ;)

@pussinboots pussinboots added a commit to pussinboots/angularjs-crypto that referenced this issue Jan 23, 2015
@pussinboots pussinboots Workaround for long paths on win7 with virtualbox
For a description see here mitchellh/vagrant#1953 and the committed
script can be found here https://github.com/ericprieto/win-unix-vm-long-pathnames-workaround.
891d3a7
@drm2
drm2 commented Mar 18, 2015

I've spent about 16 hours working on this today (I wish I was exaggerating), and I finally have a solution to the Vagrant/Windows issue at hand. I have been working on a Node.js development environment built on Vagrant, and I have been relentlessly determined to get this working.

This is the commit that I implemented it in, but I'll go ahead and post the main snippet of code here:

config.vm.provider "virtualbox" do |v|
    v.customize ["sharedfolder", "add", :id, "--name", "www", "--hostpath", (("//?/" + File.dirname(__FILE__) + "/www").gsub("/","\\"))]
end

config.vm.provision :shell, inline: "mkdir /home/vagrant/www"
config.vm.provision :shell, inline: "mount -t vboxsf -o uid=`id -u vagrant`,gid=`getent group vagrant | cut -d: -f3` www /home/vagrant/www", run: "always"

In the code above, I am appending \\?\ to the current directory absolute path. This will actually force the Windows API to allow an increase in the MAX_PATH variable (normally capped at 260). Read more about max path. This is happening during the sharedfolder creation which is intentionally handled by VBoxManage and not Vagrant's "synced_folder" method. The last bit is pretty self-explanatory; we create the new shared folder and then make sure it's mounted each time the machine is accessed or touched since Vagrant likes to reload its mounts/shared folders on each load.

I absolutely hope this helps everyone that was struggling with this like I was!

@c7tincu
c7tincu commented Apr 20, 2015

@drmyersii Thanks so much for posting this. Saved my day! 👍

@thebeline

@drmyersii - Being new to Vagrant, how would I execute your fix only when executing using VirtualBox?

I understand that to dissable a Vagrant "synced_folder" I would config.vm.synced_folder "./", "/var/www/#{$project}", disabled: true, so I imagine a full solution would look like:

# If VirtualBox, would this look like:
# config.vm.provider "virtualbox" do |v| its self?
  # Dissable the old solder
  config.vm.synced_folder "./", "/var/www/#{$project}", disabled: true

  config.vm.provider "virtualbox" do |v|
    v.customize ["sharedfolder", "add", :id, "--name", "project", "--hostpath", (("//?/" + File.dirname(__FILE__) ).gsub("/","\\"))]
  end

  config.vm.provision :shell, inline: "mount -t vboxsf -o uid=`id -u vagrant`,gid=`getent group vagrant | cut -d: -f3` project /var/www/#{$project}", run: "always"

# End if VirtualBox, again, just:
# end

What would the conditional look like? Or would I wrap it all in config.vm.provider "virtualbox" do |v|?

Sorry if this is really basic.

@drm2
drm2 commented Apr 22, 2015

Hello, @thebeline!

You probably want to check out override in the vagrant documentation. This allows you to override non-provider specific configuration options. Good luck!

http://docs.vagrantup.com/v2/providers/configuration.html

@thebeline

Thanks.

@thebeline

@drmyersii - This does not seem to be working, specifically the shell execution. Can you have a look? Thanks.

@justechn
justechn commented May 6, 2015

@drmyersii - I'm having a hard time getting this to work

  # settings for VirtualBox provider
  config.vm.provider "virtualbox" do |v, override|
      override.vm.synced_folder "./", "/vagrant", disabled: true

      v.customize ["sharedfolder", "add", :id, "--name", "vagrant", "--hostpath", (("//?/" + File.dirname(__FILE__)).gsub("/","\\"))]

      override.vm.provision :shell, inline: "mkdir /vagrant"
      override.vm.provision :shell, inline: "mount -t vboxsf -o uid=`id -u vagrant`,gid=`getent group vagrant | cut -d: -f3` vagrant /vagrant", run: "always"
  end

When I open virtualbox and look at the shared folders it looks correct, but the share is not working (my shared dir on the guest is empty)

Does anything look wrong in my setup?

@nicocrm nicocrm referenced this issue in Varying-Vagrant-Vagrants/VVV May 7, 2015
Closed

Error in grunt build - new install #641

@Pomax
Pomax commented May 7, 2015

This got fixed in #5495

As a note to all, a patch for this behaviour was been merged in, so if you found this issue looking for a way to enable long pathname support: the next version of Vagrant should no longer suffer from this problem. Until then, you should be able to apply the same fix to your vagrant files.

@chernetsov0

@Pomax thanks for fixing this, mate! Appreciate it.

@Pomax
Pomax commented May 13, 2015

credit goes to @jfbibeau and everyone who helped review and land

@justechn

Any idea on when the next version of Vagrant is going to be available? Wondering if it is worth patching out my whole team or wait for the update?

@Pomax
Pomax commented May 13, 2015

patch first, wait second. Worst case the update does exactly the same as your manual patching, and in the mean time you have a working Vagrant.

@drm2
drm2 commented May 13, 2015

I second @Pomax. It's honestly not much work (as you can see from my comment above), and it would be just as easy to remove once you update.

@justechn

@drmyersii - As you could see from my previous message I could not get your fix to work. However, after I applied the official fix from #5495 everything works great.

@drm2
drm2 commented May 15, 2015

@ryanmc2033 - Oh! Sorry about that! I didn't even see that you were having issues with my solution. I will take a look at your use case sometime today to see if it's possibly just a mix up in your configuration. I definitely recommend using the official patch if you can though. My solution was just a way to get it working without having to re-compile the source.

@justechn

@drmyersii - maybe I did something different, but I didn't have to recompile the source. I just found the two files in my install directory, applied the changes in the commit. Then did a destroy and up in vagrant and it worked. Or at least it appears to be working, at least the problems I was having have gone away, but I will have to triple check to make sure I am not imagining things.

@Jakobud
Jakobud commented Aug 11, 2015

Did this definitely get fixed in 1.7.4? I experienced a similar issue tonight installing some npm modules. It threw all sorts of errors when installing in the shared directory, but outside the shared directory (completely within the guest filesystem) it installed just fine with no errors.

@jfbibeau
Contributor

It got fixed in 1.7.3, but reverted in 1.7.4 as it was causing issues with some versions of VirtualBox. Check out #5933 I believe.

@Jakobud
Jakobud commented Aug 11, 2015

Oh dang no wonder. I thought this was fixed. Okay so any clue when this is going to make it back into a release?

@Pomax
Pomax commented Aug 11, 2015

that is unfortunate. Leaving it in 1.7.4 would have forced the virtualbox team to actually get off their ass and fix something for a change =(

@Jakobud
Jakobud commented Aug 11, 2015

Yeah bummer. I guess it's back to the good old manually-mounted synced folder workaround:

https://gist.github.com/Jakobud/0768ff6b6051e79eef60

@Pomax
Pomax commented Aug 11, 2015

it would have been more sensible to make it a feature flag, rather than ripping out the fix, so that Windows users not using virtualbox 5 could have used vagrant up --unc or something, thus solving the problem for everyone.

@msimkunas

@Jakobud Does this solution work with samba shares on Windows? I'd like to be using something that doesn't slow down performance to a crawl.

@Jakobud
Jakobud commented Aug 18, 2015

Samba shares..... So what do you mean, you running a Linux guest with a samba share that your Windows host is connecting to?

@msimkunas

I'm running a Linux guest on a Windows host and I wish to improve filesystem performance. Isn't samba the only way for Windows hosts?

-----Pradinis laiškas-----
Nuo: "Jake Wilson" notifications@github.com
Išsiųsta: ‎2015-‎08-‎18 07:41
Kam: "mitchellh/vagrant" vagrant@noreply.github.com
Kopija: "Mantas Šimkūnas" mantas.simkunas@gmail.com
Tema: Re: [vagrant] long paths in the vagrant ubuntu disk on win7 causesproblems with npm, etc. (#1953)

Samba shares..... So what do you mean, you running a Linux guest with a samba share that your Windows host is connecting to?

Reply to this email directly or view it on GitHub.

@Jakobud
Jakobud commented Aug 18, 2015

No no, you need to use Vagrant's sync'ed folders:

http://docs.vagrantup.com/v2/synced-folders/basic_usage.html

You end up with folders on your host that are automatically and instantly synced with folders on the guest system.

@Jakobud
Jakobud commented Aug 18, 2015

Oh okay. Why not use regular synced folders? Seems like smb would be only useful if you had multiple physical computers you wanted to connect to a single guest environment. If you are just using one Windows host and one guest, normal synced folders I would think would be great. There is obviously the above mentioned 260-character bug issue, but the gist I posted gets around the problem.

@msimkunas

@Jakobud A shared folder without NFS/SMB is very slow. ag/grep/etc performance is almost unbearable while fuzzy searching a bigger project in vim. I wanted to go for a more performant solution, but I simply gave up on this, because Windows is just so difficult to work with in this regard.

The only way I can work without hassle is by running my projects on a folder inside the guest machine. I'm using a Linux VM with a window manager, chromium, firefox and all other GUI tools that help me forget about the Windows host completely. I wish there was a simple way to alleviate this problem though.

@msimkunas

@Jakobud I could probably use rsync, but I couldn't manage to get it to work properly (could be just me though...). Either way, I just don't like the idea of duplicating files on my machine.

@Jakobud
Jakobud commented Aug 18, 2015

Hmmm, my synced folders sync instantly from what I can see. There is no lag.

@vmadman
vmadman commented Sep 1, 2015

This is killing me... I was on Vagrant 1.7.1 and I've been having the problem for what seems over a year. I get all excited with all of the "fixed" talk above so I upgrade to Vagrant 1.7.4 .. only to realize that it was fixed then removed. The Vagrantfile workaround above looks complicated and I could not immediately see how I could apply it to my situation... idk what Virtualbox is doing on their end.

Can we at least get the fix put back into Vagrant with some sort of special flag, as @Pomax suggested? (vagrant up --unc)

I know its not perfect, but considering the alternatives its marvelous and magical by contrast.. at least its a viable workaround! Please?

@sergeikretov

Still having the same issue on Win 8.1 pro with VirtualBox 4.3.30 and vagrant 1.7.2. Is it going to be fixed? Cause I tried also VirtualBox 5.0.4 and vagrant 1.7.4, but same stuff and no luck.

@Pomax
Pomax commented Sep 16, 2015

that's becuase it only works with virtualbox <5 and vagrant 1.7.3 -- the fix was landed in 1.7.3 and then broke incredibly hard with virtualbox 5, so got rolled back.

@vmadman
vmadman commented Sep 17, 2015

Yeah, I don't think they care much about this -- its been going on for a very long time. However, I reverted to 1.7.3 and it has been working so far.

@sergeikretov

VirtualBox 4.3.30 + Vagrant 1.7.3 combo worked fine to me, thank you. But this fix really needs to be in the next vagrant version and support virtualbox 5.

@Pomax
Pomax commented Sep 17, 2015

I honestly couldn't care less about virtualbox 5, really. I just want this patch back in, under a runtime flag, so it can be used as necessary. And then Virtualbox could sort out their own mess and actually maybe fix something for a change.

@dhampik
dhampik commented Oct 14, 2015

@mitchellh This issue should probably be reopened, since fix in version 1.7.3 is reverted in 1.7.4 and basically the only working version is Virtualbox 4.3 + Vagrant 1.7.3.

I heard some people are unable to install Virtualbox 4.3 on Windows 10, so they cannot use Vagrant at all for their needs and that issue is a blocker. Maybe we should cooperate with Virtualbox guys to solve that?

@Ingramz Ingramz referenced this issue Oct 18, 2015
Closed

Support for long share paths on Windows hosts #6409

4 of 4 tasks complete
@Perni1984
  • the UNC path fix is back in the 1.8 Version of Vagrant which should be available soon, in the meantime one can add the PR by Hand (#6493)
  • a suitable way to use npm inside an ubuntu/vivid64 box on Windows 10, Vagrant 1.7.4 and VirtualBox 5.0.10 is to use smb for the synced Folders, force smb protocol Version 3.02 and add emulated symlink compatability through mfsymlinks. After this, my Setup with npm works absolutely fast and flawlessly:
config.vm.synced_folder "project", "/var/www", type: "smb", mount_options: ["vers=3.02","mfsymlinks"]

important: the guest OS must have a Linux kernel Version > 3.18, otherwise mfsymlinks is not working!

@Jakobud
Jakobud commented Dec 20, 2015

FYI with Node version 5.x, npm is different than it used to be. Now all dependencies are installed in a flat folder structure instead of having endlessly nested dependency folders in node_modules. So it makes it a lot easier on Vagrant installs in Windows even without the UNC fix.

@Perni1984

@Jakobud: while that's true, I still experienced some weird problems in Windows for some of my node.js apps (especially my gulpfile related dependencies of certain Projects) without mfsymlinks and the UNC fix.

@dantheman213

Hey guys, I created a script to fix this issue. My solution was to execute the project in a part of the VM that wasn't being shared with Windows. The script can be run over and over again to run a project. It copies the base project to a temporary execution directory. If you're running it 2nd time + it checks for file differences and will run npm install if your "package.json" file has changed. Lastly it runs npm install inside the execution directory and voila! Workaround complete. :)

Check it out here guys:
https://gist.github.com/dantheman213/7a505004d26c480e8274

@fchastanet

Thanks dantheman213, very good idea

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment