Corrupt synced folder cache using Vagrant 1.4.2 and VMWare provider #2768

Closed
danielbachhuber opened this Issue Jan 6, 2014 · 25 comments

Comments

Projects
None yet
@danielbachhuber
Contributor

danielbachhuber commented Jan 6, 2014

Using Vagrant 1.4.2, the VMWare provider (VMWare Version 6.0.2-1398658) and Ubuntu 13.04, I'm regularly experiencing corrupted synced folder caches when editing synced files using Sublime Text on my Mac.

Given this edit history on a sample file, the file consistently ends up as the following inside the VM (notice the incorrect file termination point):

image

Making another edit to the file (even whitespace) will fix the cached representation of the file. The issue can also be resolved by running echo 1 > /proc/sys/vm/drop_caches as sudo inside the VM.

My co-worker @joehoyle runs a similar setup with Vagrant 1.3.3 and hasn't ever experienced the issue. I haven't yet tested downgrading because I upgraded the VMWare provider which requires 1.4.0

Any suggestions as to what the cause might be, or what additional debugging I can do? I switched to VMWare because of caching issues associated with VirtualBox and NFS, and it's frustrating to say the least that one caching issue has been replaced by another.

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Jan 6, 2014

Member

This is a bug with VMware 6.0.2 guest additions. It is sporadic. Downgrade to fix.

Member

mitchellh commented Jan 6, 2014

This is a bug with VMware 6.0.2 guest additions. It is sporadic. Downgrade to fix.

@mitchellh mitchellh closed this Jan 6, 2014

@danielbachhuber

This comment has been minimized.

Show comment
Hide comment
@danielbachhuber

danielbachhuber Jan 6, 2014

Contributor

Thanks for the reply, @mitchellh. Downgrading to 6.0.1 didn't fix the problem. I've now downgraded to 5.0.4 and am seeing this issue:

[default] Starting the VMware VM...
An error occurred while executing `vmrun`, a utility for controlling
VMware machines. The command and output are below:

Command: ["start", "/Users/danielbachhuber/salty-wordpress/.vagrant/machines/default/vmware_fusion/5f9afd2d-5363-4869-95c2-6889cc7eb141/raring64.vmx", "nogui", {:notify=>[:stdout, :stderr]}]

Stdout: 2014-01-06T10:28:56.534| ServiceImpl_Opener: PID 916
Error: Unknown error

Stderr:

Destroying the box and re-provisioning doesn't resolve.

Contributor

danielbachhuber commented Jan 6, 2014

Thanks for the reply, @mitchellh. Downgrading to 6.0.1 didn't fix the problem. I've now downgraded to 5.0.4 and am seeing this issue:

[default] Starting the VMware VM...
An error occurred while executing `vmrun`, a utility for controlling
VMware machines. The command and output are below:

Command: ["start", "/Users/danielbachhuber/salty-wordpress/.vagrant/machines/default/vmware_fusion/5f9afd2d-5363-4869-95c2-6889cc7eb141/raring64.vmx", "nogui", {:notify=>[:stdout, :stderr]}]

Stdout: 2014-01-06T10:28:56.534| ServiceImpl_Opener: PID 916
Error: Unknown error

Stderr:

Destroying the box and re-provisioning doesn't resolve.

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Jan 6, 2014

Member

@danielbachhuber The bug is in the guest additions, not the host VMware.

Member

mitchellh commented Jan 6, 2014

@danielbachhuber The bug is in the guest additions, not the host VMware.

@danielbachhuber

This comment has been minimized.

Show comment
Hide comment
@danielbachhuber

danielbachhuber Jan 6, 2014

Contributor

Derp derp. Here are some related threads I came across:

Based on those threads, I then performed the following steps inside my VM:

  1. Checked my current version of VMWare Tools: vmware-config-tools.pl -h
VMware Tools 9.2.2 build-893683 configurator.
  1. Downloaded VMWare Tools for 6.0.1: cd /tmp; wget https://softwareupdate.vmware.com/cds/vmw-desktop/fusion/6.0.1/1331545/packages/com.vmware.fusion.tools.linux.zip.tar
  2. Extracted and mounted the ISO: tar -xvf com.vmware.fusion.tools.linux.zip.tar; unzip com.vmware.fusion.tools.linux.zip; mkdir /mnt/vmware-tools; mount -o loop linux.iso /mnt/vmware-tools/
  3. Ran the upgrader (I have a 64-bit box): cd /mnt/vmware-tools; ./vmware-tools-upgrader-64

However, after doing a vagrant reload, I seem to be back at the same version and build number of VMWare Tools. I haven't been able to reproduce the issue though, so maybe it's been fixed. I'll give it a few days before I make the determination.

Contributor

danielbachhuber commented Jan 6, 2014

Derp derp. Here are some related threads I came across:

Based on those threads, I then performed the following steps inside my VM:

  1. Checked my current version of VMWare Tools: vmware-config-tools.pl -h
VMware Tools 9.2.2 build-893683 configurator.
  1. Downloaded VMWare Tools for 6.0.1: cd /tmp; wget https://softwareupdate.vmware.com/cds/vmw-desktop/fusion/6.0.1/1331545/packages/com.vmware.fusion.tools.linux.zip.tar
  2. Extracted and mounted the ISO: tar -xvf com.vmware.fusion.tools.linux.zip.tar; unzip com.vmware.fusion.tools.linux.zip; mkdir /mnt/vmware-tools; mount -o loop linux.iso /mnt/vmware-tools/
  3. Ran the upgrader (I have a 64-bit box): cd /mnt/vmware-tools; ./vmware-tools-upgrader-64

However, after doing a vagrant reload, I seem to be back at the same version and build number of VMWare Tools. I haven't been able to reproduce the issue though, so maybe it's been fixed. I'll give it a few days before I make the determination.

@danielbachhuber

This comment has been minimized.

Show comment
Hide comment
@danielbachhuber

danielbachhuber Jan 6, 2014

Contributor

Hah. Now when I try to cd into a synced folder inside the VM my user session crashes. Fantastic.

Contributor

danielbachhuber commented Jan 6, 2014

Hah. Now when I try to cd into a synced folder inside the VM my user session crashes. Fantastic.

@danielbachhuber

This comment has been minimized.

Show comment
Hide comment
@danielbachhuber

danielbachhuber Jan 6, 2014

Contributor

Here's the call trace for the kernel panic (from VMWare GUI):

image

Contributor

danielbachhuber commented Jan 6, 2014

Here's the call trace for the kernel panic (from VMWare GUI):

image

@danielbachhuber

This comment has been minimized.

Show comment
Hide comment
@danielbachhuber

danielbachhuber Jan 7, 2014

Contributor

I did vagrant destroy to get back to square one. @joehoyle is on VMware Tools 9.6.0 build-1294478 with no issues, so I guess I'll attempt to find a copy of that version or earlier tomorrow and install it.

@bdcravens I see a similar report from you in #2533. Did you manage to work out a solution?

Contributor

danielbachhuber commented Jan 7, 2014

I did vagrant destroy to get back to square one. @joehoyle is on VMware Tools 9.6.0 build-1294478 with no issues, so I guess I'll attempt to find a copy of that version or earlier tomorrow and install it.

@bdcravens I see a similar report from you in #2533. Did you manage to work out a solution?

@danielbachhuber

This comment has been minimized.

Show comment
Hide comment
@danielbachhuber

danielbachhuber Jan 7, 2014

Contributor

Actually, I just switched to NFS after installing nfs-common inside the VM and it seems to work alright. We originally switched away from NFS because it had caching issues with VirtualBox. I did some quick testing and that particular caching issue doesn't seem to be manifesting itself.

Contributor

danielbachhuber commented Jan 7, 2014

Actually, I just switched to NFS after installing nfs-common inside the VM and it seems to work alright. We originally switched away from NFS because it had caching issues with VirtualBox. I did some quick testing and that particular caching issue doesn't seem to be manifesting itself.

@bdcravens

This comment has been minimized.

Show comment
Hide comment
@bdcravens

bdcravens Jan 7, 2014

@danielbachhuber No the underlying issue wasn't resolved, but like you, I switched to NFS and it seemed to resolve the behavior.

@danielbachhuber No the underlying issue wasn't resolved, but like you, I switched to NFS and it seemed to resolve the behavior.

@andyfowler

This comment has been minimized.

Show comment
Hide comment
@andyfowler

andyfowler Jan 7, 2014

Contributor

Just throwing this here in case it helps: my team was all having issues with NFS not updating files if the file length stayed the same. In our case, we were all OS X host, Ubuntu guest, Virtualbox provider, NFS synced folder. The fix was to stop using atomic saves in Sublime Text (which is the default, at least in ST3). In ST: Sublime Text > Preferences > Settings- User

{
    "atomic_save": false
}
Contributor

andyfowler commented Jan 7, 2014

Just throwing this here in case it helps: my team was all having issues with NFS not updating files if the file length stayed the same. In our case, we were all OS X host, Ubuntu guest, Virtualbox provider, NFS synced folder. The fix was to stop using atomic saves in Sublime Text (which is the default, at least in ST3). In ST: Sublime Text > Preferences > Settings- User

{
    "atomic_save": false
}
@joehoyle

This comment has been minimized.

Show comment
Hide comment
@joehoyle

joehoyle Jan 7, 2014

@andyfowler it's possible you just solved a very long running problem... will report back!

joehoyle commented Jan 7, 2014

@andyfowler it's possible you just solved a very long running problem... will report back!

@andyfowler

This comment has been minimized.

Show comment
Hide comment
@andyfowler

andyfowler Jan 7, 2014

Contributor

Please do! This one was killing us for months now. Trying to get the word out, because I know I've seen quite a few others with it... https://twitter.com/andyfowler/status/420605123568472065

Contributor

andyfowler commented Jan 7, 2014

Please do! This one was killing us for months now. Trying to get the word out, because I know I've seen quite a few others with it... https://twitter.com/andyfowler/status/420605123568472065

@danielbachhuber

This comment has been minimized.

Show comment
Hide comment
@danielbachhuber

danielbachhuber Jan 7, 2014

Contributor

Just throwing this here in case it helps: my team was all having issues with NFS not updating files if the file length stayed the same

I think this is just an issue with VirtualBox, though. I did some basic testing yesterday with VMWare and didn't run into this problem (but had it all the time with VB). I'll report back if I run into it again with VMWare.

Contributor

danielbachhuber commented Jan 7, 2014

Just throwing this here in case it helps: my team was all having issues with NFS not updating files if the file length stayed the same

I think this is just an issue with VirtualBox, though. I did some basic testing yesterday with VMWare and didn't run into this problem (but had it all the time with VB). I'll report back if I run into it again with VMWare.

@andyfowler

This comment has been minimized.

Show comment
Hide comment
@andyfowler

andyfowler Jan 7, 2014

Contributor

@danielbachhuber I've been doubtful that this is a VirtualBox issue, since it's really just the NFS connection between host and guest, which is all via network, and not any VirtualBox magic.

But to your original, our issue was when editing files that didn't change in size: I haven't experienced any actual corruption via NFS. I had a hell of a time using the VirtualBox native shared folders, though, which is what pushed us to NFS.

Contributor

andyfowler commented Jan 7, 2014

@danielbachhuber I've been doubtful that this is a VirtualBox issue, since it's really just the NFS connection between host and guest, which is all via network, and not any VirtualBox magic.

But to your original, our issue was when editing files that didn't change in size: I haven't experienced any actual corruption via NFS. I had a hell of a time using the VirtualBox native shared folders, though, which is what pushed us to NFS.

danielbachhuber added a commit to danielbachhuber/danielsublime that referenced this issue Jan 8, 2014

@danielbachhuber danielbachhuber referenced this issue in Varying-Vagrant-Vagrants/VVV Feb 4, 2014

Closed

NFS shares? #95

@astockwell

This comment has been minimized.

Show comment
Hide comment
@astockwell

astockwell Feb 17, 2014

Indeed @andyfowler you are correct, not just a VirtualBox issue! Was having the same issue with an Ubuntu guest (12.04 LTS) on VMWare (latest), OS X host (10.9.1), running ST3 (latest) and also a ruby (2.1.0) Guard post-processor. Narrowed it down to the VMWare guest literally not seeing updates to files within the shared folder when the file size didn't change. NFS seems to have solved it! Many thanks as this would have been a deal-breaker for MAMP-killing.

Indeed @andyfowler you are correct, not just a VirtualBox issue! Was having the same issue with an Ubuntu guest (12.04 LTS) on VMWare (latest), OS X host (10.9.1), running ST3 (latest) and also a ruby (2.1.0) Guard post-processor. Narrowed it down to the VMWare guest literally not seeing updates to files within the shared folder when the file size didn't change. NFS seems to have solved it! Many thanks as this would have been a deal-breaker for MAMP-killing.

@ladyrassilon

This comment has been minimized.

Show comment
Hide comment
@ladyrassilon

ladyrassilon Apr 9, 2014

I'm still seeing an issue with this, I'm just using conventional (not NFS) syncing, and the only way I can get the syncing to work is to do this every time I make a change.

sudo echo 1 > /proc/sys/vm/drop_caches

Every time I want to sync.

I don't get any problems with virtualbox, I'm using 1.5.1, but this has been a problem since 1.4.2 (I'm just going to test again with 1.5.2 to see if that works)

Its the reason I'm forced to use virtualbox as opposed to vmware.

I'm still seeing an issue with this, I'm just using conventional (not NFS) syncing, and the only way I can get the syncing to work is to do this every time I make a change.

sudo echo 1 > /proc/sys/vm/drop_caches

Every time I want to sync.

I don't get any problems with virtualbox, I'm using 1.5.1, but this has been a problem since 1.4.2 (I'm just going to test again with 1.5.2 to see if that works)

Its the reason I'm forced to use virtualbox as opposed to vmware.

@glucas

This comment has been minimized.

Show comment
Hide comment
@glucas

glucas Apr 18, 2014

Running Vagrant 1.5.3 on Windows, VMware Workstation, and an Ubuntu 12.04 guest. I'm (intermittently?) seeing synced folders get out of date. When this happens I can see the files up to date under /mnt/hgfs/-xxx, but the path where vagrant has mounted the folder is out of sync.

@ladyrassilon - thanks, at least this gives me a way to force the sync.

glucas commented Apr 18, 2014

Running Vagrant 1.5.3 on Windows, VMware Workstation, and an Ubuntu 12.04 guest. I'm (intermittently?) seeing synced folders get out of date. When this happens I can see the files up to date under /mnt/hgfs/-xxx, but the path where vagrant has mounted the folder is out of sync.

@ladyrassilon - thanks, at least this gives me a way to force the sync.

@nacht

This comment has been minimized.

Show comment
Hide comment
@nacht

nacht Apr 29, 2014

this ticket appears to be closed - but want to add to the fact that our organisation is seeing this. Currently running
Vagrant 1.5.2
OS X 10.9.2
VMWare Fusion Pro 6.0.2
with Ubuntu 12.04 guest

nacht commented Apr 29, 2014

this ticket appears to be closed - but want to add to the fact that our organisation is seeing this. Currently running
Vagrant 1.5.2
OS X 10.9.2
VMWare Fusion Pro 6.0.2
with Ubuntu 12.04 guest

@helderco

This comment has been minimized.

Show comment
Hide comment
@helderco

helderco Apr 30, 2014

@nacht try to upgrade your vmware to 6.0.3

@nacht try to upgrade your vmware to 6.0.3

@adambiggs

This comment has been minimized.

Show comment
Hide comment
@adambiggs

adambiggs May 2, 2014

@helderco VMWare 6.0.3 did nothing to solve the problem for me...

@helderco VMWare 6.0.3 did nothing to solve the problem for me...

@ladyrassilon

This comment has been minimized.

Show comment
Hide comment
@ladyrassilon

ladyrassilon May 3, 2014

Since Vmware (fusion) 6.0.3 I've had no folder syncing issues.

Did you start from a fresh vm?

Since Vmware (fusion) 6.0.3 I've had no folder syncing issues.

Did you start from a fresh vm?

@adambiggs

This comment has been minimized.

Show comment
Hide comment
@adambiggs

adambiggs May 5, 2014

I tried vagrant destroy/vagrant up.

I'm not 100% sure it's the same issue, but using VMWare's shared folder, I get all kinds of strange errors when trying to run composer for a Symfony2 project. But when switching to NFS, the problems go a way (leaving me with a different set of NFS problems...).

I tried vagrant destroy/vagrant up.

I'm not 100% sure it's the same issue, but using VMWare's shared folder, I get all kinds of strange errors when trying to run composer for a Symfony2 project. But when switching to NFS, the problems go a way (leaving me with a different set of NFS problems...).

@glenbot

This comment has been minimized.

Show comment
Hide comment
@glenbot

glenbot May 8, 2014

I still had this issue in VMware Fusion 6.0.3 and I wrote up a post on how I fixed it. http://codrspace.com/glenbot/fix-vagrant-vmhgfs-file-corruption-bug-in-vmware-6-0-2-and-6-0-3/

glenbot commented May 8, 2014

I still had this issue in VMware Fusion 6.0.3 and I wrote up a post on how I fixed it. http://codrspace.com/glenbot/fix-vagrant-vmhgfs-file-corruption-bug-in-vmware-6-0-2-and-6-0-3/

@atombender

This comment has been minimized.

Show comment
Hide comment
@atombender

atombender Feb 20, 2015

This issue should be reopened. Experiencing this problem with Virtualbox (latest, 4.3.22 r98236, image is Ubuntu 12.04 with kernel 3.2.0-23-generic) and ordinary vboxsf mounts. @ladyrassilon's manual VM cache drop workaround resolves the problem every time.

This issue should be reopened. Experiencing this problem with Virtualbox (latest, 4.3.22 r98236, image is Ubuntu 12.04 with kernel 3.2.0-23-generic) and ordinary vboxsf mounts. @ladyrassilon's manual VM cache drop workaround resolves the problem every time.

@rvdh

This comment has been minimized.

Show comment
Hide comment
@rvdh

rvdh Feb 29, 2016

I agree with @atombender - I'm experiencing this issue as well. Debian 8 image with latest VirtualBox Version 5.0.14 r105127.

rvdh commented Feb 29, 2016

I agree with @atombender - I'm experiencing this issue as well. Debian 8 image with latest VirtualBox Version 5.0.14 r105127.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment