New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rsync-push and rsync-pull #3062

Closed
p0deje opened this Issue Mar 5, 2014 · 81 comments

Comments

Projects
None yet
@p0deje
Contributor

p0deje commented Mar 5, 2014

Until Vagrant 1.5 is released, I suggest to add one more feature to vagrant rsync. Currently, it's only possible to push local files to VM, but there is no way to pull them back. It'd be super useful for CI environments, when we need to retrieve test results and artifacts. It'd be pretty easy to implement (by changing position of local_path remote_path arguments).

It would also make sense to change commands to something like:

$ vagrant rsync-push
$ vagrant rsync-pull

I can work on this if you think it worths.

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Mar 5, 2014

Member

I think adding just rsync-pull would work! Vagrant 1.5 is pretty much wrapped up for features but I'd be happy to look at a pull to get it in in a 1.5.1 or something.

Member

mitchellh commented Mar 5, 2014

I think adding just rsync-pull would work! Vagrant 1.5 is pretty much wrapped up for features but I'd be happy to look at a pull to get it in in a 1.5.1 or something.

@p0deje

This comment has been minimized.

Show comment
Hide comment
@p0deje

p0deje Mar 5, 2014

Contributor

Maybe then vagrant rsync --pull?

Contributor

p0deje commented Mar 5, 2014

Maybe then vagrant rsync --pull?

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Mar 5, 2014

Member

Perfect.

Member

mitchellh commented Mar 5, 2014

Perfect.

@smerrill

This comment has been minimized.

Show comment
Hide comment
@smerrill

smerrill Apr 5, 2014

Contributor

If you'd like to try this functionality out, I made a gem that does (manual) pulls from the guest to the host. Check out https://github.com/smerrill/vagrant-rsync-back if you'd like to help test.

The gem itself is mostly implemented as a monkeypatch to RsyncHelper, so if people have good experiences with it, I can put together a pull request.

Contributor

smerrill commented Apr 5, 2014

If you'd like to try this functionality out, I made a gem that does (manual) pulls from the guest to the host. Check out https://github.com/smerrill/vagrant-rsync-back if you'd like to help test.

The gem itself is mostly implemented as a monkeypatch to RsyncHelper, so if people have good experiences with it, I can put together a pull request.

@sidneygijzen

This comment has been minimized.

Show comment
Hide comment
@sidneygijzen

sidneygijzen Apr 26, 2014

+1 for this feature.

I' ve also tested a little bit the vagrant-rsync-back plugin from @smerrill. Works fine so far on Arch Linux with Vagrant 1.5.4.

+1 for this feature.

I' ve also tested a little bit the vagrant-rsync-back plugin from @smerrill. Works fine so far on Arch Linux with Vagrant 1.5.4.

@rails-novice

This comment has been minimized.

Show comment
Hide comment
@rails-novice

rails-novice Apr 30, 2014

been working with vagrant-rsync-back by @smerrill (thanks) for a few weeks now, haven't noticed any issues so far with 1.5.2. Would be nice to have the watcher integrate this feature, it would improve the workflow a lot (stop watcher, rsync-back, start watcher).

been working with vagrant-rsync-back by @smerrill (thanks) for a few weeks now, haven't noticed any issues so far with 1.5.2. Would be nice to have the watcher integrate this feature, it would improve the workflow a lot (stop watcher, rsync-back, start watcher).

@p0deje

This comment has been minimized.

Show comment
Hide comment
@p0deje

p0deje May 7, 2014

Contributor

For now we're using smth like this in our CI run script:

vagrant ssh-config > vagrant-ssh-config
rsync --archive -z --exclude .vagrant/ --exclude Vagrantfile --exclude .git --exclude tmp --exclude .bundle -e 'ssh -F vagrant-ssh-config' default:/vagrant/ .
rm vagrant-ssh-config
Contributor

p0deje commented May 7, 2014

For now we're using smth like this in our CI run script:

vagrant ssh-config > vagrant-ssh-config
rsync --archive -z --exclude .vagrant/ --exclude Vagrantfile --exclude .git --exclude tmp --exclude .bundle -e 'ssh -F vagrant-ssh-config' default:/vagrant/ .
rm vagrant-ssh-config
@mrkeyboard

This comment has been minimized.

Show comment
Hide comment
@mrkeyboard

mrkeyboard May 11, 2014

Would it be possible to have pull included two ways using vagrant rsync-auto?

Would it be possible to have pull included two ways using vagrant rsync-auto?

@starrychloe

This comment has been minimized.

Show comment
Hide comment
@starrychloe

starrychloe Jun 25, 2014

Consider Unison: http://www.cis.upenn.edu/~bcpierce/unison/

I don't recommend using rsync for bidirectional synch. It is not a trivial task. Here is background info (or pattern) for doing master-master synchronization, in the context of databases, but it applies to files as well: http://msdn.microsoft.com/en-us/library/ff650702.aspx

You have to keep track of deleted files and conflicts, and that is what Unison does with 'shadow tables', only it keeps databases of the files in ~/.unison.

I've gotten around the problem of one-way rsync for commands that need to be executed in the guest but write files that need to be available on the host with

$ (cd /vagrant/project; rake db:migrate)

This changes to the default vboxfs (which is dog slow), does the command, writes to guest disk, synchs to host disk, then rsync will synch back to the /rsync synched folder.

Consider Unison: http://www.cis.upenn.edu/~bcpierce/unison/

I don't recommend using rsync for bidirectional synch. It is not a trivial task. Here is background info (or pattern) for doing master-master synchronization, in the context of databases, but it applies to files as well: http://msdn.microsoft.com/en-us/library/ff650702.aspx

You have to keep track of deleted files and conflicts, and that is what Unison does with 'shadow tables', only it keeps databases of the files in ~/.unison.

I've gotten around the problem of one-way rsync for commands that need to be executed in the guest but write files that need to be available on the host with

$ (cd /vagrant/project; rake db:migrate)

This changes to the default vboxfs (which is dog slow), does the command, writes to guest disk, synchs to host disk, then rsync will synch back to the /rsync synched folder.

@karanr

This comment has been minimized.

Show comment
Hide comment
@karanr

karanr Jun 27, 2014

+1 I couldn't agree more with @laurentlemaire, have been watching this issue since March 14 because it's a killer feature for web developers, hope it makes it to the next release of vagrant :) ; BTW i know this might not be the most appropriate place, however, a sincere thanks to @mitchellh for vagrant

karanr commented Jun 27, 2014

+1 I couldn't agree more with @laurentlemaire, have been watching this issue since March 14 because it's a killer feature for web developers, hope it makes it to the next release of vagrant :) ; BTW i know this might not be the most appropriate place, however, a sincere thanks to @mitchellh for vagrant

@matthew-dean

This comment has been minimized.

Show comment
Hide comment
@matthew-dean

matthew-dean Jul 24, 2014

Blarg, two-way sync worked so well (and automatically, without the need for rsync-auto) in 1.4.x. The current one-way sync just doesn't make any sense from a user perspective. I care little about performance when syncing, I just want to "vagrant up" and forget it.

Is there a way to revert to the previous shared-folder way of doing things?

If not, +1 to get back to some two-way syncing sanity.

Blarg, two-way sync worked so well (and automatically, without the need for rsync-auto) in 1.4.x. The current one-way sync just doesn't make any sense from a user perspective. I care little about performance when syncing, I just want to "vagrant up" and forget it.

Is there a way to revert to the previous shared-folder way of doing things?

If not, +1 to get back to some two-way syncing sanity.

@jdhiro

This comment has been minimized.

Show comment
Hide comment
@jdhiro

jdhiro Jul 24, 2014

@matthew-dean shared folder sync'ing wasn't removed, we are still using it.

This is a feature request for the people who can't/won't use shared folders because virtualbox has the worlds slowest implementation. Some of our devs see 1+ minute load times with vb shared folders.

(Personally, I'm using the Parallels provider, and shared folders are blazing fast.)

jdhiro commented Jul 24, 2014

@matthew-dean shared folder sync'ing wasn't removed, we are still using it.

This is a feature request for the people who can't/won't use shared folders because virtualbox has the worlds slowest implementation. Some of our devs see 1+ minute load times with vb shared folders.

(Personally, I'm using the Parallels provider, and shared folders are blazing fast.)

@matthew-dean

This comment has been minimized.

Show comment
Hide comment
@matthew-dean

matthew-dean Jul 24, 2014

Hmm, ok. Maybe there's just something wrong with how I have it set up then, but syncing used to work before I upgraded Vagrant from 1.4.3 to 1.6.3. How do you use the parallels provider?

Hmm, ok. Maybe there's just something wrong with how I have it set up then, but syncing used to work before I upgraded Vagrant from 1.4.3 to 1.6.3. How do you use the parallels provider?

@jdhiro

This comment has been minimized.

Show comment
Hide comment
@jdhiro

jdhiro Jul 24, 2014

@matthew-dean https://github.com/Parallels/vagrant-parallels

If you happen to have a copy of Parallels, I highly recommend it. It blows away virtualbox.

jdhiro commented Jul 24, 2014

@matthew-dean https://github.com/Parallels/vagrant-parallels

If you happen to have a copy of Parallels, I highly recommend it. It blows away virtualbox.

@smerrill

This comment has been minimized.

Show comment
Hide comment
@smerrill

smerrill Jul 28, 2014

Contributor

@khromov It has not been removed; you may still use NFS or VirtualBox/VMware shared folders.

Contributor

smerrill commented Jul 28, 2014

@khromov It has not been removed; you may still use NFS or VirtualBox/VMware shared folders.

@jenmo917

This comment has been minimized.

Show comment
Hide comment
@jenmo917

jenmo917 Jul 29, 2014

I use the following solution as a workaround to get a two-way sync with acceptable performance. First, use the standard VirtualBox shared folders and sync to /vagrant. Then use Unison inside the VM, to sync between /vagrant and /var/www. I made a simple shell script that runs the Unison sync every 3 sec.

<-VirtualBox shared folder-> /vagrant <-Unison file sync-> /var/www

Maybe it's not optimal but I see no other solution right now.

I use the following solution as a workaround to get a two-way sync with acceptable performance. First, use the standard VirtualBox shared folders and sync to /vagrant. Then use Unison inside the VM, to sync between /vagrant and /var/www. I made a simple shell script that runs the Unison sync every 3 sec.

<-VirtualBox shared folder-> /vagrant <-Unison file sync-> /var/www

Maybe it's not optimal but I see no other solution right now.

@duro

This comment has been minimized.

Show comment
Hide comment
@duro

duro Jul 29, 2014

Contributor

@jenmo917 That is an interesting solution, and it actually may help solve some of my problems. My biggest complaint with all current two-way syncing solutions is that they do not trigger OS file system events at the final destination in the VM (which I use to trigger build watchers like Grunt/Gulp). Rsync does a real write to disk, so the events are triggered, but it has been a real challenge to get it working two-way (and by challenge, I mean impossible). If your Unison solution works the same way as rsync at the final destination folder, then I may be able to use it.

Once question, is there a reason that you only run Unison every 3 seconds? Could it be run more frequently?

Contributor

duro commented Jul 29, 2014

@jenmo917 That is an interesting solution, and it actually may help solve some of my problems. My biggest complaint with all current two-way syncing solutions is that they do not trigger OS file system events at the final destination in the VM (which I use to trigger build watchers like Grunt/Gulp). Rsync does a real write to disk, so the events are triggered, but it has been a real challenge to get it working two-way (and by challenge, I mean impossible). If your Unison solution works the same way as rsync at the final destination folder, then I may be able to use it.

Once question, is there a reason that you only run Unison every 3 seconds? Could it be run more frequently?

@kylemacfarlane

This comment has been minimized.

Show comment
Hide comment
@kylemacfarlane

kylemacfarlane Jul 29, 2014

I've found that for front end development (i.e. sass, js, etc) that Virtualbox shared folders are plenty fast. The only problem as @duro points out is that they don't fire inotify events (and neither does Samba). I use my own build scripts and make sure to use regular file system polling to detect changes (e.g. with Watchdog use watchdog.observers.polling.PollingObserver). For frontend stuff where you only have say 50 files this is super fast and well under a second. I don't know if it's possible to change the type of watcher in Grunt or Gulp but that's what would need to be done.

Now for backend development where I have tens of thousands of files, shared folders and file system polling really starts to fall apart and can take anywhere from 3 to 60 seconds to detect changes (not to mention high CPU usage). If I'm doing a lot of this I just reboot and switch to rsync-auto.

I've found that for front end development (i.e. sass, js, etc) that Virtualbox shared folders are plenty fast. The only problem as @duro points out is that they don't fire inotify events (and neither does Samba). I use my own build scripts and make sure to use regular file system polling to detect changes (e.g. with Watchdog use watchdog.observers.polling.PollingObserver). For frontend stuff where you only have say 50 files this is super fast and well under a second. I don't know if it's possible to change the type of watcher in Grunt or Gulp but that's what would need to be done.

Now for backend development where I have tens of thousands of files, shared folders and file system polling really starts to fall apart and can take anywhere from 3 to 60 seconds to detect changes (not to mention high CPU usage). If I'm doing a lot of this I just reboot and switch to rsync-auto.

@kylemacfarlane

This comment has been minimized.

Show comment
Hide comment
@kylemacfarlane

kylemacfarlane Jul 29, 2014

I don't use Grunt but perhaps something like this alternate watcher is what people need: https://github.com/unbalanced/grunt-simple-watch

I don't use Grunt but perhaps something like this alternate watcher is what people need: https://github.com/unbalanced/grunt-simple-watch

@jenmo917

This comment has been minimized.

Show comment
Hide comment
@jenmo917

jenmo917 Jul 30, 2014

@duro I use 3 sec because it's enough for my purpose (PHP development in SugerCRM). When I save something in my IDE I have to wait approximately 3 sec until I can see the change in the web-browser. I tried to do the sync more frequently but I experienced issues with high CPU-load when I did that.

@duro I use 3 sec because it's enough for my purpose (PHP development in SugerCRM). When I save something in my IDE I have to wait approximately 3 sec until I can see the change in the web-browser. I tried to do the sync more frequently but I experienced issues with high CPU-load when I did that.

@WyseNynja

This comment has been minimized.

Show comment
Hide comment
@WyseNynja

WyseNynja Jul 31, 2014

I've been using unison instead of the built-in vagrant support and it's been working well. This also lets me easily sync with Vagrants launched by other people and even production machines all with pretty much the same sync commands

I've been using unison instead of the built-in vagrant support and it's been working well. This also lets me easily sync with Vagrants launched by other people and even production machines all with pretty much the same sync commands

@duro

This comment has been minimized.

Show comment
Hide comment
@duro

duro Jul 31, 2014

Contributor

@WyseNynja and @jenmo917, have you guys tried using the vagrant-unison plugin?

https://github.com/mrdavidlaing/vagrant-unison

I have not, I'm just wondering if you have tested it.

Contributor

duro commented Jul 31, 2014

@WyseNynja and @jenmo917, have you guys tried using the vagrant-unison plugin?

https://github.com/mrdavidlaing/vagrant-unison

I have not, I'm just wondering if you have tested it.

@jenmo917

This comment has been minimized.

Show comment
Hide comment
@jenmo917

jenmo917 Jul 31, 2014

@duro, yes I have, and I had problems with it. I think it was because it didn't have auto-sync and you had to trigger the sync yourself each time. I took the "on change mechanism" from vagrant rsync-auto and implemented it into the vagrant unison plugin. The problem I faced was that Unison was triggered on changes on the host machine, but not on the guest machine. When you think about it is quite natural because Rsync, where I found the code, is a one-way sync. This was how far i went with the unison plugin.

@duro, yes I have, and I had problems with it. I think it was because it didn't have auto-sync and you had to trigger the sync yourself each time. I took the "on change mechanism" from vagrant rsync-auto and implemented it into the vagrant unison plugin. The problem I faced was that Unison was triggered on changes on the host machine, but not on the guest machine. When you think about it is quite natural because Rsync, where I found the code, is a one-way sync. This was how far i went with the unison plugin.

@WyseNynja

This comment has been minimized.

Show comment
Hide comment
@WyseNynja

WyseNynja Aug 2, 2014

vagrant-unison can also only sync one directory. Additionally, I wanted to be able to easily sync to vagrants launched by other people and a vagrant plugin doesn't really make that easy

vagrant-unison can also only sync one directory. Additionally, I wanted to be able to easily sync to vagrants launched by other people and a vagrant plugin doesn't really make that easy

@btrepp

This comment has been minimized.

Show comment
Hide comment
@btrepp

btrepp Aug 6, 2014

I've been bitten using rsync assuming it would run like shared folders did :(. My fault entirely but yeah.
Was hacking on files inside the VM, this is what I usually did with shared folders, vagrant bootstraps the environment and then I work on the project. Was using rsync assuming it would be magical, broke some OS config playing around, easy enough I'll vagrant destroy and vagrant reload.
....Whoops there's all my changes, which were commited to git inside the VM, but never pushed out!, they were destroyed :(.

Also I feel this may be useful for my issues with building a windows machine. Windows shared folders are.... special at best. Atm I can't get vagrant to up a config using them, vbox shared folders worked perfectly here, but the requirement is visual studio in these machines.... which is also kinds of special, refuses to run projects of vbox shared folders.

A two-way rsync would let windows and its strange file system do whatever it wants, and just sync the changes around.

btrepp commented Aug 6, 2014

I've been bitten using rsync assuming it would run like shared folders did :(. My fault entirely but yeah.
Was hacking on files inside the VM, this is what I usually did with shared folders, vagrant bootstraps the environment and then I work on the project. Was using rsync assuming it would be magical, broke some OS config playing around, easy enough I'll vagrant destroy and vagrant reload.
....Whoops there's all my changes, which were commited to git inside the VM, but never pushed out!, they were destroyed :(.

Also I feel this may be useful for my issues with building a windows machine. Windows shared folders are.... special at best. Atm I can't get vagrant to up a config using them, vbox shared folders worked perfectly here, but the requirement is visual studio in these machines.... which is also kinds of special, refuses to run projects of vbox shared folders.

A two-way rsync would let windows and its strange file system do whatever it wants, and just sync the changes around.

@kensykora

This comment has been minimized.

Show comment
Hide comment
@kensykora

kensykora Aug 7, 2014

@btrepp I think in general you should be following the rule that if you need to hack inside of the VM then you're probably doing it wrong. Both configuration and code changes should be made using either a configuration manager (puppet, chef, ansible) or code changes through the rsync tool in this case.

For doing .Net stuff, vagrant definitely isn't ready for prime time here unless you have a Windows host and can build using Visual Studio on your host. If you are using an OSX host I'd recommend running vagrant ssh -c msbuild myproj.sln and working that into your dev workflow.

@btrepp I think in general you should be following the rule that if you need to hack inside of the VM then you're probably doing it wrong. Both configuration and code changes should be made using either a configuration manager (puppet, chef, ansible) or code changes through the rsync tool in this case.

For doing .Net stuff, vagrant definitely isn't ready for prime time here unless you have a Windows host and can build using Visual Studio on your host. If you are using an OSX host I'd recommend running vagrant ssh -c msbuild myproj.sln and working that into your dev workflow.

@jdhiro

This comment has been minimized.

Show comment
Hide comment
@jdhiro

jdhiro Aug 8, 2014

@kensykora who's rule is that exactly? ;-) Some frameworks require (or are easiest to use) with guest-side commands. For example, we use Laravel's artisan command for easily generating migration files.

jdhiro commented Aug 8, 2014

@kensykora who's rule is that exactly? ;-) Some frameworks require (or are easiest to use) with guest-side commands. For example, we use Laravel's artisan command for easily generating migration files.

@btrepp

This comment has been minimized.

Show comment
Hide comment
@btrepp

btrepp Aug 8, 2014

@kensykora the situation you've described basically means 'never use vagrant for anything with an ide'.

If I'm required to keep a build environment outside of vagrant (what most ides sell themselves as). Then what is the purpose of vagrant?.

Also I'm typically a major vim user (windows is an experiment in order to get the windows guys drinking the automation coolade), but in regards to vim. Say I had a whole bunch of specific plugins to make it easier to code in the specific framework we are using. If there is vim inside a VM then all developers get those plugins easy.

Under your suggested scenario. I also have to tell all the other devs they need plugin x,y,z to be productive, which means they would have to hunt down how to install it?. What if one of these guys used windows and the other Linux as the host?.

I see the power of vagrant of automating the dev environment. Project from two months ago? Vagrant up and get those same tools ready to rock.

btrepp commented Aug 8, 2014

@kensykora the situation you've described basically means 'never use vagrant for anything with an ide'.

If I'm required to keep a build environment outside of vagrant (what most ides sell themselves as). Then what is the purpose of vagrant?.

Also I'm typically a major vim user (windows is an experiment in order to get the windows guys drinking the automation coolade), but in regards to vim. Say I had a whole bunch of specific plugins to make it easier to code in the specific framework we are using. If there is vim inside a VM then all developers get those plugins easy.

Under your suggested scenario. I also have to tell all the other devs they need plugin x,y,z to be productive, which means they would have to hunt down how to install it?. What if one of these guys used windows and the other Linux as the host?.

I see the power of vagrant of automating the dev environment. Project from two months ago? Vagrant up and get those same tools ready to rock.

@kensykora

This comment has been minimized.

Show comment
Hide comment
@kensykora

kensykora Aug 8, 2014

Guys I apologize, I'm not trying to turn this rsync issue into a philosophical debate about dev workflows. Let's steer the conversation back to the issue at hand.

Definitely open to change on my opinion and I'd be happy to discuss this over twitter.

Guys I apologize, I'm not trying to turn this rsync issue into a philosophical debate about dev workflows. Let's steer the conversation back to the issue at hand.

Definitely open to change on my opinion and I'd be happy to discuss this over twitter.

@wbkang

This comment has been minimized.

Show comment
Hide comment
@wbkang

wbkang Aug 8, 2014

Here's my attempt to summarize the issue:

  • VirtualBox shared folder provides two-way synchronization but it does not fire inotify notifications so we can't run programs to watch changes and compile them (e.g., Less compiler).
    • The polling approach falls apart quickly because VirtualBox shared folder is slow.
  • Setting up a shared folder via rsync is fast and fires inotify but it is only one-way.

My workaround is to set up the source directory as two shared folders, one via rsync and another via virtualbox shared folder. e.g.,

config.vm.synced_folder ".", "/vagrant", type: "rsync"
config.vm.synced_folder ".", "/vagrant_sf"

I set things up so that any program that watches file changes via inotify reads from /vagrant. Any changes that needs to be persisted goes out to /vagrant_sf (e.g., alembic migration files). This workflow seems to work reasonably well.

wbkang commented Aug 8, 2014

Here's my attempt to summarize the issue:

  • VirtualBox shared folder provides two-way synchronization but it does not fire inotify notifications so we can't run programs to watch changes and compile them (e.g., Less compiler).
    • The polling approach falls apart quickly because VirtualBox shared folder is slow.
  • Setting up a shared folder via rsync is fast and fires inotify but it is only one-way.

My workaround is to set up the source directory as two shared folders, one via rsync and another via virtualbox shared folder. e.g.,

config.vm.synced_folder ".", "/vagrant", type: "rsync"
config.vm.synced_folder ".", "/vagrant_sf"

I set things up so that any program that watches file changes via inotify reads from /vagrant. Any changes that needs to be persisted goes out to /vagrant_sf (e.g., alembic migration files). This workflow seems to work reasonably well.

@pirog pirog referenced this issue Aug 19, 2014

Closed

Code sync solution #55

@brpaz

This comment has been minimized.

Show comment
Hide comment
@brpaz

brpaz Aug 21, 2014

I think an effective way of doing two-way sync is a must.
Lets stay I am manager of an IT company. I want that once a new developer arrives to the company, he just needs to do a git clone of the project following by vagrant up and it them has a complete environment ready to work. In the host system, he would only need to have the browser, the IDE, git and vagrant. All the tools needed for the project (bundler, composer, grunt, sass etc) are bundled in the VM and are executed inside the VM.
Then he moves to another project. Same procedure. git clone -> vagrant up and voila.
This would be an amazing way of work and a truly portable environment.
But for this to work, It has to be fast. I thought the rsync option would be the fastest compared with shared folders and nfs, until I read it only supports one way sync.

brpaz commented Aug 21, 2014

I think an effective way of doing two-way sync is a must.
Lets stay I am manager of an IT company. I want that once a new developer arrives to the company, he just needs to do a git clone of the project following by vagrant up and it them has a complete environment ready to work. In the host system, he would only need to have the browser, the IDE, git and vagrant. All the tools needed for the project (bundler, composer, grunt, sass etc) are bundled in the VM and are executed inside the VM.
Then he moves to another project. Same procedure. git clone -> vagrant up and voila.
This would be an amazing way of work and a truly portable environment.
But for this to work, It has to be fast. I thought the rsync option would be the fastest compared with shared folders and nfs, until I read it only supports one way sync.

@khromov

This comment has been minimized.

Show comment
Hide comment
@khromov

khromov Aug 21, 2014

For those having issues with the performance of VirtualBox Shared Folder, I suggest trying VMWare (Workstation or Fusion). In our testing, we could see a 10x performance increase in load times on a Laravel web application when switching over from VirtualBox SF to VMWare Workstation SF. (From 600ms to 60ms generation time)

khromov commented Aug 21, 2014

For those having issues with the performance of VirtualBox Shared Folder, I suggest trying VMWare (Workstation or Fusion). In our testing, we could see a 10x performance increase in load times on a Laravel web application when switching over from VirtualBox SF to VMWare Workstation SF. (From 600ms to 60ms generation time)

@patrickheeney

This comment has been minimized.

Show comment
Hide comment
@patrickheeney

patrickheeney Aug 21, 2014

@brpaz This is exactly my goal with this as well. Right now it is such a mess to keep the front-end devs in the right workflow. I can't even count the number of times I had to remote in and npm install or fix host machine bower, grunt, saas, node etc because I can't bundle it in the VM because of performance issues. Frustrating to say the least.

@khromov The speed issues I am struggling with is the build tools that monitor folders. 600ms to 60s is awesome but I wish we had a way to cut down 2 minute build times to the 2 seconds when running it from the host machine. I will check out that plugin though for faster laravel speeds!

@brpaz This is exactly my goal with this as well. Right now it is such a mess to keep the front-end devs in the right workflow. I can't even count the number of times I had to remote in and npm install or fix host machine bower, grunt, saas, node etc because I can't bundle it in the VM because of performance issues. Frustrating to say the least.

@khromov The speed issues I am struggling with is the build tools that monitor folders. 600ms to 60s is awesome but I wish we had a way to cut down 2 minute build times to the 2 seconds when running it from the host machine. I will check out that plugin though for faster laravel speeds!

@purpleidea

This comment has been minimized.

Show comment
Hide comment
@purpleidea

purpleidea May 12, 2015

Contributor

On Tue, May 12, 2015 at 1:56 AM, Aleksandar Kostadinov <
notifications@github.com> wrote:

@purpleidea https://github.com/purpleidea, if I'm using another config
management system, why should I use vagrant? Also reading about ansible for
example I don't see any big difference - the issue remains the same. I
think vagrant is mostly useful for developing, testing and demos. For
production you'd have pre-generated certificates and keys. While for the
former use cases you need to generate things on the fly to avoid security
issues.

My point is that vagrant is for dev and not prod, and if you use the same
mechanism in both to distribute your files, then your dev env. will more
accurately resemble prod, and tests are more likely to accurately represent
real life.

Contributor

purpleidea commented May 12, 2015

On Tue, May 12, 2015 at 1:56 AM, Aleksandar Kostadinov <
notifications@github.com> wrote:

@purpleidea https://github.com/purpleidea, if I'm using another config
management system, why should I use vagrant? Also reading about ansible for
example I don't see any big difference - the issue remains the same. I
think vagrant is mostly useful for developing, testing and demos. For
production you'd have pre-generated certificates and keys. While for the
former use cases you need to generate things on the fly to avoid security
issues.

My point is that vagrant is for dev and not prod, and if you use the same
mechanism in both to distribute your files, then your dev env. will more
accurately resemble prod, and tests are more likely to accurately represent
real life.

@akostadinov

This comment has been minimized.

Show comment
Hide comment
@akostadinov

akostadinov May 12, 2015

@purpleidea, sure that might be possible sometimes. But you can't use your production keys and certificates in dev environment unless you want to get into trouble (assuming you even have access to the prod certificates). And I think more and more systems nowadays use some sort of certificates or keys to talk securely to each other. This use case - securely sharing data between the provisioned machines is not covered well in current vagrant when deploying non-locally.

@purpleidea, sure that might be possible sometimes. But you can't use your production keys and certificates in dev environment unless you want to get into trouble (assuming you even have access to the prod certificates). And I think more and more systems nowadays use some sort of certificates or keys to talk securely to each other. This use case - securely sharing data between the provisioned machines is not covered well in current vagrant when deploying non-locally.

@purpleidea

This comment has been minimized.

Show comment
Hide comment
@purpleidea

purpleidea May 12, 2015

Contributor

On Tue, May 12, 2015 at 2:12 AM, Aleksandar Kostadinov <
notifications@github.com> wrote:

But you can't use your production keys and certificates in dev environment

Use different copies of course!

This use case - securely sharing data between the provisioned machines is
not covered well in current vagrant when deploying non-locally.

I don't think it's supposed to be, IMO.

I think I've offered my point clearly enough. It's up to you if you want to
take these suggestions or not. I don't think it's helpful for me to discuss
this any further. Cheers!

Contributor

purpleidea commented May 12, 2015

On Tue, May 12, 2015 at 2:12 AM, Aleksandar Kostadinov <
notifications@github.com> wrote:

But you can't use your production keys and certificates in dev environment

Use different copies of course!

This use case - securely sharing data between the provisioned machines is
not covered well in current vagrant when deploying non-locally.

I don't think it's supposed to be, IMO.

I think I've offered my point clearly enough. It's up to you if you want to
take these suggestions or not. I don't think it's helpful for me to discuss
this any further. Cheers!

@akostadinov

This comment has been minimized.

Show comment
Hide comment
@akostadinov

akostadinov May 12, 2015

I don't think it's supposed to be, IMO.

Why?
I don't think it's helpful for me to discuss this any further.
Not helpful for me too unless you bring anything new to the table. Generating certificates and installing properly on all machines is often not trivial, also may depend on user preferences about number of machines, their role and certificate types. It's perfectly legitimate to allow that to happen automated within vagrant IMHO. For small projects it may not be an issue but for big projects it becomes an issue.

I don't think it's supposed to be, IMO.

Why?
I don't think it's helpful for me to discuss this any further.
Not helpful for me too unless you bring anything new to the table. Generating certificates and installing properly on all machines is often not trivial, also may depend on user preferences about number of machines, their role and certificate types. It's perfectly legitimate to allow that to happen automated within vagrant IMHO. For small projects it may not be an issue but for big projects it becomes an issue.

@rarkins

This comment has been minimized.

Show comment
Hide comment
@rarkins

rarkins May 22, 2015

We decided to go with unison instead of rsync, but using a manual installation instead of the plugin. I've documented the process here in case it helps anyone.

rarkins commented May 22, 2015

We decided to go with unison instead of rsync, but using a manual installation instead of the plugin. I've documented the process here in case it helps anyone.

@nha

This comment has been minimized.

Show comment
Hide comment
@nha

nha Jul 21, 2015

@rarkins Thanks that looks useful. bidirectional sync is very important for certain workflows.

As a workaround, is there a way to mount several folders ?

nha commented Jul 21, 2015

@rarkins Thanks that looks useful. bidirectional sync is very important for certain workflows.

As a workaround, is there a way to mount several folders ?

@rarkins

This comment has been minimized.

Show comment
Hide comment
@rarkins

rarkins Jul 21, 2015

@nha I don't have a need for that, but would guess that the simple but inefficient way is to have multiple .prf files and run multiple simultaneous unison commands. An alternative might be symlinks, as described here?

Also, are you sure you can't just sync your main directory and exclude paths within it you don't want sync'd?

rarkins commented Jul 21, 2015

@nha I don't have a need for that, but would guess that the simple but inefficient way is to have multiple .prf files and run multiple simultaneous unison commands. An alternative might be symlinks, as described here?

Also, are you sure you can't just sync your main directory and exclude paths within it you don't want sync'd?

@giovdk21

This comment has been minimized.

Show comment
Hide comment
@giovdk21

giovdk21 Nov 13, 2015

+1 for two-way sync with rsync or at least an handy and reliable rsync-pull, but the former would be ideal.

+1 for two-way sync with rsync or at least an handy and reliable rsync-pull, but the former would be ideal.

@russjaguar

This comment has been minimized.

Show comment
Hide comment
@russjaguar

russjaguar Dec 24, 2015

We were able to get @rarkins solution working. I see there might be a way to automate the management a bit by having profiles name after the boxes... maybe... Would be nice if the plugin were to get maintained as the NFS module is hit-and-miss within the 20 or so boxes we end up working with. Can be very annoying to make changes, not see them loading and thing you're doing something wrong only for them to suddenly load 1-2 minutes later.

We were able to get @rarkins solution working. I see there might be a way to automate the management a bit by having profiles name after the boxes... maybe... Would be nice if the plugin were to get maintained as the NFS module is hit-and-miss within the 20 or so boxes we end up working with. Can be very annoying to make changes, not see them loading and thing you're doing something wrong only for them to suddenly load 1-2 minutes later.

@Perni1984

This comment has been minimized.

Show comment
Hide comment
@Perni1984

Perni1984 Feb 18, 2016

@rarkins: Thanks, you saved my life. Unison is working great for bidirectional synching between windows and vms. Now I can develop and run symfony2 with full speed inside my vms on my windows host. For me, this works even better than vagrant-winnfsd.

@rarkins: Thanks, you saved my life. Unison is working great for bidirectional synching between windows and vms. Now I can develop and run symfony2 with full speed inside my vms on my windows host. For me, this works even better than vagrant-winnfsd.

@flynfish

This comment has been minimized.

Show comment
Hide comment
@flynfish

flynfish Mar 10, 2016

@dmatora:

  1. does running vagrant plugin install vagrant-unison install your forked version? If not, how do we get yours installed?
  2. Is there a way to change the ssh username to something besides 'vagrant' ?
  3. I would have posted these questions on your fork, but issues are turned off

@dmatora:

  1. does running vagrant plugin install vagrant-unison install your forked version? If not, how do we get yours installed?
  2. Is there a way to change the ssh username to something besides 'vagrant' ?
  3. I would have posted these questions on your fork, but issues are turned off
@dmatora

This comment has been minimized.

Show comment
Hide comment
@dmatora

dmatora Mar 11, 2016

@flynfish no, it does't.
Simple way to install it would be to download https://www.dropbox.com/s/1yu1s7pr3qlr8ag/vagrant-unison-0.0.15.gem and do vagrant plugin install vagrant-unison-0.0.15.gem
but it's currently outdated.

I'm sorry to say that guys, but i moved to using nfs_guest, i tried to use it before unison, and nfs silently failed for me. But later i discovered than on OSX you need to have localhost in your /etc/hosts file in order for nfs to work and that solved it. With nfs_guest i don't have fs_events, but performance is close to filesystem, and i don't have sync failures, so i don't fix those anymore. Yet CPU load is so much smaller.

dmatora commented Mar 11, 2016

@flynfish no, it does't.
Simple way to install it would be to download https://www.dropbox.com/s/1yu1s7pr3qlr8ag/vagrant-unison-0.0.15.gem and do vagrant plugin install vagrant-unison-0.0.15.gem
but it's currently outdated.

I'm sorry to say that guys, but i moved to using nfs_guest, i tried to use it before unison, and nfs silently failed for me. But later i discovered than on OSX you need to have localhost in your /etc/hosts file in order for nfs to work and that solved it. With nfs_guest i don't have fs_events, but performance is close to filesystem, and i don't have sync failures, so i don't fix those anymore. Yet CPU load is so much smaller.

@mikestaub

This comment has been minimized.

Show comment
Hide comment
@mikestaub

mikestaub Mar 11, 2016

I have been using the vagrant-unison2 plugin with success. I was also using NFS but ran into issues with windows and npm.

I have been using the vagrant-unison2 plugin with success. I was also using NFS but ran into issues with windows and npm.

@smerrill smerrill referenced this issue Apr 6, 2016

Closed

Two-way sync? #30

@geerlingguy geerlingguy referenced this issue May 11, 2016

Closed

DrupalCon NOLA 2016 - Battle Plan for 3.0.0 #604

4 of 4 tasks complete

@chrisroberts chrisroberts added this to the 1.9.0 milestone Sep 30, 2016

@seantcanavan

This comment has been minimized.

Show comment
Hide comment
@seantcanavan

seantcanavan Oct 6, 2016

I've seen a couple good suggestions here so far. I'm completely out of options because network-based shares don't work over VPN but our VPN client is not up for discussion which leaves openconnect out of discussion.

Rsync appears to be the only option as appallingly slow disk read and write speeds have ground our build and test processes down to a halt on the standard sync share.

But the issue with Rsync is that it only supports one way sync and of course requires cygwin to be installed as well. Bundling a two-way rsync into Vagrant would be a huge huge huge win.

It looks like there is hope though. I'm excited that this has been tagged as milestone 1.9.0 but 1.8.7 is only 33% complete as of this moment in time. An interim solution will absolutely have to be implemented.

I've seen a couple good suggestions here so far. I'm completely out of options because network-based shares don't work over VPN but our VPN client is not up for discussion which leaves openconnect out of discussion.

Rsync appears to be the only option as appallingly slow disk read and write speeds have ground our build and test processes down to a halt on the standard sync share.

But the issue with Rsync is that it only supports one way sync and of course requires cygwin to be installed as well. Bundling a two-way rsync into Vagrant would be a huge huge huge win.

It looks like there is hope though. I'm excited that this has been tagged as milestone 1.9.0 but 1.8.7 is only 33% complete as of this moment in time. An interim solution will absolutely have to be implemented.

@purpleidea

This comment has been minimized.

Show comment
Hide comment
@dustymabe

This comment has been minimized.

Show comment
Hide comment
@dustymabe

dustymabe Oct 7, 2016

Contributor

@seantcanavan check out vagrant-sshfs - i've actually been thinking about proposing it for inclusion in vagrant itself. for now it lives pretty well on its own as a plugin.

Contributor

dustymabe commented Oct 7, 2016

@seantcanavan check out vagrant-sshfs - i've actually been thinking about proposing it for inclusion in vagrant itself. for now it lives pretty well on its own as a plugin.

@seantcanavan

This comment has been minimized.

Show comment
Hide comment
@seantcanavan

seantcanavan Oct 7, 2016

Thanks for the ideas @purpleidea and @dustymabe. Unfortunately the ruby gem bundler uses a custom user agent string which the corporate firewall immediately shoots down which makes plugin installation impossible so I was hoping for a baked-in solution.

Looks like I gotta get the network guys on the phone...

Cheers.

seantcanavan commented Oct 7, 2016

Thanks for the ideas @purpleidea and @dustymabe. Unfortunately the ruby gem bundler uses a custom user agent string which the corporate firewall immediately shoots down which makes plugin installation impossible so I was hoping for a baked-in solution.

Looks like I gotta get the network guys on the phone...

Cheers.

@dustymabe

This comment has been minimized.

Show comment
Hide comment
@dustymabe

dustymabe Oct 7, 2016

Contributor

@seantcanavan can you download the gem and install them directly (i.e. install from file instead of from rubygems.org) I think for vagrant-sshfs you need win32-process and vagrant-sshfs. if you get those two gems and then install directly from file (i.e. vagrant plugin install ./win32-process-X-X.gem && vagrant plugin install ./vagrant-sshfs-X-X.gem) then it should work

Contributor

dustymabe commented Oct 7, 2016

@seantcanavan can you download the gem and install them directly (i.e. install from file instead of from rubygems.org) I think for vagrant-sshfs you need win32-process and vagrant-sshfs. if you get those two gems and then install directly from file (i.e. vagrant plugin install ./win32-process-X-X.gem && vagrant plugin install ./vagrant-sshfs-X-X.gem) then it should work

@seantcanavan

This comment has been minimized.

Show comment
Hide comment
@seantcanavan

seantcanavan Oct 10, 2016

@dustymabe Finally got the gem files built for win32-process-0.8.3 and vagrant-sshfs-1.2.0. I created a gems directory at my project's root folder where the vagrant file is located and put both gem files in there. I executed vagrant plugin install ./gems/win32-process-0.8.3.gem from the root directory and received the error /usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in all=': undefined method group_by' for nil:NilClass (NoMethodError)

EDIT01: Appears to be an issue with Ubuntu 16.04. Vagrant 1.8.1 and ruby 2.3.1.
EDIT02: Patched /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb successfully. Directions here
EDIT03: New error after trying to install: Could not find gem 'win32-process (= 0.8.3)' in any of the gem sources listed in your Gemfile or available on this machine.
EDIT04: Appears Ubunbut 16.04 is kicking me in the teeth again... Investigating: #7493
EDIT05: Apparently need to upgrade Vagrant version. Command is "wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb && sudo dpkg -i vagrant_1.8.6_x86_64.deb". Restarted my box and plugin install SUCCEEDED!
EDIT06: vagrant destroy now produces There was an error while trying to load the gem 'win32-process'. Gem Load Error is: unable to resolve type 'uintptr_t' I feel like I'm on a roller coaster.
EDIT07: I've filed an issue with the ffi plugin, which is a dependency of the win32-process plugin, here: ffi/ffi#530. I might have to open a different issue if the error message is a red herring and not really an ffi issue.
EDIT08: Installing win32-process on OSX and Linux totally breaks vagrant. Employing brain power to realize win32-process is a windows-only plugin solves the issue.
EDIT09: Getting a new error now on vagrant up: /opt/vagrant/embedded/gems/gems/vagrant-1.8.6/lib/vagrant/action/builtin/mixin_synced_folders.rb:137:inblock in synced_folders': Internal error. Report this as a bug. Invalid: sshfs (RuntimeError)` Double checking my syntax. Seems easy enough - like sshfs is an unknown synced_folder type.
EDIT10: Appears my locally built gem file is causing the error. When I manage to bypass the firewall and install remotely via the gem bundler it works fine. But more importantly success! Finally!

seantcanavan commented Oct 10, 2016

@dustymabe Finally got the gem files built for win32-process-0.8.3 and vagrant-sshfs-1.2.0. I created a gems directory at my project's root folder where the vagrant file is located and put both gem files in there. I executed vagrant plugin install ./gems/win32-process-0.8.3.gem from the root directory and received the error /usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in all=': undefined method group_by' for nil:NilClass (NoMethodError)

EDIT01: Appears to be an issue with Ubuntu 16.04. Vagrant 1.8.1 and ruby 2.3.1.
EDIT02: Patched /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb successfully. Directions here
EDIT03: New error after trying to install: Could not find gem 'win32-process (= 0.8.3)' in any of the gem sources listed in your Gemfile or available on this machine.
EDIT04: Appears Ubunbut 16.04 is kicking me in the teeth again... Investigating: #7493
EDIT05: Apparently need to upgrade Vagrant version. Command is "wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb && sudo dpkg -i vagrant_1.8.6_x86_64.deb". Restarted my box and plugin install SUCCEEDED!
EDIT06: vagrant destroy now produces There was an error while trying to load the gem 'win32-process'. Gem Load Error is: unable to resolve type 'uintptr_t' I feel like I'm on a roller coaster.
EDIT07: I've filed an issue with the ffi plugin, which is a dependency of the win32-process plugin, here: ffi/ffi#530. I might have to open a different issue if the error message is a red herring and not really an ffi issue.
EDIT08: Installing win32-process on OSX and Linux totally breaks vagrant. Employing brain power to realize win32-process is a windows-only plugin solves the issue.
EDIT09: Getting a new error now on vagrant up: /opt/vagrant/embedded/gems/gems/vagrant-1.8.6/lib/vagrant/action/builtin/mixin_synced_folders.rb:137:inblock in synced_folders': Internal error. Report this as a bug. Invalid: sshfs (RuntimeError)` Double checking my syntax. Seems easy enough - like sshfs is an unknown synced_folder type.
EDIT10: Appears my locally built gem file is causing the error. When I manage to bypass the firewall and install remotely via the gem bundler it works fine. But more importantly success! Finally!

@dustymabe

This comment has been minimized.

Show comment
Hide comment
@dustymabe

dustymabe Oct 10, 2016

Contributor

@seantcanavan wow - so you finally got something working?

Contributor

dustymabe commented Oct 10, 2016

@seantcanavan wow - so you finally got something working?

@seantcanavan

This comment has been minimized.

Show comment
Hide comment
@seantcanavan

seantcanavan Oct 10, 2016

@dustymabe Yes!! I hope I'm not derailing and someone else will find this info useful. My last question is this: I've pulled in vagrant-sshfs from outside our firewall once and installed it locally and it works great. Now I need to package it in a gem file to add it to our source so people inside the firewall can reliably retrieve it. I found the plugin installed at /home/username/.vagrant.d/gems/gems/vagrant-sshfs-1.2.0 but when I run bundle install inside the install dir I get

[!] There was an error parsing Gemfile: You cannot specify the same gem twice coming from different sources.
You specified that vagrant-sshfs (>= 0) should come from source at . and source at .
. Bundler cannot continue.

from /home/username/.vagrant.d/gems/gems/vagrant-sshfs-1.2.0/Gemfile:13

-------------------------------------------

group :plugins do"

gem "vagrant-sshfs" , path: "."

# Add vagrant-libvirt plugin here, otherwise you won't be able to

-------------------------------------------

Do you have a quick and easy way I can package your plugin to install locally via command line for future use?

seantcanavan commented Oct 10, 2016

@dustymabe Yes!! I hope I'm not derailing and someone else will find this info useful. My last question is this: I've pulled in vagrant-sshfs from outside our firewall once and installed it locally and it works great. Now I need to package it in a gem file to add it to our source so people inside the firewall can reliably retrieve it. I found the plugin installed at /home/username/.vagrant.d/gems/gems/vagrant-sshfs-1.2.0 but when I run bundle install inside the install dir I get

[!] There was an error parsing Gemfile: You cannot specify the same gem twice coming from different sources.
You specified that vagrant-sshfs (>= 0) should come from source at . and source at .
. Bundler cannot continue.

from /home/username/.vagrant.d/gems/gems/vagrant-sshfs-1.2.0/Gemfile:13

-------------------------------------------

group :plugins do"

gem "vagrant-sshfs" , path: "."

# Add vagrant-libvirt plugin here, otherwise you won't be able to

-------------------------------------------

Do you have a quick and easy way I can package your plugin to install locally via command line for future use?

@dustymabe

This comment has been minimized.

Show comment
Hide comment
@dustymabe

dustymabe Oct 10, 2016

Contributor

@seantcanavan commenting out this line should work to fix that error: https://github.com/dustymabe/vagrant-sshfs/blob/master/Gemfile#L13

the other thing is - why not just download the gem from my releases page and use that?
https://github.com/dustymabe/vagrant-sshfs/releases/download/v1.2.0/vagrant-sshfs-1.2.0.gem

Contributor

dustymabe commented Oct 10, 2016

@seantcanavan commenting out this line should work to fix that error: https://github.com/dustymabe/vagrant-sshfs/blob/master/Gemfile#L13

the other thing is - why not just download the gem from my releases page and use that?
https://github.com/dustymabe/vagrant-sshfs/releases/download/v1.2.0/vagrant-sshfs-1.2.0.gem

@chrisroberts

This comment has been minimized.

Show comment
Hide comment
@chrisroberts

chrisroberts Mar 1, 2017

Collaborator

Hi there,

We would really like this, but this issue has been open for over a year with no one working on it, Leaving it open is unfortunately making our issue count look higher than it is. I’m going to close this and if someone wants to work on it I’d still be open to a PR!

Cheers!

Collaborator

chrisroberts commented Mar 1, 2017

Hi there,

We would really like this, but this issue has been open for over a year with no one working on it, Leaving it open is unfortunately making our issue count look higher than it is. I’m going to close this and if someone wants to work on it I’d still be open to a PR!

Cheers!

@mcandre

This comment has been minimized.

Show comment
Hide comment
@mcandre

mcandre Jan 3, 2018

Contributor

Crap. I spent all this time and effort setting up Vagrant boxes in the hope of developing a suite of buildbots for cross-compiling systems language applications for different operating system kernels. And then I find out that rsync-based Vagrant synced folders--and there is no other option besides rsync-based Vagrant synced folders for the vast majority of guest operating systems--does not support bidirectional syncing. So there's little point in vagrant ssh -c 'clang -o /vagrant/hello hello.c' if the hello binary won't even be transferred to the host. :(

I have a few suggestions for implementing bidirectional rsync more quickly:

  • https://github.com/deajan/osync , which claims to just depend on bash and rsync, and therefore could be easily installed automatically by Vagrant into rsync-enabled guests.
  • Continue using /vagrant as a one-way source host->guest folder, and introduce a second folder like /vagrant-output that one-way syncs to the host as <host-cwd>/vagrant-output. When Vagrant rsyncs the /vagrant source folder, it should exclude vagrant-output.
  • The user can also manually rsync files out of the guest to the host, but this represents a leaky API for Vagrant.

rsync configuration is something that Vagrant should really master, for more capable VM's. In the meantime, I might configure my buildbots to publish their artifacts to S3 instead of trying to copy the artifacts to the host :/

Contributor

mcandre commented Jan 3, 2018

Crap. I spent all this time and effort setting up Vagrant boxes in the hope of developing a suite of buildbots for cross-compiling systems language applications for different operating system kernels. And then I find out that rsync-based Vagrant synced folders--and there is no other option besides rsync-based Vagrant synced folders for the vast majority of guest operating systems--does not support bidirectional syncing. So there's little point in vagrant ssh -c 'clang -o /vagrant/hello hello.c' if the hello binary won't even be transferred to the host. :(

I have a few suggestions for implementing bidirectional rsync more quickly:

  • https://github.com/deajan/osync , which claims to just depend on bash and rsync, and therefore could be easily installed automatically by Vagrant into rsync-enabled guests.
  • Continue using /vagrant as a one-way source host->guest folder, and introduce a second folder like /vagrant-output that one-way syncs to the host as <host-cwd>/vagrant-output. When Vagrant rsyncs the /vagrant source folder, it should exclude vagrant-output.
  • The user can also manually rsync files out of the guest to the host, but this represents a leaky API for Vagrant.

rsync configuration is something that Vagrant should really master, for more capable VM's. In the meantime, I might configure my buildbots to publish their artifacts to S3 instead of trying to copy the artifacts to the host :/

@dustymabe

This comment has been minimized.

Show comment
Hide comment
@dustymabe

dustymabe Jan 3, 2018

Contributor

@mcandre would vagrant-sshfs work for you?

Contributor

dustymabe commented Jan 3, 2018

@mcandre would vagrant-sshfs work for you?

@cromulus

This comment has been minimized.

Show comment
Hide comment
@cromulus

cromulus Jan 4, 2018

cromulus commented Jan 4, 2018

@daks

This comment has been minimized.

Show comment
Hide comment
@daks

daks Jan 4, 2018

Contributor

@mcandre I use vagrant-scp to retrieve artifacts from Vagrant machines.

Contributor

daks commented Jan 4, 2018

@mcandre I use vagrant-scp to retrieve artifacts from Vagrant machines.

@mcandre

This comment has been minimized.

Show comment
Hide comment
@mcandre

mcandre Feb 3, 2018

Contributor

@dustymabe vagrant-sshfs is interesting. Unfortunately, it is not clear that this plugin supports bidirectional sync in a single Vagrant box configuration--the documentation seems to indicate that either guest->host syncing can be configured, xor host->guest, but not both at once.

Also, this plugin does not support Windows, which for my purposes (cross-compilation) is a no-go.

Update

Good news, the vagrant-rsync-back plugin appears to work reasonably reliably, at least to-and-from UNIX-style operating systems like Debian, Illumos, and MINIX. Will be testing this with Windows as well. It stinks that vagrant rsync-back must be manually called in order to perform the other half of bidirectional sync, but at least it works! Would love to see this capability backported into the main Vagrant system so that rsync synced folders can work bidirectionally out of the box, so to speak.

Contributor

mcandre commented Feb 3, 2018

@dustymabe vagrant-sshfs is interesting. Unfortunately, it is not clear that this plugin supports bidirectional sync in a single Vagrant box configuration--the documentation seems to indicate that either guest->host syncing can be configured, xor host->guest, but not both at once.

Also, this plugin does not support Windows, which for my purposes (cross-compilation) is a no-go.

Update

Good news, the vagrant-rsync-back plugin appears to work reasonably reliably, at least to-and-from UNIX-style operating systems like Debian, Illumos, and MINIX. Will be testing this with Windows as well. It stinks that vagrant rsync-back must be manually called in order to perform the other half of bidirectional sync, but at least it works! Would love to see this capability backported into the main Vagrant system so that rsync synced folders can work bidirectionally out of the box, so to speak.

@dustymabe

This comment has been minimized.

Show comment
Hide comment
@dustymabe

dustymabe Feb 3, 2018

Contributor

@dustymabe vagrant-sshfs is interesting. Unfortunately, it is not clear that this plugin supports bidirectional sync in a single Vagrant box configuration--the documentation seems to indicate that either guest->host syncing can be configured, xor host->guest, but not both at once.

SSHFS is essentially a network filesystem. You can access files from the host or from inside the guest at the same time and they transparently update all locations. A longer description here.

Also, this plugin does not support Windows, which for my purposes (cross-compilation) is a no-go.

It does support windows, just not for reverse mounts (sharing guest files to a host folder), which I don't think you actually need. How about giving it a try so you'll know for sure 😃.

Contributor

dustymabe commented Feb 3, 2018

@dustymabe vagrant-sshfs is interesting. Unfortunately, it is not clear that this plugin supports bidirectional sync in a single Vagrant box configuration--the documentation seems to indicate that either guest->host syncing can be configured, xor host->guest, but not both at once.

SSHFS is essentially a network filesystem. You can access files from the host or from inside the guest at the same time and they transparently update all locations. A longer description here.

Also, this plugin does not support Windows, which for my purposes (cross-compilation) is a no-go.

It does support windows, just not for reverse mounts (sharing guest files to a host folder), which I don't think you actually need. How about giving it a try so you'll know for sure 😃.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment