rsync-auto ignores subfolder directory changes

[polynomialdonut]

Vagrant version

2.2.5

Host operating system

Ubuntu 18.04.2 LTS

Guest operating system

Ubuntu 16.04.6 LTS

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

## ADAPT: CHANGED repo inside the VM's host machine - Uncomment and adapt the following line!
DIR_HOST_CHANGED_REPO = "/home/CHANGED/git-repo"

# All Vagrant configuration is done below. The constant "VAGRANTFILE_API_VERSION"
# in Vagrant.configure configures the configuration version (we support older styles
# for backwards compatibility). Please don't change it unless you know what you're doing.
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  BOX_USER_HOME_DIR="/home/vagrant"
  # Fixes (TODO: Put into separate provisioner script/store state inside Vagrant for new base box (?))
  ## In order to work, this has to stay here on top (before other provisioner commands are run)
  config.vm.provision "Fix No-tty error message",
  type: "shell" do |s|
    s.privileged = false
    s.inline = "echo -e '\tFix for no-tty'; sudo sed -i '/tty/!s/mesg n/tty -s \\&\\& mesg n/' /root/.profile"
  end
  ## Fix 'unable to re-open stdin: No file or directory' from dpkg-reconfigure (doesn't work properly, it seems)
  config.vm.provision "Fix 'unable to re-open stdin",
  type: "shell" do |s|
    s.inline = "echo -e \"\tFix 'unable to re-open stdin: No file or directory' from dpkg-reconfigure\";  sed -i 's@\(DPkg::Pre-Install-Pkgs {\"/usr/sbin/dpkg-preconfigure --apt || true\"\;};\)@// \1@g' /etc/apt/apt.conf.d/70debconf"
  end
  config.vm.provision "shell" do |s|
    s.inline = "dpkg-reconfigure debconf -f noninteractive -p critical"
  end

  ## (omitted/sensitive information) ##

   ## Where we want to put the NFS-mounted repo inside the guest
  DIR_GUEST_CHANGED_REPO = "/usr/local/src/CHANGED-git-repo"
  ## Sync above folders from Host to guest
  DIR_HOST_CHANGED_REPO_BUILD="#{DIR_HOST_CHANGED_REPO}/build"
  #config.vm.synced_folder DIR_HOST_CHANGED_REPO, DIR_GUEST_CHANGED_REPO, type: "rsync",
  config.vm.synced_folder DIR_HOST_CHANGED_REPO, DIR_GUEST_CHANGED_REPO, type: "rsync",
	  rsync__args: ["--verbose", "--archive", "--delete", "-z", "--ignore-errors"],
          rsync__exclude: ["build/*", "build/", "*.so*", "*.o*", "localInstDir/"]
  config.gatling.rsync_on_startup = false
  # TODO: DELETE
  # config.vm.synced_folder DIR_HOST_CHANGED_REPO, DIR_GUEST_CHANGED_REPO_NFS,
  # id: "CHANGED", type: "nfs", mount_options: ['rw', 'vers=3', 'tcp'], linux__nfs_options: ['rw','no_subtree_check','all_squash','async']

  config.vm.provision "Set the Timezone to Berlin",
  type: "shell" do |s|
    s.inline = "if [ -f /etc/localtime ]; then rm /etc/localtime; fi && ln -s /usr/share/zoneinfo/Europe/Berlin /etc/localtime"
  end
  config.vm.provision "Install 'ntp' so guest and host have the same time",
  type: "shell" do |s|
    s.inline = "export DEBIAN_FRONTEND=noninteractive; apt-get -qq update && apt-get -qq -y install ntp"
  end
  config.vm.provision "Set ulimit",
  type: "shell" do |s|
    s.inline = "mv /etc/security/limits.conf /etc/security/limits.conf.orig &&
      echo -e '# For details: 'man limits.conf'\n*	-	core	unlimited\nroot	-	core	unlimited\n# End of file\n' > /etc/security/limits.conf"
  end

  LINK_GUEST_CHANGED_REPO = "#{BOX_USER_HOME_DIR}/CHANGED-git-repo"
  config.vm.provision "Link location of CHANGED repo into home for convenience",
  type: "shell" do |s|
    s.inline = "if [ ! -L #{LINK_GUEST_CHANGED_REPO} ]; then\
    ln -s #{DIR_GUEST_CHANGED_REPO} #{LINK_GUEST_CHANGED_REPO}; fi"
  end

  ## (omitted) ##

  # Might work with one of the official Ubuntu boxes from Canonical, as well... (I dare not try for now)
  # bash /usr/bin/vagrant rsync-auto CHANGED-ubuntu-16.04
  RSYNC_AUTO_START_UBUNTU_16_04 = "vagrant rsync-auto CHANGED-ubuntu-16.04&"
  RSYNC_AUTO_START_UBUNTU_16_04_4GREP = "[v]agrant rsync-auto"
  KILL_RSYNC_AUTO_UBUNTU_16_04 = "bash -c \"for pid in \$(ps x|grep --color=never '.*#{RSYNC_AUTO_START_UBUNTU_16_04_4GREP}'|awk '{print \$1}'); do kill \$pid; done\""

  config.vm.define "CHANGED-ubuntu-16.04" do |pu1604|
    pu1604.vm.box = "bento/ubuntu-16.04"
    pu1604.trigger.after [:up, :reload, :resume]  do |t|
      t.info = "rsync auto after up/reload/resume"
      t.run = {inline: "bash -c '#{RSYNC_AUTO_START_UBUNTU_16_04}'"}
      # If you don't want it running in the background switch these
      # t.run = {inline: "vagrant rsync-auto bork"}
    end
    pu1604.trigger.before :reload do |t|
      t.info = "disable rsync auto before reloading"
      t.run = {inline: "#{KILL_RSYNC_AUTO_UBUNTU_16_04}"}
      # If you don't want it running in the background switch these
      # t.run = {inline: "vagrant rsync-auto bork"}
    end
    pu1604.trigger.after [:halt, :destroy, :suspend] do |t|
      t.info = "disable rsync auto after halting/destroying/suspending"
      t.run = {inline: "#{KILL_RSYNC_AUTO_UBUNTU_16_04}"}
      # If you don't want it running in the background switch these
      # t.run = {inline: "vagrant rsync-auto bork"}
    end
  end

  ## (ommitted) ##

  config.vm.provider "virtualbox" do |vb|
     ## ADAPT: For a speed boost, uncomment the following lines and plug in sensible data... (you don't have to)
     vb.memory = 12288
     vb.cpus = 8
     # Enable getting host time
     vb.customize [ "setextradata", :id, "VBoxInternal/Devices/VMMDev/0/Config/GetHostTimeDisabled", 0 ]
     vb.customize [ "guestproperty", "set", :id, "/VirtualBox/GuestAdd/VBoxService/--timesync-set-threshold", 10000 ]
  end

end

Expected behavior

The file #{DIR_HOST_CHANGED_REPO}/modules/pfcp/messages.c should have been updated inside the guest at #{DIR_GUEST_CHANGED_REPO} after writing to it on the host side.

Actual behavior

It was not updated. No error output was printed, so rsync didn't fail, either.

Steps to reproduce

1. vagrant up
2. Change #{DIR_HOST_CHANGED_REPO}/modules/pfcp/messages.c on the host side inside the source folder of the host-to-guest sync specified in the Vagrantfile.

Possible source of error

I am suspecting that there is some conversion from the list items given by rsync__exclude to regular expressions for rsync-auto going on, and that *.so* or *.o* got converted the wrong way; afaik rsync is understands that . is a literal dot, but maybe the conversion tool assumes a . means we have a single character, so ghat *.o actually acts as if we had passed *o, which would then exlude #{DIR_HOST_CHANGED_REPO}, since the variable ends on an 'o'.

After changing the exclude line to

rsync__exclude: ["build/**", "build/", "localInstDir/", "localInstDir/**"]

syncs were quasi-instantaneous.

Also, on the initial sync, everything was copied into the guest, so there seems to be some difference between how the initial sync works (and it did) and how rsync-auto syncs work.

[briancain]

Hi @polynomialdonut - after playing around with this myself, I think the issue might be with this change #10902. I noticed that the listener wouldn't pick up changes from files inside a given subfolder, but files at the root of the share foldergets picked up... 🤔 I recommend rolling back to the previous version, as I believe what you were doing shouldn't be affected by the bug I linked to.

[polynomialdonut]

Thank you for your reply/efforts @briancain,

the changes are in a subfolder, and I just rechecked - they are picked up instantly, presumably due to the changes I mentioned above. I might retry the old settings to confirm that they would still lead to certain files being ignored (can't do it right now since I should get some things done at work, first).

[jweir]

It is a pattern matching issue. We have seen this too.

we exclude ['data/'] and we have a sub directory called edata that should not be excluded. But it is.

If I am reading this correctly exclude is interpreting ['data/'] as ['*data/']

https://github.com/hashicorp/vagrant/pull/10902/files/1b0148bc783298c7aa16a519e133bb26bcc1cc9f#diff-de2df059af671fe3c2f160dacc56994dR124

[briancain]

Hi @jweir - I don't believe that's the case. Having a folder edata alongside data, edata gets synced. I don't think this is related to the exclude params, but to how the watcher is set up. If it were an excludes issue then it wouldn't work properly on the initial sync.

brian@localghost:vagrant-sandbox % be vagrant rsync-auto bork                                           ±[●●][master]
==> bork: Doing an initial rsync...
==> bork: Rsyncing folder: /home/brian/code/vagrant-sandbox/scripts/ => /vagrant
==> bork:   - Exclude: [".vagrant/", "data/"]
==> bork: Watching: /home/brian/code/vagrant-sandbox/scripts
^C%                                                                                                                   brian@localghost:vagrant-sandbox % be vagrant ssh -c "ls -lah /vagrant" bork                            ±[●●][master]
total 20K
drwxr-xr-x  5 vagrant vagrant 4.0K Jul 19 19:05 .
drwxr-xr-x 24 root    root    4.0K Jul 19 18:59 ..
drwxr-xr-x  2 vagrant vagrant 4.0K Jul 19 19:04 edata
drwxr-xr-x  4 vagrant vagrant 4.0K Jul 15 16:49 linux
drwxr-xr-x  2 vagrant vagrant 4.0K Jan 22 17:07 windows
Connection to 127.0.0.1 closed.

If you use a * in your exclude, then yes that would be the case, but Vagrant doesn't add a *. If you have a folder structure where edata is inside of data, and you ignore the top level folder data, then edata` won't be synced because you told rsync to ignore its parent directory...

As I mentioned earlier I believe the issue seems to be that the watcher only seems to be watching the top most directory level of a folder, and doesn't seem to pick up any sub directory changes.

[jweir]

All our files are initially syncing as expected. All files and directories are watched as expected, except any directory or file which matches an exclude entry.

If excludes is ['data/', 'log'] then sub directory lib/edata is not watched, or a file named flog.rb in any directory. Yes log is missing the ending /.

'lib/edata/'.match (Foo.exclude_to_regexp 'data/')
=> #<MatchData "data/">

Listen.to takes the array of regex rules.

require 'listen'
Listen.to('.', ignore: [/data\//])  { |n,a,r| puts "#{n}, #{a}, #{r}" }.start

If I mkdir edata and then touch edata/foo nothing happens, the path is ignored. But not with

require 'listen'
Listen.to('.', ignore: [/^data\//])  { |n,a,r| puts "#{n}, #{a}, #{r}" }.start
# or 
# Listen.to('.', ignore: [/\/data\//])  { |n,a,r| puts "#{n}, #{a}, #{r}" }.start

But, you can not prepend an exclude with a / since the helper strips those out.

The previous version of exclude_to_regexp included a ^ to enforce the start of the name and included the hostpath. But I don't really understand what was going in there.

https://github.com/briancain/vagrant/blob/3b540e502feb6576770ae6bbb4358e6f165cd57a/plugins/synced_folders/rsync/helper.rb#L28

I have tried to trace the code through to understand how this all works better, but I have gotten a bit lost. In other words, I might not be helping at all here. So, if the above is not useful, I will be quiet now. 😄

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.