gitfs regression with authenticated repos

[ghost]

I just upgraded my Fedora 21 install from salt-{,minon-}2014.7.1-1.fc21.noarch to 2014.7.5-1.fc21.noarch. Now, my setup does not work anymore.

I have python-pygit2 version 0.21.4 installed.

# cat /etc/salt/minion
file_client: local
fileserver_backend:
  - git
gitfs_remotes:
  - git@git.example.com:saltstack/bar:
    - pubkey: /root/.ssh/saltstack.pub
    - privkey: /root/.ssh/saltstack

All I get is this:

# salt-call state.highstate -l debug
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: bar
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] Mako not available
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Updating git fileserver cache
[DEBUG   ] Set lock for git@git.example.com:saltstack/bar
[DEBUG   ] gitfs is fetching from git@git.example.com:saltstack/bar
[DEBUG   ] gitfs received 0 objects for remote git@git.example.com:saltstack/bar
[DEBUG   ] gitfs cleaned the following stale refs: ['refs/remotes/origin/master']
[DEBUG   ] Removed lock for git@git.example.com:saltstack/bar
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://top.sls'
local:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found
     Started: 
    Duration: 
     Changes:   

Summary
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1

The top.sls is really simple for this repo:

base:
  'bar':
    - packages.bash
    - packages.gitolite
    - packages.mypkgs
    - packages.postfix
    - packages.salt-minion
    - packages.sshd
    - packages.sudo
    - packages.yum-cron
    - users

The directory /var/cache/salt/minion/files/base/ did contain the proper files previously. Although, after removing /var/cache/salt/minion/{gitfs,files} for testing purposes, it stays empty after a call to state.highstate:

# rm -rf /var/cache/salt/minion/{files,gitfs}
# salt-call state.highstate 
[INFO    ] Wrote new gitfs_remote map to /var/cache/salt/minion/gitfs/remote_map.txt
[INFO    ] Loading fresh modules for state activity
local:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found
     Started: 
    Duration: 
     Changes:   

Summary
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
# tree -a /var/cache/salt/minion/files/base/
/var/cache/salt/minion/files/base/

0 directories, 0 files

Is this expected behavior? I can't find anything about this in the Salt 2014.7.5 Release Notes.

[ghost]

I just upgraded a CentOS 7 machine to 2014.7.5-1.el7 and that machine keeps working.

salt-minion is installed from EPEL.

[ghost]

I upgraded another Fedora machine and it fails as described initially, too.

[ghost]

Comparing the debug output of the CentOS vs the Fedora machines I see the following appearing on the Fedora (i.e. non-working) machines only:

gitfs cleaned the following stale refs: ['refs/remotes/origin/master']

[ghost]

This has nothing to do with CentOS vs Fedora.

After adding a few debug lines in gitfs.py it was clear that the ssh key is not used. Creating /root/.ssh/config with the following content works around this problem.

Host git.example.com
    IdentityFile ~/.ssh/saltstack

[rallytime]

Thanks for this report @markusr815. When you upgraded salt, did you also upgrade python-pygit2? Or is that the same as before you upgraded the version of salt on your minion? I am trying to reproduce this error right now.

[rallytime]

Ok, I can confirm this bug. Thanks very much for this excellent report - it made it easy to reproduce once I realized what was going on. The https way of authenticating explained here works just fine, but the ssh method is indeed broken.

Steps to reproduce on a Fedora 21 VM:
Configure salt-minion running on 2014.7.5 with gitfs with your github ssh key:

# cat /etc/salt/minion
file_client: local
fileserver_backend:
  - git
gitfs_remotes:
  - git@github.com:rallytime/salt-jenkins.git:
    - pubkey: /root/.ssh/<your-key>.pub
    - privkey: /root/.ssh/<your-key>

Install pygit 2:

yum install python-pygit2

Then try to run a state from the your gitfs repository listed in the minion config:

# salt-call --local state.sls git.salt -l debug
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: 45.33.113.250
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] The `lspci` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] Mako not available
[DEBUG   ] Registered VCS backend: git
[DEBUG   ] Registered VCS backend: hg
[DEBUG   ] Registered VCS backend: svn
[DEBUG   ] Registered VCS backend: bzr
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] Updating git fileserver cache
[DEBUG   ] Set lock for git@github.com:rallytime/salt-jenkins.git
[DEBUG   ] gitfs is fetching from git@github.com:rallytime/salt-jenkins.git
[DEBUG   ] gitfs received 0 objects for remote git@github.com:rallytime/salt-jenkins.git
[DEBUG   ] gitfs cleaned the following stale refs: ['refs/remotes/origin/master']
[DEBUG   ] Removed lock for git@github.com:rallytime/salt-jenkins.git
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] The `lspci` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] The `dmidecode` binary is not available on the system. GPU grains will not be available.
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://git/salt.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://git/salt/init.sls'
local:
    Data failed to compile:
----------
    No matching sls found for 'git.salt' in env 'base'

ping @terminalmage

[deuscapturus]

Fixed this by replacing /usr/lib/python2.6/site-packages/salt/fileserver/gitfs.py with the one found in here https://github.com/saltstack/salt/releases/download/v2014.7.1/salt-2014.7.1.tar.gz

[rallytime]

@deuscapturus Do you think this is a duplicate of the other bug you filed a couple of days ago? If so, I know that we usually keep the "earliest" bugs open and close new duplicates, but since I've outlined some reproduction steps and narrowed this down and you've provided some helpful info here as well, I'm inclined to keep this one open and close the other one. What do you think?

[terminalmage]

This was caused by a bugfix where branches/tags removed from the remote repo were still showing up as salt fileserver environments. pygit2 doesn't know how to detect remote refs, so I added code that uses the git CLI to run git ls-remote to detect this, and then remove the stale branches/tags from the local checkout used by gitfs. This won't work, however, if the repository is authenticated, it leads to every branch/tag being detected as a stale ref and thus removed.

I've disabled stale ref cleaning for authenticated repos in #23094, and opened libgit2/pygit2#524 to request proper support for detecting stale refs in pygit2.

[terminalmage]

I renamed this issue to reflect the actual problem.

[ryepup]

Is there a suggested workaround until the next release?

[terminalmage]

Commenting out this line and adding a cleaned = [] line will give you a temporary workaround. For example:

#cleaned = _clean_stale(repo['repo'], refs_post)
cleaned = []

[ryepup]

@terminalmage is that in my master, minion, or both?

[ryepup]

looks like it only needs to change on the master. Thanks @terminalmage!

[terminalmage]

Yes, just the master. Sorry, should have been more specific.

[deuscapturus]

FYI. Applying the workaround 'cleaned = []' will fix most gitfs issues but will still cause an issue with file.recurse giving error "Recurse failed: none of the specified sources were found". For a complete fix follow the workaround I specified above.