can't read PID from lock file due to exception if gitfs_global_lock is enabled

[onorua]

Description of Issue/Question

I've got following error messages in the master log:

2016-06-17 13:51:23,932 [salt.utils.gitfs ][WARNING ][11506] gitfs_global_lock is enabled and update lockfile /var/cache/salt/master/gitfs/e46e9d09015aaab557a04b37c7c2cbe61110cb6bb584e625760b5b38fcd8bd60/.git/update.lk is present for gitfs remote 'https://bitbucket.org/onorua/salt-states.git'.
2016-06-17 13:51:23,932 [salt.utils.gitfs ][ERROR   ][11506] Exception 'invalid literal for int() with base 10: ''' caught while fetching gitfs remote 'https://bitbucket.org/onorua/salt-states.git'

Steps to Reproduce Issue

enable gitfs_global_lock (which is default)

Create empty file: /var/cache/salt/master/gitfs/e46e9d09015aaab557a04b37c7c2cbe61110cb6bb584e625760b5b38fcd8bd60/.git/update.lk

root@ip-172-16-236-25:/home/iarosmol# du /var/cache/salt/master/gitfs/e46e9d09015aaab557a04b37c7c2cbe61110cb6bb584e625760b5b38fcd8bd60/.git/update.lk
0   /var/cache/salt/master/gitfs/e46e9d09015aaab557a04b37c7c2cbe61110cb6bb584e625760b5b38fcd8bd60/.git/update.lk
root@ip-172-16-236-25:/home/iarosmol# cat /var/cache/salt/master/gitfs/e46e9d09015aaab557a04b37c7c2cbe61110cb6bb584e625760b5b38fcd8bd60/.git/update.lk
root@ip-172-16-236-25:/home/iarosmol# 

It seems we are missing one corner case, we have https://github.com/saltstack/salt/blob/develop/salt/utils/gitfs.py#L599 call to failhard, and we do failhard if pid == 0.

Versions Report

Salt Version:
           Salt: 2016.3.1

Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.24.0
         Python: 2.7.11+ (default, Mar 23 2016, 11:35:56)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.4.0-24-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[Ch3LL]

@onorua looks like i'm able to replicate this behavior. HEre is the full stack trace:

2016-06-20 11:33:39,479 [salt.utils.gitfs ][WARNING ][5853] gitfs_global_lock is enabled and update lockfile /var/cache/salt/master/gitfs/8165d5330f3b55daa7621809ba57fa92/.git/update.lk is present for gitfs remote 'https://github.com/Ch3LL/salt-states'.
2016-06-20 11:33:39,480 [salt.utils.gitfs ][ERROR   ][5853] Exception 'invalid literal for int() with base 10: ''' caught while fetching gitfs remote 'https://github.com/Ch3LL/salt-states'
Traceback (most recent call last):
  File "/home/ch3ll/git/salt/salt/utils/gitfs.py", line 2146, in fetch_remotes
    if repo.fetch():
  File "/home/ch3ll/git/salt/salt/utils/gitfs.py", line 382, in fetch
    with self.gen_lock(lock_type='update'):
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/ch3ll/git/salt/salt/utils/gitfs.py", line 510, in gen_lock
    self._lock(lock_type=lock_type, failhard=True)
  File "/home/ch3ll/git/salt/salt/utils/gitfs.py", line 416, in _lock
    pid = int(fd_.readline().rstrip())
ValueError: invalid literal for int() with base 10: ''

ping @terminalmage

[terminalmage]

Fixed in #34179. I was correctly catching the ValueError when the pidfile is empty, however this means that when we catch this exception, a bare call to raise raises the ValueError instead of the exception caught because the lock file was present.

[terminalmage]

I'll add that in normal operation, the process of obtaining a lock will lay down a lock file containing a pid, so this should not affect normal usage.

[onorua]

Thanks for the fix!