IOError: [Errno 13] Permission denied: '/var/cache/salt/master/.dfn' when using python salt.wheel module

[onsmribah]

Hi,

I am using salt.wheel python module to manage minion keys. But when it throws the error below when trying to delete a minion key. However, running salt-key -d <minion_id> works fine.
Am using an unprivileged user to run salt-master and i also changed the ownership of the following dirs :
/var/cache/salt
/var/log/salt
/etc/salt/pki

salt version :

$ salt-master --versions-report
                  Salt: 2015.5.3
                Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
                Jinja2: 2.8
              M2Crypto: 0.21.1
        msgpack-python: 0.4.6
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 14.0.1
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: Not Installed
               Tornado: 4.2.1
 Debian source package: 2015.5.3+ds-1trusty1

Exception occurred in wheel key.delete: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 317, in low
    data['return'] = self.functions[fun](*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/wheel/key.py", line 57, in delete
    return skey.delete_key(match)
  File "/usr/lib/python2.7/dist-packages/salt/key.py", line 755, in delete_key
    salt.crypt.dropfile(self.opts['cachedir'], self.opts['user'])
  File "/usr/lib/python2.7/dist-packages/salt/crypt.py", line 50, in dropfile
    with salt.utils.fopen(dfn, 'w+') as fp_:
  File "/usr/lib/python2.7/dist-packages/salt/utils/__init__.py", line 1045, in fopen
    fhandle = open(*args, **kwargs)
IOError: [Errno 13] Permission denied: '/var/cache/salt/master/.dfn'

[jfindlay]

@onsmribah, thanks for the report.

[dmacvicar]

I am seeing the same, but running salt-key

[dmacvicar]

The file /var/cache/salt/master/.dfn is there when calling salt-key, it will fail. If you wit a bit, the file goes away and then salt-key works.

[dmacvicar]

I can reproduce this almost instantly now:

git clone git@github.com:dmacvicar/salt-opensuse-playground.git
git checkout salty
vagrant up
vagrant ssh master
sudo -s
salt-key -R
salt-key -D

Proceed? [N/y] y
Traceback (most recent call last):
  File "/usr/bin/salt-key", line 10, in <module>
    salt_key()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 285, in salt_key
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/key.py", line 32, in run
    key.run()
  File "/usr/lib/python2.7/site-packages/salt/key.py", line 427, in run
    self.delete_all()
  File "/usr/lib/python2.7/site-packages/salt/key.py", line 226, in delete_all
    self.delete('*')
  File "/usr/lib/python2.7/site-packages/salt/key.py", line 209, in delete
    self.key.delete_key(match_dict=matches)
  File "/usr/lib/python2.7/site-packages/salt/key.py", line 829, in delete_key
    salt.crypt.dropfile(self.opts['cachedir'], self.opts['user'])
  File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 59, in dropfile
    with salt.utils.fopen(dfn, 'wb+') as fp_:
  File "/usr/lib/python2.7/site-packages/salt/utils/__init__.py", line 1204, in fopen
    fhandle = open(*args, **kwargs)
IOError: [Errno 13] Permission denied: '/var/cache/salt/master/.dfn'

[dmacvicar]

So both reject_key and delete_key perform a request to rotate the key, which calls dropfile().

dropfile() (

salt/salt/crypt.py

Line 53 in 6f95752

def dropfile(cachedir, user=None):

) creates /var/cache/salt/master/.dfn with owner salt and permissions 400.

It tries to create the file with salt.utils.fopen(dfn, 'wb+') so, if it exists it should truncate it, but the logic clashes with the fact that we can't write the file if it already exists due to the umask.

If you call reject when delete has executed before and the master has not yet processed the request ( /var/cache/salt/master/.dfn) still exists, then the second operation fails to truncate the existing file.

The creation of the file has a comment:

# set a mask (to avoid a race condition on file creation) and store original.
mask = os.umask(191)

• What race condition is trying to prevent setting an umask that takes write permission out?
• If the file is being open with wb+ what is the point on removing the write, the process would never be able to open/truncate it because of the permission being 400?.
• If a second salt-key operation (remove/reject) and the previous one asked for AES key rotation by dropping /var/cache/salt/master/.dfn and the master has not yet picked the request (and delete the file), is is safe to just ignore it and do not request it again? In other words, is it safe to make dropfile() not do anything if /var/cache/salt/master/.dfn exists?

I could contribute a fix around these ideas, but I need some information about the intention of the design. @thatch45 ?

[jfindlay]

Excellent work. Thanks, @dmacvicar.