salt-ssh uses sudo to create cache dir, later fails to access it #38458

duk3luk3 · 2016-12-28T10:45:43Z

Description of Issue/Question

Any states using the file cache fail in salt-ssh 2016.11.1 due to permission problems. Apparently the file is created as root but later accessed with user rights.

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ salt-ssh --refresh --user $(whoami) -L 'localhost' state.sls test test=True -l debug
[INFO    ] Loading Saltfile from '/srv/saltstack/Saltfile'
[DEBUG   ] Reading configuration from /srv/saltstack/Saltfile
[DEBUG   ] Reading configuration from /srv/saltstack/etc/master
[DEBUG   ] Configuration file path: /srv/saltstack/etc/master
....
[ERROR   ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var/cache/salt/minion/salt-ssh'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/__init__.py", line 967, in run_wfunc
    result = self.wfuncs[self.fun](*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/wrapper/state.py", line 95, in sls
    st_kwargs['id_'])
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/state.py", line 204, in prep_trans_tar
    path = file_client.cache_file(name, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 178, in cache_file
    return self.get_url(path, '', True, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 486, in get_url
    result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1025, in get_file
    rel_path, saltenv, cachedir=cachedir) as cache_dest:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 150, in _cache_loc
    os.makedirs(destdir)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var/cache/salt/minion/salt-ssh'
[DEBUG   ] LazyLoaded nested.output
localhost:
    An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var/cache/salt/minion/salt-ssh'

$ ls -al /var/tmp/
total 28
drwxrwxrwt  7 root     root    4096 Dec 28 11:26 .
drwxr-xr-x 12 root     root    4096 Nov 23  2015 ..
drwxrwx---  4 root     tumuser 4096 Dec 28 11:33 .erlacher_f62e6b_salt

Setup

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ cat states/test/
init.sls  test.foo  

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ cat states/test/init.sls 
'/tmp/foo':
  file.managed:
    - source: salt://test/test.foo

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ cat states/test/test.foo 
testfoo

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ grep -A 5 'localhost' etc/roster 
localhost:
  host: vmrbg51
  sudo: True
ubuntumirror:
  host: vmrbg59
  sudo: True

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ cat Saltfile 
salt-ssh:
  config_dir: etc
  ssh_max_procs: 30
  wipe_ssh: true

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ cat etc/master 
pki_dir: /srv/saltstack/pki/
pillar_roots:
  base:
  - /srv/saltstack/pillar
file_roots:
  base:
  - /srv/saltstack/states
  - /srv/saltstack/secret
ssh_minion_opts:
  log_level: debug
  log_level_logfile: debug
  log_file: /var/log/salt/minion
ssh_list_nodegroups:
  lsvpn: vmrbg44, localhost

Steps to Reproduce Issue

See above

Versions Report

$ salt-ssh --versions-report
Salt Version:
           Salt: 2016.11.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: Not Installed
 
System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-93-generic
         system: Linux
        version: Ubuntu 14.04 trusty

Happens also with repository version of salt -

$ salt-ssh --versions-report
Salt Version:
           Salt: 2016.11.0-562-g5cba84c
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.2
 
System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-93-generic
         system: Linux
        version: Ubuntu 14.04 trusty

EDIT: Does not happen with v2016.9 tag (f76dc0f) from repo

The text was updated successfully, but these errors were encountered:

mirko · 2017-01-02T22:08:11Z

Having the same issue after an upgrade (from a much older version though).

$ salt-ssh 'X' 'state.sls' 'backup'
[ERROR   ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.Y_7c7bc9_salt/running_data'

gtmanfred · 2017-01-03T15:41:47Z

Thanks for reporting this, I am able to reproduce the issue, we will get it fixed.

Thanks,
Daniel

duk3luk3 · 2017-02-09T12:08:50Z

Is there a schedule for this, or a workaround? I think that this breaks salt-ssh for anyone using sudo.

gtmanfred · 2017-02-09T17:19:57Z

Let me ask and see if we have dev time to fix this in 2016.11.3

gtmanfred · 2017-02-09T17:30:31Z

It looks like we are going to be getting in all the 2016.11.3 blockers in this week, so we aren't going to be able to get this one in. It will be a blocker for 2016.11.4, so it will be in that next release.

Thanks,
Daniel

shallot · 2017-03-16T14:04:08Z

I think I ran across this bug, but there also seems to be a twist with trying to run without sudo on the master (where you run salt-ssh from). It seems to be trying to create file cache directories on the salt master but using the thin_dir from the roster, which is out of sync with the default behavior of everything else.

So with this roster:

minion-name:
  host: 1.2.3.4
  user: ubuntu
  priv: /path/to/key-file
  sudo: True
  thin_dir: /.salt-ssh-thin-dir

I get:

% salt-ssh -l debug [...]state.apply <state-using-file://something>
[...]
[DEBUG   ] In saltenv 'base', looking at rel_path 'file-settings/foobar' to resolve 'salt://file-settings/foobar'
[ERROR   ] An Exception occurred while executing state.apply: [Errno 13] Permission denied: '/.salt-ssh-thin-dir'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/__init__.py", line 970, in run_wfunc
    result = self.wfuncs[self.fun](*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/wrapper/state.py", line 288, in apply_
    return sls(mods, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/wrapper/state.py", line 95, in sls
    st_kwargs['id_'])
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/state.py", line 204, in prep_trans_tar
    path = file_client.cache_file(name, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 179, in cache_file
    return self.get_url(path, '', True, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 487, in get_url
    result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1029, in get_file
    rel_path, saltenv, cachedir=cachedir) as cache_dest:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 151, in _cache_loc
    os.makedirs(destdir)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/.salt-ssh-thin-dir'

I added some debug to see that it's trying to operate on this as destdir:

/.salt-ssh-thin-dir/running_data/var/cache/salt/minion/salt-ssh/minion-name/files/base/file-settings

Since none of this exists on the master, it fails at the first path component.

On the minion itself, /.salt-ssh-thin-dir/running_data/var/cache/salt/minion/ does exist, but is also owned by root, so it probably wouldn't work because of a lack of remote sudo, just like the original bug reporter mentioned.

shallot · 2017-03-17T16:04:35Z

For the benefit of random users who might be reading this dazed and confused at why salt:// file URIs with file source don't work with salt-ssh, it should probably be noted that this issue - while most annoying - isn't necessarily critical, because many use cases of source can be replaced with the use of contents, esp. with external pillars, which does seem to work fine with salt-ssh.

gtmanfred · 2017-03-30T20:14:40Z

This appears to have been introduced here, 71e0bd0

gtmanfred · 2017-03-30T21:12:22Z

@shallot @duk3luk3 does this happen on anything that is not localhost?

I am testing against other machines, and not getting the same error.

We do not need to specify the entire path here. _cache_loc in salt.fileclient will do that for us. If we specify cachedir here, it will use the /var/tmp/*/running_data/var/cache path which we do not want to use when on the master. This is intelligent enough to use the /var/tmp path on the minion and a /var/cache/salt/master type path on the master. Fixes saltstack#38458

duk3luk3 · 2017-03-31T06:14:43Z

Yes this definitely causes the same failure when executing the same state on localhost or a remote minion.

erlacher at vmrbg51 in /srv/saltstack on default < >
$ salt-ssh --refresh --user $(whoami) -L 'localhost' state.sls hostdb test=True
[ERROR   ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var'
localhost:
    An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var'

erlacher at vmrbg51 in /srv/saltstack on default <!>
$ salt-ssh --refresh --user $(whoami) -L 'vmrbg6' state.sls hostdb test=True

[ERROR   ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var'
vmrbg6:
    An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var'

erlacher at vmrbg51 in /srv/saltstack on default <!>
$ salt-ssh --versions-report
Salt Version:
           Salt: 2016.11.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-93-generic
         system: Linux
        version: Ubuntu 14.04 trusty

EDIT: I just tested it with the test state I set out in the initial report too, same results.

EDIT: Here is a self-contained reproduction: https://github.com/duk3luk3/salt-ssh-minimal/tree/salt-38458. It reproduces both from my arch linux workstation using salt-ssh 2016.11.3 as well as from my normal salt master machine using 2016.11.2.

EDIT: Your PR seems to fix it though! 👍

gtmanfred · 2017-03-31T14:14:00Z

It actually will work if you remove /var/tmp/*_salt and run it against a remote minion, it will work.

The problem was the commit I linked, accidentally moved the file cache on the master for salt:// links from /var/cache/salt/master/ to /var/tmp/<user>_<uuid>_salt/running_data, which would get overwritten when the thin tarball gets copied to the salt-ssh minion, and since you were using sudo, the running_data directory is owned by root:root on the minion, so this causes it to fail when the salt-ssh command tries to copy over more files for the salt_states.tgz

gtmanfred · 2017-03-31T20:15:11Z

The above commit has been merged, I am closing this issue

Thanks!
Daniel

shallot · 2017-04-04T08:01:47Z

Confirming that gtmanfred@2f0e2ed on top of 2016.11.3 fixes the problem, thanks!
When will this be in a released version?

gtmanfred · 2017-04-04T15:45:15Z

It will be in 2016.11.4 .

https://github.com/saltstack/salt/issues?q=is%3Aopen+is%3Aissue+label%3A2016.11.4+label%3ABlocker

mirko mentioned this issue Jan 2, 2017

salt-ssh, tty, sudo: Permission denied: '/tmp/.salt_xxxxxx_salt/running_data/var' #29699

Closed

gtmanfred added Bug broken, incorrect, or confusing behavior Core relates to code central or existential to Salt severity-high 2nd top severity, seen by most users, causes major problems P2 Priority 2 Salt-SSH TEAM Core labels Jan 3, 2017

gtmanfred added this to the Approved milestone Jan 3, 2017

gtmanfred added the severity-critical top severity, seen by most users, serious issues label Jan 3, 2017

gtmanfred added ZRELEASED - 2016.11.3 and removed ZRELEASED - 2016.11.3 labels Feb 9, 2017

gtmanfred added the ZRELEASED - 2016.11.4 label Feb 9, 2017

gtmanfred self-assigned this Mar 30, 2017

gtmanfred added the fixed-pls-verify fix is linked, bug author to confirm fix label Mar 31, 2017

gtmanfred mentioned this issue Mar 31, 2017

allow file_client to figure out cachedir #40442

Merged

gtmanfred closed this as completed Mar 31, 2017

NunoHiggs mentioned this issue Jun 9, 2020

Issues first bootstrapping minions: Add with SSH failed with salt permission denied. uyuni-project/uyuni#2312

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

salt-ssh uses sudo to create cache dir, later fails to access it #38458

salt-ssh uses sudo to create cache dir, later fails to access it #38458

duk3luk3 commented Dec 28, 2016 •

edited

Loading

mirko commented Jan 2, 2017

gtmanfred commented Jan 3, 2017

duk3luk3 commented Feb 9, 2017

gtmanfred commented Feb 9, 2017

gtmanfred commented Feb 9, 2017

shallot commented Mar 16, 2017

shallot commented Mar 17, 2017

gtmanfred commented Mar 30, 2017

gtmanfred commented Mar 30, 2017

duk3luk3 commented Mar 31, 2017 •

edited

Loading

gtmanfred commented Mar 31, 2017 •

edited

Loading

gtmanfred commented Mar 31, 2017

shallot commented Apr 4, 2017

gtmanfred commented Apr 4, 2017

salt-ssh uses sudo to create cache dir, later fails to access it #38458

salt-ssh uses sudo to create cache dir, later fails to access it #38458

Comments

duk3luk3 commented Dec 28, 2016 • edited Loading

Description of Issue/Question

Setup

Steps to Reproduce Issue

Versions Report

mirko commented Jan 2, 2017

gtmanfred commented Jan 3, 2017

duk3luk3 commented Feb 9, 2017

gtmanfred commented Feb 9, 2017

gtmanfred commented Feb 9, 2017

shallot commented Mar 16, 2017

shallot commented Mar 17, 2017

gtmanfred commented Mar 30, 2017

gtmanfred commented Mar 30, 2017

duk3luk3 commented Mar 31, 2017 • edited Loading

gtmanfred commented Mar 31, 2017 • edited Loading

gtmanfred commented Mar 31, 2017

shallot commented Apr 4, 2017

gtmanfred commented Apr 4, 2017

duk3luk3 commented Dec 28, 2016 •

edited

Loading

duk3luk3 commented Mar 31, 2017 •

edited

Loading

gtmanfred commented Mar 31, 2017 •

edited

Loading