Failure using become on HP-UX: Failed to set file mode on remote files

[rpettersen]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Core

ANSIBLE VERSION

ansible 2.2.0.0
  config file = /ansible/projects/test/ansible.cfg
  configured module search path = Default w/o overrides

CONFIGURATION

[defaults] 
become_method=sudo
sudo_flags=-H
pipelining=True
retry_files_enabled=False
allow_world_readable_tmpfiles=True
forks=20
serial=20

OS / ENVIRONMENT

Running Ansible from RHEL 6 towards node with HP-UX (11+, 11.23 and others)
Failure only on HP-UX

SUMMARY

Whenever using playbook with "become: True", it fails with:
Failed to set file mode on remote files (rc: 2, err: chmod: can't change /tmp/ansible-tmp-1478507525.44-151149964677265/: Not owner\nchmod: can't access /tmp/ansible-tmp-1478507525.44-151149964677265/setup.py\n)

STEPS TO REPRODUCE

Run a playbook with become: True, with any task. The setup task will fail.

playbook.yml:

---
# This playbook to be run towards HP-UX host to test permission bug with become
#
- name: "Test playbook"
  hosts: all
  become: true
  become_user: xREMOTEUSERx
  become_method: sudo
  tasks:
  - name: Run whoami
    command: whami

EXPECTED RESULTS

Playbook should run successfully, running command whoami as xREMOTEUSERx.

ACTUAL RESULTS

Fails during setup with "Failed to set file mode on remote files....":

$ ansible-playbook -vvv -i "xREMOTEHOSTx," play-test.yml 
Using /ansible/projects/test/ansible.cfg as config file

PLAYBOOK: play-test.yml ********************************************************
1 plays in play-test.yml

PLAY [Test playbook] ***********************************************************

TASK [setup] *******************************************************************
Using module file /pyVE/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1478507525.44-151149964677265 `" && echo ansible-tmp-1478507525.44-151149964677265="` echo /tmp/ansible-tmp-1478507525.44-151149964677265 `" ) && sleep 0'"'"''
<xREMOTEHOSTx> PUT /tmp/tmplSznsB TO /tmp/ansible-tmp-1478507525.44-151149964677265/setup.py
<xREMOTEHOSTx> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r '[xREMOTEHOSTx]'
<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'setfacl -m u:xREMOTEUSERx:rx /tmp/ansible-tmp-1478507525.44-151149964677265/ /tmp/ansible-tmp-1478507525.44-151149964677265/setup.py && sleep 0'"'"''
<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'chmod u+x /tmp/ansible-tmp-1478507525.44-151149964677265/ /tmp/ansible-tmp-1478507525.44-151149964677265/setup.py && sleep 0'"'"''
<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'chown xREMOTEUSERx /tmp/ansible-tmp-1478507525.44-151149964677265/ /tmp/ansible-tmp-1478507525.44-151149964677265/setup.py && sleep 0'"'"''
 [WARNING]: Using world-readable permissions for temporary files Ansible needs to create when becoming an unprivileged user. This may be insecure. For
information on securing this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user

<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'chmod a+rx /tmp/ansible-tmp-1478507525.44-151149964677265/ /tmp/ansible-tmp-1478507525.44-151149964677265/setup.py && sleep 0'"'"''
fatal: [xREMOTEHOSTx]: FAILED! => {
    "failed": true, 
    "msg": "Failed to set file mode on remote files (rc: 2, err: chmod: can't change /tmp/ansible-tmp-1478507525.44-151149964677265/: Not owner\nchmod: can't access /tmp/ansible-tmp-1478507525.44-151149964677265/setup.py\n)"
}

PLAY RECAP *********************************************************************
xREMOTEHOSTx                   : ok=0    changed=0    unreachable=0    failed=1   

File permissions on the files left on remote host:

$ ls -lasd /tmp/ans*
   0 drwx------   2 xREMOTEUSERx     xMYUSERx          96 Nov  7 09:32 /tmp/ansible-tmp-1478507525.44-151149964677265

[alikins]

Just to verify, I assume setting the config variable 'allow_world_readable_tmpfiles=False' fails as well (results in errors like 'Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user') ?

[rpettersen]

Correct, setting allow_world_readable_tmpfiles=False gives:
Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (....)

[abadger]

What are the permissions on /tmp/ ? I don't see where this is happening in our code but a setuid bit on the /tmp/ directory (if it's owned by xREMOTEUSERx) would do this.

needs_info

[abadger]

I see another thing in your setup that could mitigate this. I see you have pipelining=True in your ansible.cfg file but it's not being used here. The reason is that it needs to appear in a [ssh_connection] section of the config file:

[defaults] 
become_method=sudo
sudo_flags=-H
retry_files_enabled=False
allow_world_readable_tmpfiles=True
forks=20
serial=20
[ssh_connection]
pipelining=True

There's a few modules where this doesn't work since the module uploads other temporary files. (The most notable of these is copy).

[abadger]

If it's not due to some configuration on the remote box (like setuid bit on the /tmp/ directory) then the only other way I can see our code hitting this is if invoking chown xREMOTEUSERx filename succeeds in changing the ownership of the file but reports failure (return code is nonzero). That seems like it would be a major bug in HP-UX's chown so I think that's less likely than a setuid bit set on the directory but it would be the next thing to look at.

[rpettersen]

Permission on /tmp are the same as they are on Linux:

# ls -laRs /tmp/
  16 drwxrwxrwt   9 root       root          8192 Nov 22 10:32 .
  16 drwxr-xr-x  19 root       root          8192 Nov 22 10:32 ..
   0 drwx------   2 xREMOTEUSERx     xMYSUERx          96 Nov 22 10:32 ansible-tmp-1479807140.27-162933712974831

/tmp/ansible-tmp-1479807140.27-162933712974831:
total 208
   0 drwx------   2 xREMOTEUSERx     xMYSUERx          96 Nov 22 10:32 .
  16 drwxrwxrwt   9 root       root          8192 Nov 22 10:32 ..
 192 -rwx------   1 xMYSUERx     xMYSUERx       94784 Nov 22 10:32 setup.py

Removed "pipelining=True" from playbook:

$ ansible-playbook -vvv -i "xREMOTEHOSTx," play-hpux_bugcheck.yml           
Using /ansible/projects/test/ansible.cfg as config file

PLAYBOOK: play-hpux_bugcheck.yml ***********************************************
1 plays in play-hpux_bugcheck.yml

PLAY [Test playbook] ***********************************************************

TASK [setup] *******************************************************************
Using module file /pyVE/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1479817938.56-249393849198957 `" && echo ansible-tmp-1479817938.56-249393849198957="` echo /tmp/ansible-tmp-1479817938.56-249393849198957 `" ) && sleep 0'"'"''
<xREMOTEHOSTx> PUT /tmp/tmpI8w7Mi TO /tmp/ansible-tmp-1479817938.56-249393849198957/setup.py
<xREMOTEHOSTx> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r '[xREMOTEHOSTx]'
<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'setfacl -m u:xREMOTEUSERx:rx /tmp/ansible-tmp-1479817938.56-249393849198957/ /tmp/ansible-tmp-1479817938.56-249393849198957/setup.py && sleep 0'"'"''
<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'chmod u+x /tmp/ansible-tmp-1479817938.56-249393849198957/ /tmp/ansible-tmp-1479817938.56-249393849198957/setup.py && sleep 0'"'"''
<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'chown xREMOTEUSERx /tmp/ansible-tmp-1479817938.56-249393849198957/ /tmp/ansible-tmp-1479817938.56-249393849198957/setup.py && sleep 0'"'"''
 [WARNING]: Using world-readable permissions for temporary files Ansible needs to create when becoming an unprivileged user. This may be insecure. For
information on securing this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user

<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'chmod a+rx /tmp/ansible-tmp-1479817938.56-249393849198957/ /tmp/ansible-tmp-1479817938.56-249393849198957/setup.py && sleep 0'"'"''
fatal: [xREMOTEHOSTx]: FAILED! => {
    "failed": true, 
    "msg": "Failed to set file mode on remote files (rc: 2, err: chmod: can't change /tmp/ansible-tmp-1479817938.56-249393849198957/: Not owner\nchmod: can't access /tmp/ansible-tmp-1479817938.56-249393849198957/setup.py\n)"
}

PLAY RECAP *********************************************************************
xREMOTEHOSTx                   : ok=0    changed=0    unreachable=0    failed=1   

Moved "pipelining=True" to ssh_connection section in ansible.cfg. Reran playbook, and get a different error:

$ ansible-playbook -vvv -i "xREMOTEHOSTx," play-hpux_bugcheck.yml 
Using /ansible/projects/test/ansible.cfg as config file

PLAYBOOK: play-hpux_bugcheck.yml ***********************************************
1 plays in play-hpux_bugcheck.yml

PLAY [Test playbook] ***********************************************************

TASK [setup] *******************************************************************
Using module file /pyVE/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<xREMOTEHOSTx> ESTABLISH SSH CONNECTION FOR USER: None
<xREMOTEHOSTx> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o ControlPath=/home/xMYUSERx/.ansible/cp/ansible-ssh-%h-%p-%r xREMOTEHOSTx '/bin/sh -c '"'"'sudo -H -u xREMOTEUSERx /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-cxkxkpixfadimlxqnbglnolzvhvlpxus; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
fatal: [xREMOTEHOSTx]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_name": "setup"
    }, 
    "module_stderr": "Last successful login:       Tue Nov 22 10:03:32 MET 2016   \nLast authentication failure: Thu Jan 22 14:55:59 MET 2015 212.18.129.111  \nTraceback (most recent call last):\n  File \"<stdin>\", line 10, in <module>\n  File \"/opt/iexpress/python/lib/python2.7/posixpath.py\", line 347, in abspath\n    cwd = os.getcwd()\nOSError: [Errno 13] Permission denied\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE"
}

PLAY RECAP *********************************************************************
xREMOTEHOSTx                   : ok=0    changed=0    unreachable=0    failed=1   

Looks like this command is the one failing:

# chown xREMOTEUSERx /tmp/ansible-tmp-1479807140.27-162933712974831/ /tmp/ansible-tmp-1479807140.27-162933712974831/setup.py
/tmp/ansible-tmp-1479807140.27-162933712974831/setup.py: Permission denied
# echo $?
1

First part succeeds, but then it is not able to change owner of setup.py.
If I change the order it succeeds:

# chown xREMOTEUSERx /tmp/ansible-tmp-1479807140.27-162933712974831/setup.py /tmp/ansible-tmp-1479807140.27-162933712974831/
# echo $?
0

[rpettersen]

So first, second part of this command fails, but is ignored:

# chown xREMOTEUSERx /tmp/ansible-tmp-1479807140.27-162933712974831/ /tmp/ansible-tmp-1479807140.27-162933712974831/setup.py
/tmp/ansible-tmp-1479807140.27-162933712974831/setup.py: Permission denied

Then this command fails because xMYSUSERx does not have permission on /tmp/ansible-tmp-1479807140.27-162933712974831:

# chmod a+rx /tmp/ansible-tmp-1479817938.56-249393849198957/ /tmp/ansible-tmp-1479817938.56-249393849198957/setup.py
chmod: can't access /tmp/ansible-tmp-1479817938.56-249393849198957/
chmod: can't access /tmp/ansible-tmp-1479817938.56-249393849198957/setup.py

[abadger]

@rpettersen. Ah ha! Okay. If part of the chown works and part of it fails then it makes sense. I think I can work up a fix for that.

[abadger]

Still working on this. A straightforward reorder wasn't enough because some modules call execute_module more than once (for instance, copy stats multiple files via the stat module in addition to running either the copy or file module). Subsequent calls to execute_module will run into the same problem since the directory will be owned by the become user and we won't be able to place more things into the temp directory at that point. Working on a fix via config file: you'll have to enable a config option that says not to try chown in fixup_perms because it doesn't follow modern semantics.

[abadger]

I've had a potentially better idea but it's a holiday in the United States so I'll post the idea here and try to get feedback from the other committers interested in this code on Monday.

Stepping back from this particular problem I realized that we have two competing problems in this area of the code. On some platforms the privileged user is not named "root" ("toor" on freebsd.) This was
discussed here: #15373 and a fix implemented here: #15482 . The change implemented was to try to chown the file and directory and if that failed fallback to another method.

This brings us to this bug. On platforms where unprivileged users on the remote side can give away files via chown, trying to chown first leads to the temp directory being unusable later by ansible (note: this is a per-task temp directory so it only becomes a problem when a task needs to use the temp directory more than once. copy is the action module we've identified as being the hardest to work around in this regard). We need to skip chown on these platforms as there's no way to make chown work correctly there.

Yesterday I proposed a config value to disable use of chown as a way to fix this. While that would work, it doesn't follow naturally from things the user might naturally relate. Disabling chown for unprivileged users is something that isn't mentioned in the playbook or needed for ansible's operation except for this one little quirk. It would be better to test a feature of the connection that ansible users are already familiar work with.

That brings me back to the remote_user. remote_user is something that ansible users have to be familiar with as it's the user that they are connecting to the remote machine via. Last time we stopped checking remote_user because we couldn't autodetect whether the remote_user had administrative capabilities but now we're thinking of using a config value to here anyway... So perhaps we should use a config value (and matching inventory variable) to tell ansible "What account names are administrators on the remote machine". Then a user who's logging into a remote machine with the administrative account "toor" would have to add an inventory var ansible_administrative_users=root,toor for their machine and the code would use chown to make the temp files properly readable. If they did not set this then chown would not be tried.

CC'ing @bcoca, @mattclay, and @nitzmahone as they're familiar with the code and @n-st as he reported the previous problem on FreeBSD.

[rpettersen]

I suppose having the choice in these cases to let the tmp-files be world readable is not an option (0755)?
I first thought that was what allow_world_readable_tmpfiles was for, and then not needing additional chown and chmod.
For me, it would be best if the solution/workaround to this problem could be set_fact on a play level, as I use most plays across multiple OS'es.

[abadger]

@rpettersen. allow_world_readable_tmpfiles will be needed in your case in addition to this new code.

The reason that we still try setfacl and chown when allow_world_readable_tmpfiles is set is for security. In most cases (not using become, using become to a privileged account, and ssh'ing as a privileged account that becomes an unprivileged one, unprivileged that becomes a second unprivileged account with working setfacl) the code can make things work without resorting to world readable tmpfiles (which risks disclosure of sensitive information like no_log module parameters).

So we only use allow_world_readable_tmpfiles as a fallback when those other methods have failed. The problem is determining failure in the chown case. It's not safe to try chown in the case that you have as the normal user can give away the directory but then not operate on it afterwards. For the case of the FreeBSD toor account, ansible has no way of autodetecting the accounts which are privileged on the remote machine so up to now it has had to try the remote chown in order to decide if the chown would work or not.

[bcoca]

fixed via #31677, which adds fine grained configuration for remote temps and admin users

[abadger]

This is not fixed until the code in this section of the old PR is updated to use the new config values from #31677 and merged: https://github.com/ansible/ansible/pull/20065/files#diff-69a29e19a76e98587550dab380483594

[abadger]

Alternative to doing something similar to #31677 would be to modify fixup_perms2() to be two-phase. call fixup_perms2([list, of, fiiles], commit=False) and the files are queued for permission changing later. Call fixup_perms2([list, of, files], commit=True) and the current list of files along with any previously submitted files will have their permissions changed. This way the fixup_perms2() function can order the files and directories that it was given.

Doing that allows custom plugins to do things wrong, though... they may use commit inappropriately which would either lead to this issue cropping up (If they committed too early) or not having the correct permissions on the files when they were executed (if they didn't call fixup_perms with commit=True between the last files being uploaded and the module being executed).

[dagwieers]

What is the status of this issue ?

[rgroten]

This issue is also affecting us on HP-UX

[bcoca]

'admin_users' were moved to shell plugins and this is now configurable globally, per group or per host. That should be enough to enable this functionality on HP/UX, but we cannot really test as we don't have access to the proprietary software.

So I'm closing this ticket for now, reopen if you still continue seeing issues.