postgresql_user & become_user fails

[matonb]

ISSUE TYPE

• Bug Report

COMPONENT NAME

postgresql_user

ANSIBLE VERSION

ansible 2.3.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

OS / ENVIRONMENT

CentOS 7.3 - CentOS 7.3

SUMMARY

STEPS TO REPRODUCE

---
- name: "Create foreman PostgreSQL user"
  become:      true
  become_user: postgres
  postgresql_user:
    name:            foreman
    password:        "{{foreman_db_password}}"
    role_attr_flags: LOGIN,NOSUPERUSER,NOCREATEDB,NOCREATEUSER

EXPECTED RESULTS

PostgreSQL user foreman created

ACTUAL RESULTS

Doesn't become postgres user and fails to connect to database

TASK [linux/postgresql : Create foreman PostgreSQL user] ***********************
fatal: [testlab004]: FAILED! => {"changed": false, "failed": true, "msg": "unable to connect to database: FATAL:  Peer authentication failed for user \"postgres\"\n"}

PostgreSQL server log:

< 2017-08-19 10:00:46.657 BST > LOG:  provided user name (postgres) and authenticated user name (root) do not match
< 2017-08-19 10:00:46.657 BST > FATAL:  Peer authentication failed for user "postgres"

[ansibot]

cc @matburt @nerzhul
click here for bot help

[nrwahl2]

Could you run it like this and then run an ls -l on /tmp/testfile.txt to verify that the task is failing to become the postgres user? In your "ACTUAL RESULTS" you've said it doesn't become the postgres user.

---
- name: "Create test file as user postgres"
  become:      true
  become_user: postgres
  shell: 'echo test > /tmp/testfile.txt'

[matonb]

I just ran this test:

- name: Debug
  become:      yes
  become_user: postgres
  command:     whoami
  register:    whoami
- debug: msg="{{whoami.stdout}}"

Actual output:

TASK [linux/postgresql : debug] ************************************************
ok: [testlab004] => {
    "msg": "root"
}

[nrwahl2]

Huh. Try it with ansible-playbook -vvvv? Curious where it's failing.

[matonb]

TASK [linux/postgresql : Create foreman PostgreSQL user] ***********************
task path: /home/matonb/ansible/roles/linux/postgresql/tasks/createusers.yml:8
Using module file /usr/lib/python2.7/site-packages/ansible/modules/database/postgresql/postgresql_user.py
<192.168.1.4> ESTABLISH SSH CONNECTION FOR USER: ansible
<192.168.1.4> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/matonb/.ssh/ansible_rsa"' -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/matonb/.ansible/cp/cb6e190d5c 192.168.1.4 '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<192.168.1.4> (0, '/home/ansible\n', 'OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 56: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 17857\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<192.168.1.4> ESTABLISH SSH CONNECTION FOR USER: ansible
<192.168.1.4> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/matonb/.ssh/ansible_rsa"' -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/matonb/.ansible/cp/cb6e190d5c 192.168.1.4 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589 `" && echo ansible-tmp-1503299607.13-241685803658589="` echo /home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589 `" ) && sleep 0'"'"''
<192.168.1.4> (0, 'ansible-tmp-1503299607.13-241685803658589=/home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589\n', 'OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 56: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 17857\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<192.168.1.4> PUT /tmp/tmpipKbPh TO /home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589/postgresql_user.py
<192.168.1.4> SSH: EXEC sshpass -d12 sftp -o BatchMode=no -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/matonb/.ssh/ansible_rsa"' -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/matonb/.ansible/cp/cb6e190d5c '[192.168.1.4]'
<192.168.1.4> (0, 'sftp> put /tmp/tmpipKbPh /home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589/postgresql_user.py\n', 'OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 56: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 17857\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 6 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/ansible size 0\r\ndebug3: Looking up /tmp/tmpipKbPh\r\ndebug3: Sent message fd 6 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589/postgresql_user.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:1724\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 1724 bytes at 65536\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<192.168.1.4> ESTABLISH SSH CONNECTION FOR USER: ansible
<192.168.1.4> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/matonb/.ssh/ansible_rsa"' -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/matonb/.ansible/cp/cb6e190d5c 192.168.1.4 '/bin/sh -c '"'"'chmod u+x /home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589/ /home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589/postgresql_user.py && sleep 0'"'"''
<192.168.1.4> (0, '', 'OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 56: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 17857\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<192.168.1.4> ESTABLISH SSH CONNECTION FOR USER: ansible
<192.168.1.4> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/matonb/.ssh/ansible_rsa"' -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/matonb/.ansible/cp/cb6e190d5c -tt 192.168.1.4 '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=imgajdoprytcwdrqvrhxwqcaznxqipgu] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-imgajdoprytcwdrqvrhxwqcaznxqipgu; /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589/postgresql_user.py; rm -rf "/home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<192.168.1.4> (0, '\r\n{"msg": "unable to connect to database: FATAL:  Peer authentication failed for user \\"postgres\\"\\n", "failed": true, "invocation": {"module_args": {"login_password": "", "ssl_rootcert": null, "ssl_mode": "prefer", "name": "foreman", "no_password_changes": false, "login_user": "postgres", "login_host": "", "expires": null, "db": "", "role_attr_flags": "LOGIN,NOSUPERUSER,NOCREATEDB,NOCREATEUSER", "login_unix_socket": "", "state": "present", "user": "foreman", "encrypted": false, "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "port": "5432", "fail_on_user": true, "priv": null}}}\r\n', 'OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 56: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 17857\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to 192.168.1.4 closed.\r\n')
fatal: [testlab004]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "db": "", 
            "encrypted": false, 
            "expires": null, 
            "fail_on_user": true, 
            "login_host": "", 
            "login_password": "", 
            "login_unix_socket": "", 
            "login_user": "postgres", 
            "name": "foreman", 
            "no_password_changes": false, 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": "5432", 
            "priv": null, 
            "role_attr_flags": "LOGIN,NOSUPERUSER,NOCREATEDB,NOCREATEUSER", 
            "ssl_mode": "prefer", 
            "ssl_rootcert": null, 
            "state": "present", 
            "user": "foreman"
        }
    }, 
    "msg": "unable to connect to database: FATAL:  Peer authentication failed for user \"postgres\"\n"
}
	to retry, use: --limit @/home/matonb/ansible/linux_postgresql.retry

PLAY RECAP *********************************************************************
testlab004                 : ok=11   changed=1    unreachable=0    failed=1   

[nrwahl2]

Yep... to highlight the relevant line from the above snippet:

<192.168.1.4> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/matonb/.ssh/ansible_rsa"' -o User=ansible -o ConnectTimeout=10 -o ControlPath=/home/matonb/.ansible/cp/cb6e190d5c -tt 192.168.1.4 '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=imgajdoprytcwdrqvrhxwqcaznxqipgu] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-imgajdoprytcwdrqvrhxwqcaznxqipgu; /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589/postgresql_user.py; rm -rf "/home/ansible/.ansible/tmp/ansible-tmp-1503299607.13-241685803658589/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''

sudo -H -S -p "[sudo via ansible, key=imgajdoprytcwdrqvrhxwqcaznxqipgu] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-imgajdoprytcwdrqvrhxwqcaznxqipg

I'm going to continue to follow this in the hopes that someone more experienced with Ansible privilege escalation can enlighten me. The docs don't cover this scenario and I've been unable to replicate it on my system. It's only attempting to become root; it's never attempting to become postgres user.

Unless it's an issue with sudo rules.

[matonb]

@nrwahl2 Thanks for your help so far

[matonb]

@nrwahl2

sudoers...

I connect to the target machines as user ansible and become root

ansible	ALL=(ALL)	NOPASSWD: ALL

[nrwahl2]

One last question from me: Is that the last line in /etc/sudoers? If not, could you try the playbook again after moving it to the bottom to ensure nothing is taking precedence over it? I doubt that would cause the behavior you're seeing it; however, I want to rule out as many procedural/system config issues as possible until someone smarter than I am can jump in and fix this :)

[matonb]

It's not a ordering issue in sudoers, it something in our inventory (I used a fresh Ansible inventory and the whoami test which worked as expected).
No idea what yet, but when I find it I'll post what is causing this strange behavior....

[matonb]

ansible_become_user: 'root'

Was in one of the group_vars files which appears to have been causing the problem.

This is surely a bug?, if a role specifically sets become_user why is it overridden by a group_var value?

[nrwahl2]

tl;dr: Not a bug, but counter-intuitive.

From what I can tell, this behavior is not explicitly documented for privilege escalation. The closest thing I can find in the Privilege Escalation doc is this:

Only one method may be enabled per host
Methods cannot be chained. You cannot use sudo /bin/su - to become a user, you need to have privileges to run the command as that user in sudo or be able to su directly to it (the same for pbrun, pfexec or other supported methods).

(http://docs.ansible.com/ansible/latest/become.html#only-one-method-may-be-enabled-per-host)

But that's not really what we're dealing with here, is it? Further, the Variables doc suggests exactly what you were thinking -- that the role- or task-specific setting should override the host- or group-specific setting:

In 2.x, we have made the order of precedence more specific (with the last listed variables winning prioritization):

role defaults [1]
inventory INI or script group vars [2]
inventory group_vars/all
playbook group_vars/all
inventory group_vars/*
playbook group_vars/*
inventory INI or script host vars [2]
inventory host_vars/*
playbook host_vars/*
host facts
play vars
play vars_prompt
play vars_files
role vars (defined in role/vars/main.yml)
block vars (only for tasks in block)
task vars (only for the task)
role (and include_role) params
include params
include_vars
set_facts / registered vars
extra vars (always win precedence)

(http://docs.ansible.com/ansible/latest/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable)

Shortly after that list, there's a brief aside about remote_user. This section suggests that ansible_become_user is a connection variable and is treated differently from group_vars variables:

Another important thing to consider (for all versions) is that connection variables override config, command line and play/role/task specific options and directives. For example:

ansible -u lola myhost
This will still connect as ramon because ansible_ssh_user is set to ramon in inventory for myhost. For plays/tasks this is also true for remote_user:

- hosts: myhost
  tasks:
   - command: i'll connect as ramon still
     remote_user: lola
This is done so host-specific settings can override the general settings. These variables are normally defined per host or group in inventory, but they behave like other variables. If you want to override the remote user globally (even over inventory) you can use extra vars:

ansible... -e "ansible_user=<user>"
You can also override as a normal variable in a play:

- hosts: all
  vars:
    ansible_user: lola
  tasks:
    - command: i'll connect as lola!

As you can see above, you can still override connection variables like ansible_become_user -- but you can't do it by setting become_user in the task like you were trying to do.

Welp, I've learned a lot tonight.

See my own test cases below:

# Inventory file (set aliases to mask my hostnames):
test1 ansible_host=10.246.169.45

[my_group]
test2 ansible_host=10.247.10.207


### Test case 1: set to root in group_vars, not set in task ###
# group_vars/my_group.yml
---
ansible_become: true
ansible_become_user: root

# test.yml
---
- hosts: all
  connection: ssh
  gather_facts: no
  tasks:
    - name: print whoami
      command: whoami
      register: result
    - debug: var=result.stdout

# Run results
TASK [debug] ******************************************************************************************************
ok: [test1] => {
    "result.stdout": "nwahl"
}
ok: [test2] => {
    "result.stdout": "root"
}



### Test case 2: not set in group_vars, become_user set to testuser in task ###
# group_vars/my_group.yml
---

# test.yml
---
- hosts: all
  connection: ssh
  gather_facts: no
  tasks:
    - name: print whoami
      command: whoami
      become: true
      become_user: testuser
      register: result
    - debug: var=result.stdout

# Run results
TASK [debug] ******************************************************************************************************
ok: [test1] => {
    "result.stdout": "testuser"
}
ok: [test2] => {
    "result.stdout": "testuser"
}



### Test case 3: set to root in group_vars, become_user set to testuser in task ###
# group_vars/my_group.yml
---
ansible_become: true
ansible_become_user: root

# test.yml
---
- hosts: all
  connection: ssh
  gather_facts: no
  tasks:
    - name: print whoami
      command: whoami
      become: true
      become_user: testuser
      register: result
    - debug: var=result.stdout

# Run results
TASK [debug] ******************************************************************************************************
ok: [test1] => {
    "result.stdout": "testuser"
}
ok: [test2] => {
    "result.stdout": "root"



### Test case 4: set to root in group_vars, ansible_become_user set to testuser in task ###
# group_vars/my_group.yml
---
ansible_become: true
ansible_become_user: root

# test.yml
---
- hosts: all
  connection: ssh
  gather_facts: no
  tasks:
    - name: print whoami
      command: whoami
      register: result
      ansible_become: true
      ansible_become_user: testuser
    - debug: var=result.stdout

# Run results
ERROR! 'ansible_become_user' is not a valid attribute for a Task

The error appears to have been in '/u/nwahl/test.yml': line 6, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  tasks:
    - name: print whoami
      ^ here



### Test case 5: set to root in group_vars, ansible_become_user set to testuser in playbook vars ###
# group_vars/my_group.yml
---
ansible_become: true
ansible_become_user: root

# test.yml
---
- hosts: all
  connection: ssh
  gather_facts: no
  vars:
    ansible_become: true
    ansible_become_user: testuser
  tasks:
    - name: print whoami
      command: whoami
      register: result
    - debug: var=result.stdout

# Run results
TASK [debug] ******************************************************************************************************
ok: [test1] => {
    "result.stdout": "testuser"
}
ok: [test2] => {
    "result.stdout": "testuser"
}

[matonb]

@nrwahl2

Thanks for persisting with this, your comments have been very helpful.

---

As you can see above, you can still override connection variables like ansible_become_user -- but you can't do it by setting become_user in the task like you were trying to do.

Indeed, not very intuitive at all.... But a lesson learnt.

Thanks again for your help.

[ansibot]

cc @Andersson007 @amenonsen @andytom @dagwieers @jbscalia @kostiantyn-nemchenko
click here for bot help

[ansibot]

cc @Dorn- @kustodian
click here for bot help

[kostiantyn-nemchenko]

notabug