Reboot action missing shutdown command from path on CentOS 7 (Ansible 2.7)

[shoekstra]

SUMMARY

Cannot reboot CentOS 6 hosts using the reboot action, get an error back that shutdown is not found in the path.

When SSHing with the same command ansible uses for reboot:

$ ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/Users/shoekstra/.ansible/cp/4f62e7acb1 -tt hostxyz '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-nniwnwnvijaznsoinmdjgxwtwwykewyp; which shutdown'"'"'"'"'"'"'"'"' && sleep 0'"'"''
...
...
which: no shutdown in (/usr/local/bin:/usr/bin)
...

When logged in (same result using sudo):

> echo $PATH
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/shoekstra/.local/bin:/home/shoekstra/bin

ISSUE TYPE

• Bug Report

COMPONENT NAME

reboot module

ANSIBLE VERSION

ansible 2.7.0
  config file = /Users/shoekstra/git/services/ansible-services/ansible.cfg
  configured module search path = [u'/Users/shoekstra/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/shoekstra/.local/share/virtualenvs/ansible-services-zeNucIgQ/lib/python2.7/site-packages/ansible
  executable location = /Users/shoekstra/.local/share/virtualenvs/ansible-services-zeNucIgQ/bin/ansible
  python version = 2.7.15 (default, Sep 18 2018, 20:16:18) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]

CONFIGURATION

ANSIBLE_PIPELINING(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = True
CACHE_PLUGIN(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = .facts
CACHE_PLUGIN_TIMEOUT(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = 86400
COMMAND_WARNINGS(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = False
DEFAULT_FORKS(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = 10
DEFAULT_GATHERING(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = smart
DEFAULT_ROLES_PATH(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = [u'/Users/shoekstra/git/services/ansible-services/roles']
DEFAULT_SCP_IF_SSH(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = true
DEFAULT_STDOUT_CALLBACK(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = skippy
HOST_KEY_CHECKING(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = False
INVENTORY_ENABLED(/Users/shoekstra/git/services/ansible-services/ansible.cfg) = [u'host_list', u'script', u'yaml', u'ini', u'constructed']

OS / ENVIRONMENT

Target OS is CentOS 7

CentOS Linux release 7.5.1804 (Core)

STEPS TO REPRODUCE

roles/reboot/tasks/reboot_linux.yml:

---
- name: reboot host
  reboot:

playbooks/reboot_if_pending.yml

---
- hosts: all
  become: true
  gather_facts: false
  serial:
  - "10%"
  - "20%"
  - "100%"

  tasks:
  - include_role: name=reboot tasks_from=detect_linux

  - name: End play if reboot not needed
    meta: end_play
    when: reboot_needed is not defined

  - include_role: name=reboot tasks_from=reboot_linux

EXPECTED RESULTS

shutdown command is found in path and host is restarted.

ACTUAL RESULTS

reboot action fails because it cannot find shutdown in the path

<hostxyz> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/Users/shoekstra/.ansible/cp/4f62e7acb1 -tt hostxyz '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-nniwnwnvijaznsoinmdjgxwtwwykewyp; shutdown -r 0 "Reboot initiated by Ansible"'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<hostxyz> (127, '/bin/sh: shutdown: command not found\r\n', 'OpenSSH_7.6p1, LibreSSL 2.6.2\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 77233\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 127\r\nShared connection to hostxyz closed.\r\n')
fatal: [hostxyz]: FAILED! => {
    "changed": false,
    "elapsed": 0,
    "msg": "Shutdown command failed. Error was /bin/sh: shutdown: command not found, OpenSSH_7.6p1, LibreSSL 2.6.2\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 77233\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 127\r\nShared connection to hostxyz closed.",
    "rebooted": false,
    "start": "2018-10-16T15:23:52.773845"
}

[shoekstra]

I've also tried setting the path via the environment option for the reboot action:

- name: reboot host
  reboot:
  environment:
    PATH: "/usr/sbin/"

This didn't seem to make a difference.

Also verified my connection/command is run as root by adding this to my playbook:

  tasks:
  - command: id
    register: id

  - debug:
      var: id.stdout_lines

stdout_lines output:

"uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023"

I did modify the reboot.py so that DEFAULT_SHUTDOWN_COMMAND = 'shutdown' is DEFAULT_SHUTDOWN_COMMAND = '/usr/sbin/shutdown' and then the action worked as expected.

Any pointers/suggestions would be welcome.

[Akasurde]

cc @samdoran

[shoekstra]

Something I forgot to add was the sudo command locally once SSHed:

$ sudo -H -S -n -u root /bin/sh -c 'which shutdown'
/usr/sbin/shutdown

So it appears the problem exists only via SSH, command locally works as expected.

[samdoran]

This is failing because the secure_path doesn't include shutdown. The default secure_path on CentOS 6 and CentOS 7 is secure_path = /sbin:/bin:/usr/sbin:/usr/bin. You can see the current secure_path by running sudo -l. What is the secure_path on the host you are trying reboot?

You list the target OS as "CentOS 6 CentOS Linux release 7.5.1804 (Core)". Is the target OS CentOS 6 or 7?

It's curious that /usr/sbin/shutdown is the path, which looks more like CentOS 7. I see /sbin/shutdown on CentOS 6.9. Regardless, the way to fix this is to add /sbin or /usr/sbin to secure_path.

We have a get_bin_path() method for modules that looks for a binary on the remote in a set of paths. I could look into doing something similar for this action plugin if fixing the secure_path isn't an option for you.

needs_info

[ansibot]

Hi @shoekstra, thank you for submitting this issue!

click here for bot help

[shoekstra]

Hi @samdoran,

Sorry for the confusion, it is indeed CentOS 7.

It turns out I didn't have secure_path set. When I set this the reboot action worked as expected.

Thanks for the help and prompt reply!

Stephen

[Alb0t]

I also do not have secure path set. I do not wish to change my sudoers file just to be able to reboot using ansible.. this is really silly..

[samdoran]

@Alb0t Agreed. I'll reopen this and work on a fix.

[Idanatcox]

I see the same issue ansible 2.7.2, RHEL 7.5

[BradFelmey]

Also confirmed on Ansible 2.7.2/RHEL 7.6. Agree with @Alb0t that we shouldn't have to introduce sudoers behavioral changes for module success.

[ansibot]

cc @nitzmahone
click here for bot help

[samdoran]

@shoekstra @BradFelmey Can you please test with #49272 to see if that fixes the issue? I added code to search common paths on the managed node to find the absolute path the binary.

[shoekstra]

@samdoran sorry for the delay! Just tested and it works for me.

Removed Defaults secure_path="/sbin:/bin:/usr/sbin:/usr/bin" from /etc/sudoers and tested shutdown isn't found in path:

$ ssh -t myhost sudo which shutdown
which: no shutdown in (/usr/local/bin:/usr/bin)
Connection to myhost closed.

Tested reboot successfully:

$ ansible -b -m reboot myhost
myhost | CHANGED => {
    "changed": true,
    "elapsed": 94,
    "rebooted": true
}

Thanks for resolving this one!