Ansible setup action with delegate_to tries to use the current python interpreter.

[pieteravonts]

I am running ansible in a local virtualenv and I want to execute the setup action on remote docker hosts.
When running the setup action normally, the boot command uses /usr/bin/python. However when I run the setup action with the delegate_to option then it wants to use the python interpreter from the local virtualenv (which of course does not exist on the remote docker host).

Ansible version:

ansible 2.6.5
  config file = None
  configured module search path = [u'/Users/Pieter/library']
  ansible python module location = /Users/Pieter/.local/share/virtualenvs/it-4glgr_h9/lib/python2.7/site-packages/ansible
  executable location = /Users/Pieter/.local/share/virtualenvs/it-4glgr_h9/bin/ansible
  python version = 2.7.14 (default, Nov  2 2017, 16:04:18) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]

Also tested on 2.7.0, with the same result.

Ansible config dump:

DEFAULT_CALLBACK_PLUGIN_PATH(env: ANSIBLE_CALLBACK_PLUGINS) = [u'/Users/Pieter/.local/share/virtualenvs/it-4glgr_h9/lib/python2.7/site-packages/ara/plugins/callbacks']
DEFAULT_FILTER_PLUGIN_PATH(env: ANSIBLE_FILTER_PLUGINS) = [u'/Users/Pieter/filter_plugins']
DEFAULT_MODULE_PATH(env: ANSIBLE_LIBRARY) = [u'/Users/Pieter/library']
DEFAULT_MODULE_UTILS_PATH(env: ANSIBLE_MODULE_UTILS) = [u'/Users/Pieter/library/utils']
DEFAULT_STRATEGY(env: ANSIBLE_STRATEGY) = mitogen_linear
DEFAULT_STRATEGY_PLUGIN_PATH(env: ANSIBLE_STRATEGY_PLUGINS) = [u'/Users/Pieter/Git/github/dw/mitogen/ansible_mitogen/plugins/strategy']
DEFAULT_TEST_PLUGIN_PATH(env: ANSIBLE_TEST_PLUGINS) = [u'/Users/Pieter/test_plugins']

Using the current git master of mitogen (ad44ad1).

Local OS: Mac OS X 10.13.6
Remote OS: Red Hat 7.5 remote docker containers.

Local Python: 2.7.14
Remote Python: 2.7.6

Example test playbook:

---
- hosts: host1
  gather_facts: false
  tasks:
    - name: Gather facts
      setup:
        gather_subset: '!all,!min,network'
    - name: Gather facts delegated
      setup:
        gather_subset: '!all,!min,network'
      delegate_to: host2
      delegate_facts: true

I see the following output (using -vvv):
For the first task:

TASK [Gather facts] *******************************************************************************************************************************************************************************************************************************************
task path: /Users/Pieter/test.yml:5
[pid 84760] 11:49:46.296844 D mitogen: unix.connect(path='/var/folders/ym/fytsdjsx2v50b1lx9kc825mm0000gn/T/mitogen_unix_6qtK_h')
[pid 84760] 11:49:46.298653 D mitogen: unix.connect(): local ID is 1, remote is 0
[pid 84759] 11:49:46.303352 D mitogen: mitogen.docker.Stream(u'default').connect()
[pid 84759] 11:49:46.534106 D mitogen: create_child() child 84761 fd 97, parent 84759, cmd: docker exec --interactive host1 /usr/bin/python -c "import codecs,os,sys;_=codecs.decode;exec(_(_(\"eNqFkV9PwjAUxZ/Zp9hb29BAO1FgSRMJCeqDf7IoPCgxG+u0cbRNNxj46b0MEzZ88O3+cs/tOTmN6EKYomeVlZh4jlYNUpkPkBn3hUnodWBONzbAjHLGyIkj2iQHW37kVW4KiaMmuCYsmlABgGGxB/s8LsF17QvhozR2ldLIj3VaL+VOrjZlnOSyXvc3hesnSvftvvw0GkHOzpmsK+rDrXSFMvo1vFjWtlJvlQNGk+hmztBStM+OGsActxe0jV2E16o0H1KHT5Ot0dANCwYjPh5cT1kQPfDZy2x+exWOBsPLMUHEg2crp0qJOUX3d8+PoH/TCOKsTAqtE28q3vGh99RYqaFt5BJEek7GKebBkI0IRd/KwkuZFSfdgqIqQYevyOyvwbSej/Weqav/1H9T8lbKH408r7A=\".encode(),\"base64\"),\"zip\"))"
[pid 84759] 11:49:46.535775 D mitogen: mitogen.docker.Stream(u'local.84761').connect(): child process stdin/stdout=97
[...]
[pid 84760] 11:49:59.330856 D mitogen: mitogen.core.Stream(u'unix_listener.84759').on_disconnect()
[pid 84760] 11:49:59.331946 D mitogen: Waker(Broker(0x10d76ba10) rfd=48, wfd=49).on_disconnect()
[pid 84759] 11:49:59.331657 D mitogen: mitogen.core.Stream(u'unix_client.84760').on_disconnect()
ok: [host1]

For the second task:

TASK [Gather facts delegated] *********************************************************************************************************************************************************************************************************************************
task path: /Users/Pieter/test.yml:8
[pid 84763] 11:49:59.436976 D mitogen: unix.connect(path='/var/folders/ym/fytsdjsx2v50b1lx9kc825mm0000gn/T/mitogen_unix_6qtK_h')
[pid 84763] 11:49:59.438816 D mitogen: unix.connect(): local ID is 1003, remote is 0
[pid 84759] 11:49:59.471901 D mitogen: mitogen.docker.Stream(u'default').connect()
[pid 84759] 11:49:59.600275 D mitogen: create_child() child 84764 fd 98, parent 84759, cmd: docker exec --interactive host2 /Users/Pieter/.local/share/virtualenvs/it-4glgr_h9/bin/python2 -c "import codecs,os,sys;_=codecs.decode;exec(_(_(\"eNqFkV9PwjAUxZ/Zp9jbbUMD60SBJUskJKgP/smi8KDEbKzTxtE23WDgp/cyTNjwwbf7yz235+Q0YotQFz0jjSDUsaxqkMxchEzbL0IDp4NzujE+8Rj3PHriiDXJ4pYfeZXrQpCoCbYJiyZUCGhY7NE+j0t0Xbth6EIa20oqcGOV1kuxE6tNGSe5qNf9TWH7iVR9sy8/tQLM2TmTdcP6cCtsIbV6DS6Wta1QW2mRYRLdzD1Yhu2zowYxJ+0Fa2MXyFqW+kOo4Gmy1Qq78fzBiI8H11PPjx747GU2v70KRoPh5ZgCdfDZyspSEM7g/u75EfVvCjDOSqfYOnWm4Ts59J5qIxS2DTYB2rMiTgn3h5xTBt/S4EuZCU+6BYMqgcNXZObXYFrPx3rP1NV/6r8peSvlD4oZr6o=\".encode(),\"base64\"),\"zip\"))"
[pid 84759] 11:49:59.602157 D mitogen: mitogen.docker.Stream(u'local.84764').connect(): child process stdin/stdout=98
[pid 84759] 11:50:02.797570 D mitogen: mitogen.docker.Stream(u'local.84764'): child process still alive, sending SIGTERM
[pid 84763] 11:50:02.799485 D mitogen: mitogen.core.Stream(u'unix_listener.84759').on_disconnect()
[pid 84759] 11:50:02.800214 D mitogen: mitogen.core.Stream(u'unix_client.84763').on_disconnect()
[pid 84763] 11:50:02.800483 D mitogen: Waker(Broker(0x10d48dd10) rfd=48, wfd=49).on_disconnect()
fatal: [host1]: UNREACHABLE! => {
    "changed": false,
    "msg": "EOF on stream; last 300 bytes received: u'r: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"exec: \\\\\"/Users/Pieter/.local/share/virtualenvs/it-4glgr_h9/bin/python2\\\\\": stat /Users/Pieter/.local/share/virtualenvs/it-4glgr_h9/bin/python2: no such file or directory\"\\n\\r\\n'",
    "unreachable": true
}

[dw]

Hi Pieter,

Thanks for the report. I should have time for this on Sunday, it should be straightforward to fix -- sorry for the delay and inconvenience!

David

[dw]

Sorry for the insane delay :(

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!