Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When ansible_python_interpreter is set to search for Python in the env, mitogen fails. #291

ayaz opened this issue Jul 5, 2018 · 11 comments


Copy link

@ayaz ayaz commented Jul 5, 2018

Ansible: 2.5.5

If ansible_python_interpreter parameter is defined in the following manner in the playbook, the below listed error comes up:

ansible_python_interpreter: "/usr/bin/env python"

TASK [kubernetes/preinstall : Pre-upgrade | check if old credential dir exists] ************************************************************
task path: /Users/ayaz/Work/Cloudways/Cluster/kubespray/roles/kubernetes/preinstall/tasks/pre_upgrade.yml:2
Friday 29 June 2018  12:30:21 +0500 (0:00:00.099)       0:00:17.488 ***********
[pid 21418] 12:30:21.535102 D mitogen: unix.connect(path='/var/folders/_0/9b9z3xf527xg1pslfj_64q900000gn/T/mitogen_unix_jAzUOK')
[pid 21418] 12:30:21.536470 D mitogen: unix.connect(): local ID is 2019, remote is 0
[pid 21281] 12:30:21.538047 D mitogen: mitogen.parent.Stream('default').connect()
[pid 21418] 12:30:21.554305 D mitogen: mitogen.core.Stream('unix_listener.21281').on_disconnect()
[pid 21418] 12:30:21.554981 D mitogen: Waker(Broker(0x111175990) rfd=68, wfd=69).on_disconnect()
[pid 21281] 12:30:21.554914 D mitogen: mitogen.core.Stream('unix_client.21418').on_disconnect()
fatal: [av2-k8s-worker]: UNREACHABLE! => {
    "changed": false,
    "msg": "Child start failed: [Errno 2] No such file or directory. Command was: \"/usr/bin/env python\" \"-c\" \"import codecs,os,sys;_=codecs.decode;exec(_(_(\\\"eNptkNFLwzAQxp+3v6Jvl7CwJXVTKRSUPogPIhSxDzokbTMNdklJ29Xtr/faDtaKLyG/++6+77iYJaGtlqUuFaFzx9oR6Z2HsLPum9BgPsN/3pQ+4UxwTi8cszE5VMXAWWErReIxuDEkY2gRMLA6Ynwha0zde2HoQS5dqw140uS9qH5U1tQyLVQvr5rKrVJtVuWx/rIGcM/Zn7ZF2A8elKu0NW/B1baPVeagHTLcxw+vHLbhdGzoQSzIVGBTXADZ69p+KhPIozzddc+ysJksAl/4t4ICnaNT63StiGDw9PjyzDl/N1iPwg/SXTi3pTJ4V3Ap0KVTMidCXN9sKIOTLgeDc1PCoAU8+NkwGmvtVIP1euP7GLSI/ltBdCv8An8xnww=\\\".encode(),\\\"base64\\\"),\\\"zip\\\"))\"",
    "unreachable": true

Setting ansible_python_interpreter to a direct Python path resolves this issue.

Copy link

@dw dw commented Jul 9, 2018

Thanks for filing these! I should have a solution for you shortly after cutting the first stable release today. I need to figure out precisely what Ansible is doing -- that variable isn't a simple shell fragment, it also forms the '#!hashbang' line of scripts on the target, and so its syntax is quite limited.

dw added a commit that referenced this issue Jul 17, 2018
dw added a commit that referenced this issue Jul 17, 2018
Copy link

@dw dw commented Jul 18, 2018

This bug is a little more subtle than it may seem. Parsing of the hashbang varies across BSD and Linux, where Linux only parses up to one argument out of it, which may contain spaces. BSD on the other hand continues tokenization, making it possible to specify many more arguments.

While it does not impact your use case, a general fix requires an approach to the above.

My first thought was to throw an error noting the portability problems of including more than one argument, however that means Mitogen deviates from vanilla Ansible behaviour.

Just passing everything through introduces a few new issues -- an inexplicable infinite loop in bash when using the simplest 3-argument "/usr/bin/env" "ansible_bash_interpreter" setting.

Basically this needs more research to figure out the ideal behaviour. I can't check in code I know can trigger infinite loops :)

Copy link

@dw dw commented Jul 19, 2018

So this bug is actually pretty amazing. Ansible's documentation for its own variable is totally incorrect, the actual usage of "ansible_*_interpreter" means it is always a shell fragment.

Adding to the previous comment, none of the hashbang processing differences across operating systems actually matters, because even though Ansible updates the hashbang, in this case with a value that doesn't even work on Linux (it can trigger an infinite exec loop in some cases), it always invokes the module as "$ansible_foo_interpreter /path/to/module" using a shell fragment, even by explicitly stripping off the hashbang it created upfront (called via ActionBase._execute_module()).

That means you can stuff arbitrary shell into ansible_*_interpreter, and so that must be supported.

dw added a commit that referenced this issue Jul 22, 2018
dw added a commit that referenced this issue Jul 24, 2018
When running any kind of script, rewrite the hashbang like Ansible does,
but subsequently ignore it and explicitly use a fragment of shell from
the ansible_*_interpreter variable to call the interpreter, just like
Ansible does.

This fixes hashbangs containing '/usr/bin/env A=1 bash' on Linux, where
putting that into a hashbang line results in an infinite loop.
dw added a commit that referenced this issue Jul 24, 2018
This ensures failed task output matches vanilla Ansible exactly (as it
did before starting #291).
dw added a commit that referenced this issue Jul 25, 2018
@dw dw closed this in e39c602 Jul 25, 2018
Copy link

@danquack danquack commented Jul 27, 2018

@dw after upgrading to 2.2 the Ansible interpreter is giving me this error.

fatal: []: UNREACHABLE! => {"changed": false, "msg": "EOF on stream; last 300 bytes received: u\"Warning: Permanently added '' (ECDSA) to the list of known hosts.\\r\\nzsh:1: no such file or directory: /usr/bin/env python\\n\"", "unreachable": true}
When I actually go to the host that environment resolves.

Heres the vvv output

task path: /tmp/ping.yml:8
[pid 19822] 09:46:16.814934 D mitogen: unix.connect(path='/tmp/mitogen_unix_aCZPr0')
[pid 19822] 09:46:16.820545 D mitogen: unix.connect(): local ID is 1, remote is 0
[pid 19803] 09:46:16.826078 D mitogen: mitogen.ssh.Stream(u'default').connect()
[pid 19803] 09:46:16.959634 D mitogen: hybrid_tty_create_child() pid=19826 stdio=62, tty=16, cmd: "ssh" "-o" "Compression yes" "-o" "ServerAliveInterval 15" "-o" "ServerAliveCountMax 3" "-o" "StrictHostKeyChecking no" "-o" "UserKnownHostsFile /dev/null" "-o" "GlobalKnownHostsFile /dev/null" "-C" "-o" "ControlMaster=auto" "-o" "ControlPersist=60s" "" "'/usr/bin/env python'" "-c" "'import codecs,os,sys;_=codecs.decode;exec(_(_(\"eNqFkTFPwzAQhefmV2Q7W7Vap+0AkSyBOiAGhBQhOkCFktgBq6ltnKRp+fVc00pNysDk+/Te3TudE7YStpo47RShgWdtj3QRIhTWbwiNgxHWsnEzwlnEOb1wwvrkUY1OnJe2UiTpg+/Dqg8tAgZWB4wv0xpTt6EQIcjUt9pAmBrZiWqv8qZOs1J18rSp/DTTZuoO9Zc1gHuOrmxj0TXulK+0NW/xfN3FKrPTHhnuk4dXDmsxbDt5EEsyFNgQx0C2urafysTyu0nzjTJ357du9nF0e8PnFGiA01qva0UiBk+PL8+c83cDuEVuJR6bBkvxQY7nltYpg0cGnwGdeJVKEs3miwVl8KMdTiqcuPhWDNoMjj9QuHPAsqtPV71yt/+5/24ZDbb8BQ51sCk=\".encode(),\"base64\"),\"zip\"))'"
[pid 19803] 09:46:16.961421 D mitogen: mitogen.ssh.Stream(u'local.19826').connect(): child process stdin/stdout=62
[pid 19803] 09:46:17.008152 D mitogen: mitogen.ssh.Stream(u'local.19826'): received "Warning: Permanently added '' (ECDSA) to the list of known hosts.\r\n"
[pid 19803] 09:46:17.631863 D mitogen: mitogen.ssh.Stream(u'local.19826'): received 'zsh:1: no such file or directory: /usr/bin/env python\n'
[pid 19803] 09:46:17.633008 D mitogen: mitogen.ssh.Stream(u'local.19826'): child process exit status was 32512
[pid 19822] 09:46:17.634223 D mitogen: mitogen.core.Stream(u'unix_listener.19803').on_disconnect()
[pid 19822] 09:46:17.634513 D mitogen: Waker(Broker(0x7fd553b5c850) rfd=13, wfd=14).on_disconnect()
[pid 19803] 09:46:17.634543 D mitogen: mitogen.core.Stream(u'unix_client.19822').on_disconnect()
fatal: []: UNREACHABLE! => {
    "changed": false, 
    "msg": "EOF on stream; last 300 bytes received: u\"Warning: Permanently added '' (ECDSA) to the list of known hosts.\\r\\nzsh:1: no such file or directory: /usr/bin/env python\\n\"", 
    "unreachable": true
Copy link

@dw dw commented Jul 27, 2018

Oh man, you gotta be kidding me. This is the bug that keeps on giving :( Clearly the tests I added don't actually cover your case! Will look at this promptly, and hopefully another release for Mondayish

@dw dw reopened this Jul 27, 2018
Copy link

@dw dw commented Jul 28, 2018

Hi @danquack

Can you please share the relevant config when/where you're setting ansible_python_interpreter.

Copy link

@danquack danquack commented Jul 28, 2018

Running RHEL7->RHEL7, below is the playbook. Do you need any other info?

- hosts: all
  become_user: root
  gather_facts: no
    ansible_python_interpreter: "/usr/bin/env python"
  - ping:
Copy link

@dw dw commented Jul 28, 2018

I'm still in minor shock this slipped through. :)

Running this locally, it seems to work fine! I hate to ask, but is there no chance your ansible.cfg is pointing at an older ansible_mitogen version? I tried under Python 2.6/2.7/3.4

Don't waste too much time looking -- it might be a thinko on my end! But from first glance it seems the only two places I extract that variable, it is passed through the parser correctly.

edit: ooh, I had one thought. If you ever 'pip install'd an older ansible_mitogen, it may end up in sys.modules search list ahead of whatever is in ansible.cfg. That can be worked around with code, so if that's the case, I still consider this a bug!

Copy link

@danquack danquack commented Jul 28, 2018


library         = /usr/share/ansible:/opt/ansible/library
forks          = 200
gathering = implicit
gather_subset = all,!ohai,!hardware,!virtual,!network
gather_timeout = 20
roles_path     = /etc/ansible/roles:/opt/ansible/roles
host_key_checking = False
callback_whitelist = foreman
timeout = 30
remote_user = linux_admin
log_path = /var/log/ansible.log
private_key_file = /home/linux_admin/.ssh/id_rsa.ansible
strategy_plugins    = /opt/mitogen/ansible_mitogen/plugins/strategy
strategy                  = mitogen_free
bin_ansible_callbacks = True
retry_files_enabled = False
pipelining = True
ansible --version:
ansible 2.5.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/usr/share/ansible', u'/opt/ansible/library']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

dw added a commit that referenced this issue Jul 28, 2018
Given an extracted download of mitogen-2.2.tar.gz, with strategy_plugins
pointing into it, if an old version of the package was pip-installed,
then the old pip-installed package would be imported and override
whatever came from the tarball.

Instead, modify sys.path before attempting any import. This still isn't
perfect, but it's better.
Copy link

@dw dw commented Jul 28, 2018

Hi Dan :) Can you please try and let me know if the problem persists? I was able to reproduce your issue when a pip-installed old version of the package exists, and the commit above works around that.

Copy link

@danquack danquack commented Jul 29, 2018

@dw good to close

@dw dw closed this Jul 29, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants