Mitogen fail to run playbook without “/usr/bin/python” on target host

[Ansen04]

• Python 3.7.0
• ansible==2.8.6
• mitogen==0.2.9
• target is Ubuntu 16.04 LTS on AWS

ansible-playbook 2.8.6
  config file = /Users/xxx/Documents/TestCode/ansible.cfg
  configured module search path = ['/Users/xxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/xxx/Documents/TestCode/venv/lib/python3.7/site-packages/ansible
  executable location = /Users/xxx/Documents/TestCodeCode/venv/bin/ansible-playbook
  python version = 3.7.0 (default, Aug 22 2018, 15:22:33) [Clang 9.1.0 (clang-902.0.39.2)]
Using /Users/xxx/Documents/TestCode/ansible.cfg as config file
host_list declined parsing /Users/xxx/Documents/TestCode/hosts as it did not pass it's verify_file() method
script declined parsing /Users/xxx/Documents/TestCode/hosts as it did not pass it's verify_file() method
auto declined parsing /Users/xxx/Documents/TestCode/hosts as it did not pass it's verify_file() method
Parsed /Users/xxx/Documents/TestCode/hosts inventory source with ini plugin

PLAYBOOK: playbook.yml ***************************************************************************************************************************************************************************************************
1 plays in playbook.yml

PLAY [test] **************************************************************************************************************************************************************************************************************
[top  90289] 14:05:42.544910 D ansible_mitogen.process: inherited open file limits: soft=12800 hard=(infinity)
[top  90289] 14:05:42.545244 D ansible_mitogen.process: could not raise soft open file limit from 12800 to 524288: current limit exceeds maximum limit
[top  90289] 14:05:42.545495 D ansible_mitogen.process: raised soft open file limit from 12800 to 10240
[mux  90293] 14:05:42.565143 D mitogen.service: Pool(e7f0, size=32, th='MainThread'): initialized
[mux  90293] 14:05:42.570270 D ansible_mitogen.process: Service pool configured: size=32
META: ran handlers

TASK [shell] *************************************************************************************************************************************************************************************************************
task path: /Users/xxx/Documents/TestCode/playbook.yml:7
[task 90294] 14:05:42.732986 D ansible_mitogen.process: will use multiplexer 0 (/var/folders/qp/hf3ywx_n19b_jkf7fl9nmr1c0000gn/T/mitogen_unix_atnmhqow.sock) to connect to "a6"
[task 90294] 14:05:42.734415 D mitogen.unix: client: connecting to /var/folders/qp/hf3ywx_n19b_jkf7fl9nmr1c0000gn/T/mitogen_unix_atnmhqow.sock
[mux  90293] 14:05:42.735729 D mitogen.unix: listener: accepted connection from PID 90294: unix_client.90294
[task 90294] 14:05:42.735577 D mitogen.unix: client: local ID is 1, remote is 0
[mux  90293] 14:05:42.742126 D mitogen.parent: creating connection to context 2 using mitogen.ssh
[mux  90293] 14:05:42.988448 D mitogen.parent: command line for Connection(None): ssh -o "LogLevel ERROR" -l ubuntu -i /Users/xxx/.ssh/awsalan2.pem -o "Compression yes" -o "ServerAliveInterval 30" -o "ServerAliveCountMax 10" -o "BatchMode yes" -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -o "GlobalKnownHostsFile /dev/null" -C -o ServerAliveInterval=60 -o ControlMaster=auto -o ControlPersist=300s -o UserKnownHostsFile=/dev/null 192.168.100.100 /usr/bin/python -c "'import codecs,os,sys;_=codecs.decode;exec(_(_(\”………………….\”.encode(),\"base64\"),\"zip\"))'"
[mux  90293] 14:05:42.998068 D mitogen.parent: child for Connection(None) started: pid:90295 stdin:119 stdout:119 stderr:121
[mux  90293] 14:05:43.757762 D mitogen.parent: ssh.192.168.100.100: (unrecognized): bash: /usr/bin/python: No such file or directory
[mux  90293] 14:05:43.759572 D mitogen: <Side of ssh.192.168.100.100 fd 121>: empty read, disconnecting
[mux  90293] 14:05:43.760733 D mitogen: SetupProtocol(ssh.192.168.100.100): disconnecting
[mux  90293] 14:05:43.761573 D mitogen: <Side of ssh.192.168.100.100 fd 119>: empty read, disconnecting
[mux  90293] 14:05:43.762319 D mitogen.parent: failing connection ssh.192.168.100.100 due to EofError('EOF on stream; last 100 lines received:\nbash: /usr/bin/python: No such file or directory')
[mux  90293] 14:05:43.763103 D mitogen.parent: PopenProcess ssh.192.168.100.100 pid 90295: exited with return code 127
[mux  90293] 14:05:43.763859 D mitogen: BootstrapProtocol(ssh.192.168.100.100): disconnecting
[mux  90293] 14:05:43.764943 D mitogen: BootstrapProtocol(ssh.192.168.100.100): disconnecting
[task 90294] 14:05:43.767791 D mitogen: MitogenProtocol(unix_listener.90293): disconnecting
[task 90294] 14:05:43.769088 D mitogen: Waker(fd=55/56): disconnecting
[task 90294] 14:05:43.770311 D mitogen: Router(Broker(6eb8)): stats: 0 module requests in 0 ms, 0 sent (0 ms minify time), 0 negative responses. Sent 0.0 kb total, 0.0 kb avg.
[mux  90293] 14:05:43.771226 D mitogen: <Side of unix_client.90294 fd 118>: empty read, disconnecting
[mux  90293] 14:05:43.772303 D mitogen: MitogenProtocol(unix_client.90294): disconnecting
fatal: [a6]: UNREACHABLE! => {
    "changed": false,
    "msg": "EOF on stream; last 100 lines received:\nbash: /usr/bin/python: No such file or directory",
    "unreachable": true
}

Running playbook without mitogen, Ansible sets ‘interpreter_python’ as "/usr/bin/python3" automatically

changed: [a6] => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": true,
    "cmd": "df",
    "delta": "0:00:00.002836",
    "end": "2019-12-17 06:29:16.122794",
    "invocation": {
        "module_args": {
            "_raw_params": "df",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2019-12-17 06:29:16.119958",
    "stderr": "",
    "stderr_lines": [],
    "stdout": “…….”,
    "stdout_lines": [
        ………
    ]
}

ansible-config is as below

ANSIBLE_PIPELINING(/Users/xxx/Documents/TestCode/ansible.cfg) = True
ANSIBLE_SSH_ARGS(/Users/xxx/Documents/TestCode/ansible.cfg) = -C -o ServerAliveInterval=60 -o ControlMaster=auto -o ControlPersist=300s -o UserKnownHostsFile=/dev/null
ANSIBLE_SSH_RETRIES(/Users/xxx/Documents/TestCode/ansible.cfg) = 2
DEFAULT_GATHER_TIMEOUT(/Users/xxx/Documents/TestCode/ansible.cfg) = 10
DEFAULT_LOG_PATH(/Users/xxx/Documents/TestCode/ansible.cfg) = /tmp/ansible.log
DEFAULT_TIMEOUT(/Users/xxx/Documents/TestCode/ansible.cfg) = 60
DEPRECATION_WARNINGS(/Users/xxx/Documents/TestCode/ansible.cfg) = False
HOST_KEY_CHECKING(/Users/xxx/Documents/TestCode/ansible.cfg) = False
PERSISTENT_CONNECT_TIMEOUT(/Users/xxx/Documents/TestCode/ansible.cfg) = 300

[s1113950]

Please give this another shot @Ansen04 ; #658 was just merged

[tom10271]

@s1113950 Do you think the fixing contains in 0.2.9? I have python3 installed on my VPS but facing a similar error

[s1113950]

The fix is not in 0.2.9 @tom10271 . There will be 1 or 2 Mitogen versions released very soon once #715 lands though, which will contain the fix. In the meantime, you can do what you noted here: #658 (comment) and get the fix :)