Error executing "delegate_to" task with mitogen

[knightsg]

I'm running into an issue with a task that uses delegate_to. The task definition is below:

- name: get ci git user key
  become_user: git
  command: cat ~/.ssh/id_rsa
  register: key_content
  delegate_to: "{{ item }}"
  run_once: yes
  loop: "{{ groups['ci_agent_servers']|flatten(levels=1) }}"

If I run ansible-playbook with mitogen the run fails at this task with the following error (-vvv is enabled):

[pid 12414] 00:24:15.026235 D mitogen: unix.connect(path='/tmp/mitogen_unix_J1sOVi')
[pid 4820] 00:24:15.027051 D mitogen: register(Context(27112, None), mitogen.core.Stream(u'unix_client.12414'))
[pid 12414] 00:24:15.027572 D mitogen: unix.connect(): local ID is 27112, remote is 0
[pid 12414] 00:24:15.028009 D mitogen: register(Context(0, None), mitogen.core.Stream(u'unix_listener.4820'))
[pid 12414] 00:24:15.194957 D mitogen: Context(0, None).call_service_async(u'ansible_mitogen.services.ContextService', u'get', {'stack': [{u'method': u'ssh', u'kwargs': {u'username': None, u'check_host_keys': u'ignore', u'ssh_debug_level': None, u'hostname': u'172.25.0.180', u'identities_only': False, u'identity_file': u'/var/jenkins/.ssh/BBG_foxwoods.pem', u'connect_timeout': 30, u'python_path': None, u'ssh_path': u'ssh', u'password': None, u'port': 9022, u'ssh_args': [u'-C', u'-o', u'ControlMaster=auto', u'-o', u'ControlPersist=30m']}}, {u'enable_lru': True, u'method': u'sudo', u'kwargs': {u'username': u'git', u'sudo_path': None, u'python_path': None, u'password': None, u'sudo_args': [u'-H', u'-S', u'-n'], u'connect_timeout': 30}}]})
[pid 12414] 00:24:15.195701 D mitogen: Context(0, None).send_async(Message(0, 27112, 27112, 110, 1000, "\x80\x02X'\x00\x00\x00ansible_mitogen.services.ContextServiceq\x01X\x03"..750))
[pid 4820] 00:24:15.196816 D mitogen: mitogen.ssh.Stream(u'default').connect()
[pid 4820] 00:24:15.212632 D mitogen: create_child() child 12417 fd 102, parent 4820, cmd: ssh -o "LogLevel ERROR"....<params obscured>
[pid 4820] 00:24:15.214220 D mitogen: mitogen.ssh.Stream(u'local.12417').connect(): stdin=102, stdout=99, diag=107
[pid 4820] 00:24:15.286173 D mitogen: mitogen.ssh.Stream(u'local.12417'): received 'Permission denied (publickey).\r\n'
[pid 4820] 00:24:15.286578 D mitogen: mitogen.ssh.Stream(u'local.12417').on_disconnect()
[pid 4820] 00:24:15.286853 D mitogen: mitogen.parent.DiagLogStream(fd=107, u'local.12417').on_disconnect()
[pid 4820] 00:24:15.287071 D mitogen: mitogen.ssh.Stream(u'local.12417'): child process still alive, sending SIGTERM
[pid 12414] 00:24:15.291207 D mitogen: Broker(0x7f4617c59550).shutdown()
[pid 12414] 00:24:15.291608 D mitogen: mitogen.core.Stream(u'unix_listener.4820').on_disconnect()
[pid 12414] 00:24:15.291857 D mitogen: Context(0, None).on_disconnect()
[pid 12414] 00:24:15.292054 D mitogen: Waker(Broker(0x7f4617c59550) rfd=21, wfd=22).on_shutdown()
[pid 12414] 00:24:15.292247 D mitogen: Waker(Broker(0x7f4617c59550) rfd=21, wfd=22).on_disconnect()
[pid 4820] 00:24:15.292579 D mitogen: mitogen.core.Stream(u'unix_client.12414').on_disconnect()
[pid 4820] 00:24:15.293011 D mitogen: Context(27112, None).on_disconnect()
failed: [172.25.0.141] (item=172.25.0.180) => {
    "item": "172.25.0.180", 
    "msg": "SSH authentication is incorrect", 
    "unreachable": true
}
fatal: [172.25.0.141]: UNREACHABLE! => {
    "changed": false, 
    "msg": "All items completed", 
    "results": [
        {
            "_ansible_ignore_errors": null, 
            "_ansible_item_label": "172.25.0.180", 
            "_ansible_item_result": true, 
            "item": "172.25.0.180", 
            "msg": "SSH authentication is incorrect", 
            "unreachable": true
        }
    ]
}

In the post-recap output I also see the following:

ERROR! [pid 4820] 00:24:15.335860 I mitogen.ctx.local.4920: stderr: Traceback (most recent call last):
ERROR! [pid 4820] 00:24:15.336783 I mitogen.ctx.local.4920: stderr:   File "<stdin>", line 3049, in <module>
ERROR! [pid 4820] 00:24:15.338066 I mitogen.ctx.local.4920: stderr:   File "<stdin>", line 3046, in main
ERROR! [pid 4820] 00:24:15.339740 I mitogen.ctx.local.4920: stderr:   File "<stdin>", line 2710, in shutdown
ERROR! [pid 4820] 00:24:15.340779 I mitogen.ctx.local.4920: stderr:   File "<stdin>", line 2123, in defer
ERROR! [pid 4820] 00:24:15.344071 I mitogen.ctx.local.4920: stderr: mitogen.core.Error: An attempt was made to enqueue a message with a Broker that has already begun shutting down. It is likely your program called Broker.shutdown() too early.
[pid 4820] 00:24:15.344474 D mitogen.ctx.local.4920: mitogen: <IoLogger stdout>.on_disconnect()
[pid 4820] 00:24:15.347158 D mitogen.ctx.local.4920: mitogen: <IoLogger stderr>.on_disconnect()
[pid 4820] 00:24:15.347588 D mitogen.ctx.fork.4923: mitogen: _on_shutdown_msg(Message(7, 4, 4, 106, 0, ''..0))
[pid 4820] 00:24:15.352972 D mitogen.ctx.ssh.172.25.0.138:9022: mitogen: Waker(Broker(0x7f3efc293650) rfd=3, wfd=4).on_disconnect()
[pid 4820] 00:24:15.353353 D mitogen.ctx.ssh.172.25.0.138:9022: mitogen: mitogen.fork.Stream(u'fork.28866') closing CALL_FUNCTION channel
[pid 4820] 00:24:15.353744 D mitogen.ctx.ssh.172.25.0.138:9022: mitogen: mitogen.parent.DiagLogStream(fd=16, u'sudo.root').on_disconnect()
[pid 4820] 00:24:15.356079 D mitogen.ctx.ssh.172.25.0.138:9022: mitogen: mitogen.sudo.Stream(u'sudo.root') closing CALL_FUNCTION channel
ERROR! [pid 4820] 00:24:15.356466 I mitogen.ctx.ssh.172.25.0.138:9022: stderr: Traceback (most recent call last):
ERROR! [pid 4820] 00:24:15.356874 I mitogen.ctx.ssh.172.25.0.138:9022: stderr:   File "<stdin>", line 3049, in <module>
ERROR! [pid 4820] 00:24:15.357322 I mitogen.ctx.ssh.172.25.0.138:9022: stderr:   File "<stdin>", line 3046, in main
ERROR! [pid 4820] 00:24:15.357743 I mitogen.ctx.ssh.172.25.0.138:9022: stderr:   File "<stdin>", line 2710, in shutdown
ERROR! [pid 4820] 00:24:15.358084 I mitogen.ctx.ssh.172.25.0.138:9022: stderr:   File "<stdin>", line 2123, in defer

Controller
Uname: Linux XXXXXXXXXX 3.13.0-164-generic #214-Ubuntu SMP Wed Dec 5 10:42:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

Ansible
ansible-playbook 2.6.11.post0 (stable-2.6 62b4376561) last updated 2019/01/11 21:36:38 (GMT +000)
config file = None
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /opt/ansible/ansible-2.6/lib/ansible
executable location = /opt/ansible/ansible-2.6/bin/ansible-playbook
python version = 2.7.6 (default, Nov 13 2018, 12:45:42) [GCC 4.8.4]

Notes: Running ansible-playbook with 'ANSIBLE_STRATEGY=mitogen_linear' set.

Host(s)
Uname: Linux XXXXXXXXXXXX 3.13.0-164-generic #214-Ubuntu SMP Wed Dec 5 10:42:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
Python 2.7.6

[LuckySB]

#404

[dw]

Thanks for linking it @LuckySB :) The overall ticket for this is #412. It's coming real soon. Sorry for the hassle!

[knightsg]

Awesome, that's great to hear!

[dw]

This will be on master in ~10 minutes

---

This should now be fixed on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

If you are still experiencing the problem on the current master branch, please do not hesitate to reopen this issue.

Thanks for reporting this!