Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error executing "delegate_to" task with mitogen #465

Closed
knightsg opened this issue Jan 12, 2019 · 4 comments
Closed

Error executing "delegate_to" task with mitogen #465

knightsg opened this issue Jan 12, 2019 · 4 comments

Comments

@knightsg
Copy link

@knightsg knightsg commented Jan 12, 2019

I'm running into an issue with a task that uses delegate_to. The task definition is below:

- name: get ci git user key
  become_user: git
  command: cat ~/.ssh/id_rsa
  register: key_content
  delegate_to: "{{ item }}"
  run_once: yes
  loop: "{{ groups['ci_agent_servers']|flatten(levels=1) }}"

If I run ansible-playbook with mitogen the run fails at this task with the following error (-vvv is enabled):

[pid 12414] 00:24:15.026235 D mitogen: unix.connect(path='/tmp/mitogen_unix_J1sOVi')
[pid 4820] 00:24:15.027051 D mitogen: register(Context(27112, None), mitogen.core.Stream(u'unix_client.12414'))
[pid 12414] 00:24:15.027572 D mitogen: unix.connect(): local ID is 27112, remote is 0
[pid 12414] 00:24:15.028009 D mitogen: register(Context(0, None), mitogen.core.Stream(u'unix_listener.4820'))
[pid 12414] 00:24:15.194957 D mitogen: Context(0, None).call_service_async(u'ansible_mitogen.services.ContextService', u'get', {'stack': [{u'method': u'ssh', u'kwargs': {u'username': None, u'check_host_keys': u'ignore', u'ssh_debug_level': None, u'hostname': u'172.25.0.180', u'identities_only': False, u'identity_file': u'/var/jenkins/.ssh/BBG_foxwoods.pem', u'connect_timeout': 30, u'python_path': None, u'ssh_path': u'ssh', u'password': None, u'port': 9022, u'ssh_args': [u'-C', u'-o', u'ControlMaster=auto', u'-o', u'ControlPersist=30m']}}, {u'enable_lru': True, u'method': u'sudo', u'kwargs': {u'username': u'git', u'sudo_path': None, u'python_path': None, u'password': None, u'sudo_args': [u'-H', u'-S', u'-n'], u'connect_timeout': 30}}]})
[pid 12414] 00:24:15.195701 D mitogen: Context(0, None).send_async(Message(0, 27112, 27112, 110, 1000, "\x80\x02X'\x00\x00\x00ansible_mitogen.services.ContextServiceq\x01X\x03"..750))
[pid 4820] 00:24:15.196816 D mitogen: mitogen.ssh.Stream(u'default').connect()
[pid 4820] 00:24:15.212632 D mitogen: create_child() child 12417 fd 102, parent 4820, cmd: ssh -o "LogLevel ERROR"....<params obscured>
[pid 4820] 00:24:15.214220 D mitogen: mitogen.ssh.Stream(u'local.12417').connect(): stdin=102, stdout=99, diag=107
[pid 4820] 00:24:15.286173 D mitogen: mitogen.ssh.Stream(u'local.12417'): received 'Permission denied (publickey).\r\n'
[pid 4820] 00:24:15.286578 D mitogen: mitogen.ssh.Stream(u'local.12417').on_disconnect()
[pid 4820] 00:24:15.286853 D mitogen: mitogen.parent.DiagLogStream(fd=107, u'local.12417').on_disconnect()
[pid 4820] 00:24:15.287071 D mitogen: mitogen.ssh.Stream(u'local.12417'): child process still alive, sending SIGTERM
[pid 12414] 00:24:15.291207 D mitogen: Broker(0x7f4617c59550).shutdown()
[pid 12414] 00:24:15.291608 D mitogen: mitogen.core.Stream(u'unix_listener.4820').on_disconnect()
[pid 12414] 00:24:15.291857 D mitogen: Context(0, None).on_disconnect()
[pid 12414] 00:24:15.292054 D mitogen: Waker(Broker(0x7f4617c59550) rfd=21, wfd=22).on_shutdown()
[pid 12414] 00:24:15.292247 D mitogen: Waker(Broker(0x7f4617c59550) rfd=21, wfd=22).on_disconnect()
[pid 4820] 00:24:15.292579 D mitogen: mitogen.core.Stream(u'unix_client.12414').on_disconnect()
[pid 4820] 00:24:15.293011 D mitogen: Context(27112, None).on_disconnect()
failed: [172.25.0.141] (item=172.25.0.180) => {
    "item": "172.25.0.180", 
    "msg": "SSH authentication is incorrect", 
    "unreachable": true
}
fatal: [172.25.0.141]: UNREACHABLE! => {
    "changed": false, 
    "msg": "All items completed", 
    "results": [
        {
            "_ansible_ignore_errors": null, 
            "_ansible_item_label": "172.25.0.180", 
            "_ansible_item_result": true, 
            "item": "172.25.0.180", 
            "msg": "SSH authentication is incorrect", 
            "unreachable": true
        }
    ]
}

In the post-recap output I also see the following:

ERROR! [pid 4820] 00:24:15.335860 I mitogen.ctx.local.4920: stderr: Traceback (most recent call last):
ERROR! [pid 4820] 00:24:15.336783 I mitogen.ctx.local.4920: stderr:   File "<stdin>", line 3049, in <module>
ERROR! [pid 4820] 00:24:15.338066 I mitogen.ctx.local.4920: stderr:   File "<stdin>", line 3046, in main
ERROR! [pid 4820] 00:24:15.339740 I mitogen.ctx.local.4920: stderr:   File "<stdin>", line 2710, in shutdown
ERROR! [pid 4820] 00:24:15.340779 I mitogen.ctx.local.4920: stderr:   File "<stdin>", line 2123, in defer
ERROR! [pid 4820] 00:24:15.344071 I mitogen.ctx.local.4920: stderr: mitogen.core.Error: An attempt was made to enqueue a message with a Broker that has already begun shutting down. It is likely your program called Broker.shutdown() too early.
[pid 4820] 00:24:15.344474 D mitogen.ctx.local.4920: mitogen: <IoLogger stdout>.on_disconnect()
[pid 4820] 00:24:15.347158 D mitogen.ctx.local.4920: mitogen: <IoLogger stderr>.on_disconnect()
[pid 4820] 00:24:15.347588 D mitogen.ctx.fork.4923: mitogen: _on_shutdown_msg(Message(7, 4, 4, 106, 0, ''..0))
[pid 4820] 00:24:15.352972 D mitogen.ctx.ssh.172.25.0.138:9022: mitogen: Waker(Broker(0x7f3efc293650) rfd=3, wfd=4).on_disconnect()
[pid 4820] 00:24:15.353353 D mitogen.ctx.ssh.172.25.0.138:9022: mitogen: mitogen.fork.Stream(u'fork.28866') closing CALL_FUNCTION channel
[pid 4820] 00:24:15.353744 D mitogen.ctx.ssh.172.25.0.138:9022: mitogen: mitogen.parent.DiagLogStream(fd=16, u'sudo.root').on_disconnect()
[pid 4820] 00:24:15.356079 D mitogen.ctx.ssh.172.25.0.138:9022: mitogen: mitogen.sudo.Stream(u'sudo.root') closing CALL_FUNCTION channel
ERROR! [pid 4820] 00:24:15.356466 I mitogen.ctx.ssh.172.25.0.138:9022: stderr: Traceback (most recent call last):
ERROR! [pid 4820] 00:24:15.356874 I mitogen.ctx.ssh.172.25.0.138:9022: stderr:   File "<stdin>", line 3049, in <module>
ERROR! [pid 4820] 00:24:15.357322 I mitogen.ctx.ssh.172.25.0.138:9022: stderr:   File "<stdin>", line 3046, in main
ERROR! [pid 4820] 00:24:15.357743 I mitogen.ctx.ssh.172.25.0.138:9022: stderr:   File "<stdin>", line 2710, in shutdown
ERROR! [pid 4820] 00:24:15.358084 I mitogen.ctx.ssh.172.25.0.138:9022: stderr:   File "<stdin>", line 2123, in defer

Controller
Uname: Linux XXXXXXXXXX 3.13.0-164-generic #214-Ubuntu SMP Wed Dec 5 10:42:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

Ansible
ansible-playbook 2.6.11.post0 (stable-2.6 62b4376561) last updated 2019/01/11 21:36:38 (GMT +000)
config file = None
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /opt/ansible/ansible-2.6/lib/ansible
executable location = /opt/ansible/ansible-2.6/bin/ansible-playbook
python version = 2.7.6 (default, Nov 13 2018, 12:45:42) [GCC 4.8.4]

Notes: Running ansible-playbook with 'ANSIBLE_STRATEGY=mitogen_linear' set.

Host(s)
Uname: Linux XXXXXXXXXXXX 3.13.0-164-generic #214-Ubuntu SMP Wed Dec 5 10:42:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
Python 2.7.6

@LuckySB
Copy link

@LuckySB LuckySB commented Jan 14, 2019

#404

@dw
Copy link
Member

@dw dw commented Jan 20, 2019

Thanks for linking it @LuckySB :) The overall ticket for this is #412. It's coming real soon. Sorry for the hassle!

@knightsg
Copy link
Author

@knightsg knightsg commented Jan 20, 2019

Awesome, that's great to hear!

@dw
Copy link
Member

@dw dw commented Jan 22, 2019

This will be on master in ~10 minutes


This should now be fixed on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

If you are still experiencing the problem on the current master branch, please do not hesitate to reopen this issue.

Thanks for reporting this!

@dw dw closed this Jan 22, 2019
dw added a commit that referenced this issue Jan 22, 2019
dw added a commit that referenced this issue Jan 22, 2019
* origin/dmw:
  docs: Changelog typo.
  docs: s/doffers/differs/
  issue #434: add to Changelog.
  issue #401: add to Changelog.
  issue #359: add to Changelog.
  issue #396: add to Changelog.
  docs: add thanks entry
  issue #436: add to Changelog.
  issue #465: add to Changelog.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants