Closed
Description
Hi !
I'm trying to improve my deployement time on Kubespray and I have the following error on a gather_facts task with ansible_ssh_pipelining: true
Log
2018-10-25 12:27:38,447 p=23377 u=sprayuser | fatal: [mimispray-worker-1]: UNREACHABLE! => {
"changed": false,
"msg": "All items completed",
"results": [
{
"_ansible_ignore_errors": null,
"_ansible_item_label": "mimispray-etcd-1",
"_ansible_item_result": true,
"item": "mimispray-etcd-1",
"msg": "SSH authentication is incorrect",
"unreachable": true
},
...
2018-10-25 12:27:38,475 p=23377 u=sprayuser | [pid 23388] 12:27:38.475057 D mitogen.io: EpollPoller(0x7fddb6da5198): POLLIN: 761
2018-10-25 12:27:38,475 p=23377 u=sprayuser | [pid 23388] 12:27:38.475392 D mitogen.io: iter_read(761) -> b'Permission denied (publickey).\r\n'
2018-10-25 12:27:38,475 p=23377 u=sprayuser | [pid 23388] 12:27:38.475558 D mitogen: mitogen.ssh.Stream('local.24943'): received b'Permission denied (publickey).\r\n'
2018-10-25 12:27:38,475 mitogen mitogen.ssh.Stream('local.24943'): received b'Permission denied (publickey).\r\n'
2018-10-25 12:27:38,476 p=23377 u=sprayuser | [pid 23388] 12:27:38.476411 D mitogen: mitogen.ssh.Stream('local.24943'): PID 24943 exited with return code 255
2018-10-25 12:27:38,476 mitogen mitogen.ssh.Stream('local.24943'): PID 24943 exited with return code 255
2018-10-25 12:27:38,476 p=23377 u=sprayuser | [pid 23388] 12:27:38.476642 D mitogen.io: Latch(0x7fddb4041400, size=0, t='mitogen.service.Pool.7fddb6d09748.worker-10').put((<class 'mitogen.ssh.PasswordError'>, PasswordError('SSH authentication is incorrect',), <traceback object at 0x7fdd9c721a48>))
2018-10-25 12:27:38,476 p=23377 u=sprayuser | [pid 23388] 12:27:38.476829 D mitogen.io: Latch(0x7fddb4041400, size=1, t='mitogen.service.Pool.7fddb6d09748.worker-10').get(timeout=None, block=True)
2018-10-25 12:27:38,477 p=23377 u=sprayuser | [pid 23388] 12:27:38.476974 D mitogen.io: Latch(0x7fddb4041400, size=1, t='mitogen.service.Pool.7fddb6d09748.worker-10').get() -> (<class 'mitogen.ssh.PasswordError'>, PasswordError('SSH authentication is incorrect',), <traceback object at 0x7fdd9c721a48>)
2018-10-25 12:27:38,477 p=23377 u=sprayuser | [pid 23388] 12:27:38.477203 D mitogen.io: Waker(Broker(0x7fddb6d09080) rfd=18, wfd=20).defer() [fd=20]
2018-10-25 12:27:38,477 p=23377 u=sprayuser | [pid 23388] 12:27:38.477404 D mitogen.io: Latch(0x7fddb6d09898, size=0, t='mitogen.service.Pool.7fddb6d09748.worker-10').get(timeout=None, block=True)
2018-10-25 12:27:38,477 p=23377 u=sprayuser | [pid 23388] 12:27:38.477591 D mitogen.io: EpollPoller(0x7fddb6da5710).start_receive(44, None)
2018-10-25 12:27:38,477 p=23377 u=sprayuser | [pid 23388] 12:27:38.477733 D mitogen.io: EpollPoller(0x7fddb6da5710)._control(44)
2018-10-25 12:27:38,477 p=23377 u=sprayuser | [pid 23388] 12:27:38.477877 D mitogen.io: Latch(0x7fddb6d09898, size=0, t='mitogen.service.Pool.7fddb6d09748.worker-10')._get_sleep(timeout=None, block=True, rfd=44, wfd=45)
2018-10-25 12:27:38,478 p=23377 u=sprayuser | [pid 23388] 12:27:38.478056 D mitogen.io: EpollPoller(0x7fddb6d09240): POLLIN: 18
2018-10-25 12:27:38,478 p=23377 u=sprayuser | [pid 23388] 12:27:38.478245 D mitogen.io: Waker(Broker(0x7fddb6d09080) rfd=18, wfd=20).on_receive()
2018-10-25 12:27:38,478 p=23377 u=sprayuser | [pid 23388] 12:27:38.478408 D mitogen.io: Router(Broker(0x7fddb6d09080))._async_route(Message(28592, 0, 0, 1000, None, b'\x80\x02}q\x00(X\x07\x00\x00\x00contextq\x01NX\x11\x00\x00\x00init_child_resultq\x02NX\x0b\x00\x00'..124), None)
2018-10-25 12:27:38,478 p=23377 u=sprayuser | [pid 23388] 12:27:38.478567 D mitogen.io: mitogen.core.Stream('unix_client.24822')._send(Message(28592, 0, 0, 1000, None, b'\x80\x02}q\x00(X\x07\x00\x00\x00contextq\x01NX\x11\x00\x00\x00init_child_resultq\x02NX\x0b\x00\x00'..124))
2018-10-25 12:27:38,478 p=23377 u=sprayuser | [pid 23388] 12:27:38.478721 D mitogen.io: Broker(0x7fddb6d09080)._start_transmit(mitogen.core.Stream('unix_client.24822'))
2018-10-25 12:27:38,478 p=23377 u=sprayuser | [pid 23388] 12:27:38.478859 D mitogen.io: EpollPoller(0x7fddb6d09240).start_transmit(743, (<Side of mitogen.core.Stream('unix_client.24822') fd 743>, <bound method Stream.on_transmit of mitogen.core.Stream('unix_client.24822')>))
2018-10-25 12:27:38,479 p=23377 u=sprayuser | [pid 23388] 12:27:38.479004 D mitogen.io: EpollPoller(0x7fddb6d09240)._control(743)
2018-10-25 12:27:38,479 p=23377 u=sprayuser | [pid 23388] 12:27:38.479189 D mitogen.io: Broker(0x7fddb6d09080)._loop_once(None, EpollPoller(0x7fddb6d09240))
2018-10-25 12:27:38,479 p=23377 u=sprayuser | [pid 23388] 12:27:38.479338 D mitogen.io: EpollPoller(0x7fddb6d09240): POLLOUT: 743
2018-10-25 12:27:38,479 p=23377 u=sprayuser | [pid 23388] 12:27:38.479475 D mitogen.io: mitogen.core.Stream('unix_client.24822').on_transmit()
2018-10-25 12:27:38,479 p=23377 u=sprayuser | [pid 23388] 12:27:38.479628 D mitogen.io: mitogen.core.Stream('unix_client.24822').on_transmit() -> len 148
2018-10-25 12:27:38,479 p=23377 u=sprayuser | [pid 23388] 12:27:38.479767 D mitogen.io: Broker(0x7fddb6d09080)._stop_transmit(mitogen.core.Stream('unix_client.24822'))
2018-10-25 12:27:38,479 p=23377 u=sprayuser | [pid 23388] 12:27:38.479903 D mitogen.io: EpollPoller(0x7fddb6d09240).stop_transmit(743)
2018-10-25 12:27:38,480 p=23377 u=sprayuser | [pid 23388] 12:27:38.480260 D mitogen.io: EpollPoller(0x7fddb6d09240)._control(743)
2018-10-25 12:27:38,480 p=23377 u=sprayuser | [pid 23388] 12:27:38.480421 D mitogen.io: Broker(0x7fddb6d09080)._loop_once(None, EpollPoller(0x7fddb6d09240))
2018-10-25 12:27:38,480 p=23377 u=sprayuser | [pid 24822] 12:27:38.480253 D mitogen.io: EpollPoller(0x7fddafabef98): POLLIN: 48
2018-10-25 12:27:38,480 p=23377 u=sprayuser | [pid 24822] 12:27:38.480686 D mitogen.io: mitogen.core.Stream('unix_listener.23388').on_receive()
2018-10-25 12:27:38,481 p=23377 u=sprayuser | [pid 24822] 12:27:38.481009 D mitogen.io: Router(Broker(0x7fddaf06c390))._async_route(Message(28592, 0, 0, 1000, 0, b'\x80\x02}q\x00(X\x07\x00\x00\x00contextq\x01NX\x11\x00\x00\x00init_child_resultq\x02NX\x0b\x00\x00'..124), mitogen.core.Stream('unix_listener.23388'))
2018-10-25 12:27:38,481 p=23377 u=sprayuser | [pid 24822] 12:27:38.481333 D mitogen.io: Receiver(Router(Broker(0x7fddaf06c390)), 1000)._on_receive(Message(28592, 0, 0, 1000, 0, b'\x80\x02}q\x00(X\x07\x00\x00\x00contextq\x01NX\x11\x00\x00\x00init_child_resultq\x02NX\x0b\x00\x00'..124))
2018-10-25 12:27:38,481 p=23377 u=sprayuser | [pid 24822] 12:27:38.481598 D mitogen.io: Latch(0x7fddb434c748, size=0, t='mitogen-broker').put(Message(28592, 0, 0, 1000, 0, b'\x80\x02}q\x00(X\x07\x00\x00\x00contextq\x01NX\x11\x00\x00\x00init_child_resultq\x02NX\x0b\x00\x00'..124))
2018-10-25 12:27:38,482 p=23377 u=sprayuser | [pid 24822] 12:27:38.481868 D mitogen.io: Latch(0x7fddb434c748, size=1, t='mitogen-broker').put() -> waking wfd=36
2018-10-25 12:27:38,482 p=23377 u=sprayuser | [pid 24822] 12:27:38.482158 D mitogen.io: Broker(0x7fddaf06c390)._loop_once(None, EpollPoller(0x7fddafabef98))
2018-10-25 12:27:38,482 p=23377 u=sprayuser | [pid 24822] 12:27:38.482324 D mitogen.io: EpollPoller(0x7fddb4804c18): POLLIN: 28
2018-10-25 12:27:38,482 p=23377 u=sprayuser | [pid 24822] 12:27:38.482781 D mitogen.io: Latch(0x7fddb434c748, size=1, t='MainThread').get() wake -> Message(28592, 0, 0, 1000, 0, b'\x80\x02}q\x00(X\x07\x00\x00\x00contextq\x01NX\x11\x00\x00\x00init_child_resultq\x02NX\x0b\x00\x00'..124)
2018-10-25 12:27:38,483 p=23377 u=sprayuser | [pid 24822] 12:27:38.483022 D mitogen.io: Message(28592, 0, 0, 1000, 0, b'\x80\x02}q\x00(X\x07\x00\x00\x00contextq\x01NX\x11\x00\x00\x00init_child_resultq\x02NX\x0b\x00\x00'..124).unpickle()
ansible.cfg
[ssh_connection]
# Disabled for mitogen
#pipelining=True
#ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null
#control_path = ~/.ssh/ansible-%%r@%%h:%%p
[defaults]
log_path = /opt/kubespray/ansible.log
strategy_plugins = /opt/kubespray/mitogen-0.2.3/ansible_mitogen/plugins/strategy
strategy = mitogen_free
host_key_checking=False
private_key_file = /opt/kubespray/sprayuser.key
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp
stdout_callback = skippy
library = ./library
callback_whitelist = profile_tasks
roles_path = roles:$VIRTUAL_ENV/usr/local/share/kubespray/roles:$VIRTUAL_ENV/usr/local/share/ansible/roles:/usr/share/kubespray/roles
deprecation_warnings=False
inventory_ignore_extensions = ~, .orig, .bak, .ini, .cfg, .retry, .pyc, .pyo, .creds
[inventory]
ignore_patterns = artifacts, credentials
Task
- hosts: k8s-cluster:etcd:calico-rr
any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
vars:
ansible_ssh_pipelining: true
gather_facts: true
pre_tasks:
- name: gather facts from all instances
setup:
delegate_to: "{{item}}"
delegate_facts: True
with_items: "{{ groups['k8s-cluster'] + groups['etcd'] + groups['calico-rr']|default([]) }}"
Command used for launching the playbook
ansible-playbook -i inventory/mimispray/hosts.ini cluster.yml -b
I have also tried to setup strategy = mitogen_linear
but I had the same error.
Also, of course, I've got no errors when not using Mitogen.
Here is my setup
- CentOS 7.5
- Python 3.6 and Ansible 2.6.5 installed through PIP
- Last version of Kubespray
Thanks for your help :)