Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

async_status does not fail when it over async timout #74184

Closed
larsche opened this issue Apr 7, 2021 · 6 comments
Closed

async_status does not fail when it over async timout #74184

larsche opened this issue Apr 7, 2021 · 6 comments
Assignees
Labels
affects_2.9 This issue/PR affects Ansible v2.9 bug This issue/PR relates to a bug. easyfix This issue is considered easy to fix by aspiring contributors. has_pr This issue has an associated PR. module This issue/PR relates to a module. needs_info This issue requires further information. Please answer any outstanding questions. P3 Priority 3 - Approved, No Time Limitation python3 support:core This issue/PR relates to code supported by the Ansible Engineering Team. verified This issue has been verified/reproduced by maintainer

Comments

@larsche
Copy link

larsche commented Apr 7, 2021

Summary

I'm testing what happens when the task finishes after the async setting. When using poll > 0 then the async works as it should and exits. However, using poll = 0 together with the async_status it does not error out like when using poll > 0 but continues until the retries are done

Issue Type

Bug Report

Component Name

ansible.builtin.async_status

Ansible Version

$ ansible --version
ansible 2.9.19

Configuration

$ ansible-config dump --only-changed
ALLOW_WORLD_READABLE_TMPFILES(/home/larsch/IdeaProjects/ansible-postgres-presentation/ansible.cfg) = True
DEFAULT_CALLBACK_WHITELIST(/home/larsch/IdeaProjects/ansible-postgres-presentation/ansible.cfg) = [u'profile_tasks']
DEFAULT_HOST_LIST(/home/larsch/IdeaProjects/ansible-postgres-presentation/ansible.cfg) = [u'/home/larsch/IdeaProjects/ansible-postgres-presentation/inventory']
DEFAULT_VAULT_PASSWORD_FILE(/home/larsch/IdeaProjects/ansible-postgres-presentation/ansible.cfg) = /home/larsch/.vault_pass.txt
INTERPRETER_PYTHON(/home/larsch/IdeaProjects/ansible-postgres-presentation/ansible.cfg) = /usr/bin/python3

OS / Environment

Ubuntu 20.04

Steps to Reproduce

    shell: sleep 30
    async: 20
    poll: 0
    register: sleep_command

  - name: check status
    async_status:
      jid: "{{ sleep_command.ansible_job_id }}"
    register: job_result
    until: job_result.finished
    retries: 100
    delay: 5

Expected Results

fatal: []: FAILED! => {"changed": false, "msg": "async task did not complete within the requested time - 20s"}

Actual Results

ansible-playbook 2.9.19
  config file = /home/larsch/IdeaProjects/ansible-postgres-presentation/ansible.cfg
  configured module search path = [u'/home/larsch/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.17 (default, Feb 27 2021, 15:10:58) [GCC 7.5.0]
Using /home/larsch/IdeaProjects/ansible-postgres-presentation/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /home/larsch/IdeaProjects/ansible-postgres-presentation/inventory/lxd_invetory.ini as it did not pass its verify_file() method
script declined parsing /home/larsch/IdeaProjects/ansible-postgres-presentation/inventory/lxd_invetory.ini as it did not pass its verify_file() method
auto declined parsing /home/larsch/IdeaProjects/ansible-postgres-presentation/inventory/lxd_invetory.ini as it did not pass its verify_file() method
yaml declined parsing /home/larsch/IdeaProjects/ansible-postgres-presentation/inventory/lxd_invetory.ini as it did not pass its verify_file() method
Parsed /home/larsch/IdeaProjects/ansible-postgres-presentation/inventory/lxd_invetory.ini inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/default.pyc
Skipping callback 'actionable', as we already have a stdout callback.
Skipping callback 'counter_enabled', as we already have a stdout callback.
Skipping callback 'debug', as we already have a stdout callback.
Skipping callback 'dense', as we already have a stdout callback.
Skipping callback 'dense', as we already have a stdout callback.
Skipping callback 'full_skip', as we already have a stdout callback.
Skipping callback 'json', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'null', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
Loading callback plugin profile_tasks of type aggregate, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/profile_tasks.pyc
Skipping callback 'selective', as we already have a stdout callback.
Skipping callback 'skippy', as we already have a stdout callback.
Skipping callback 'stderr', as we already have a stdout callback.
Skipping callback 'unixy', as we already have a stdout callback.
Skipping callback 'yaml', as we already have a stdout callback.

PLAYBOOK: test_async.yml ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Positional arguments: test_async.yml
subset: bench-loc
become_method: sudo
inventory: (u'/home/larsch/IdeaProjects/ansible-postgres-presentation/inventory/lxd_invetory.ini',)
forks: 5
tags: (u'all',)
verbosity: 4
connection: smart
timeout: 10
1 plays in test_async.yml

PLAY [all] **********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
META: ran handlers
Trying secret FileVaultSecret(filename='/home/larsch/.vault_pass.txt') for vault_id=default

TASK [sleep async] **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
task path: /home/larsch/IdeaProjects/ansible-postgres-presentation/test_async.yml:10
Wednesday 07 April 2021  21:10:47 +0200 (0:00:00.056)       0:00:00.056 ******* 
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754 `" && echo ansible-tmp-1617822647.71-25984-252100598924754="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822647.71-25984-252100598924754=/home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/commands/command.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpM31Z_Q TO /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/AnsiballZ_command.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpM31Z_Q /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/AnsiballZ_command.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpM31Z_Q\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/AnsiballZ_command.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:20210\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 20210 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpxq9yXW TO /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/async_wrapper.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpxq9yXW /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/async_wrapper.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpxq9yXW\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/async_wrapper.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:19679\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 19679 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/ /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/AnsiballZ_command.py /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/async_wrapper.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'ANSIBLE_ASYNC_DIR='"'"'"'"'"'"'"'"'~/.ansible_async'"'"'"'"'"'"'"'"' /usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/async_wrapper.py 973870053482 20 /home/larsch/.ansible/tmp/ansible-tmp-1617822647.71-25984-252100598924754/AnsiballZ_command.py _ && sleep 0'"'"''
<bench-loc> (0, '{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "results_file": "/home/larsch/.ansible_async/973870053482.5418", "_ansible_suppress_tmpdir_delete": true}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
changed: [bench-loc] => {
    "ansible_job_id": "973870053482.5418", 
    "changed": true, 
    "finished": 0, 
    "results_file": "/home/larsch/.ansible_async/973870053482.5418", 
    "started": 1
}

TASK [check status] *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
task path: /home/larsch/IdeaProjects/ansible-postgres-presentation/test_async.yml:16
Wednesday 07 April 2021  21:10:48 +0200 (0:00:00.678)       0:00:00.734 ******* 
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822648.4-26017-128291767304306 `" && echo ansible-tmp-1617822648.4-26017-128291767304306="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822648.4-26017-128291767304306 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822648.4-26017-128291767304306=/home/larsch/.ansible/tmp/ansible-tmp-1617822648.4-26017-128291767304306\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpmCvpwJ TO /home/larsch/.ansible/tmp/ansible-tmp-1617822648.4-26017-128291767304306/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpmCvpwJ /home/larsch/.ansible/tmp/ansible-tmp-1617822648.4-26017-128291767304306/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpmCvpwJ\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822648.4-26017-128291767304306/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17198\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17198 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822648.4-26017-128291767304306/ /home/larsch/.ansible/tmp/ansible-tmp-1617822648.4-26017-128291767304306/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822648.4-26017-128291767304306/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822648.4-26017-128291767304306/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (100 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 1, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822653.94-26017-275229052507692 `" && echo ansible-tmp-1617822653.94-26017-275229052507692="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822653.94-26017-275229052507692 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822653.94-26017-275229052507692=/home/larsch/.ansible/tmp/ansible-tmp-1617822653.94-26017-275229052507692\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpdsSktV TO /home/larsch/.ansible/tmp/ansible-tmp-1617822653.94-26017-275229052507692/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpdsSktV /home/larsch/.ansible/tmp/ansible-tmp-1617822653.94-26017-275229052507692/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpdsSktV\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822653.94-26017-275229052507692/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17199\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17199 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822653.94-26017-275229052507692/ /home/larsch/.ansible/tmp/ansible-tmp-1617822653.94-26017-275229052507692/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822653.94-26017-275229052507692/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822653.94-26017-275229052507692/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (99 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 2, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822659.21-26017-41428972496810 `" && echo ansible-tmp-1617822659.21-26017-41428972496810="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822659.21-26017-41428972496810 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822659.21-26017-41428972496810=/home/larsch/.ansible/tmp/ansible-tmp-1617822659.21-26017-41428972496810\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpCQAtL1 TO /home/larsch/.ansible/tmp/ansible-tmp-1617822659.21-26017-41428972496810/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpCQAtL1 /home/larsch/.ansible/tmp/ansible-tmp-1617822659.21-26017-41428972496810/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpCQAtL1\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822659.21-26017-41428972496810/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17198\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17198 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822659.21-26017-41428972496810/ /home/larsch/.ansible/tmp/ansible-tmp-1617822659.21-26017-41428972496810/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822659.21-26017-41428972496810/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822659.21-26017-41428972496810/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (98 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 3, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822664.53-26017-2646552978932 `" && echo ansible-tmp-1617822664.53-26017-2646552978932="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822664.53-26017-2646552978932 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822664.53-26017-2646552978932=/home/larsch/.ansible/tmp/ansible-tmp-1617822664.53-26017-2646552978932\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpSXHYHm TO /home/larsch/.ansible/tmp/ansible-tmp-1617822664.53-26017-2646552978932/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpSXHYHm /home/larsch/.ansible/tmp/ansible-tmp-1617822664.53-26017-2646552978932/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpSXHYHm\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822664.53-26017-2646552978932/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17196\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17196 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822664.53-26017-2646552978932/ /home/larsch/.ansible/tmp/ansible-tmp-1617822664.53-26017-2646552978932/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822664.53-26017-2646552978932/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822664.53-26017-2646552978932/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (97 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 4, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822669.81-26017-270775847038048 `" && echo ansible-tmp-1617822669.81-26017-270775847038048="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822669.81-26017-270775847038048 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822669.81-26017-270775847038048=/home/larsch/.ansible/tmp/ansible-tmp-1617822669.81-26017-270775847038048\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpnZWexS TO /home/larsch/.ansible/tmp/ansible-tmp-1617822669.81-26017-270775847038048/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpnZWexS /home/larsch/.ansible/tmp/ansible-tmp-1617822669.81-26017-270775847038048/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpnZWexS\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822669.81-26017-270775847038048/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17198\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17198 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822669.81-26017-270775847038048/ /home/larsch/.ansible/tmp/ansible-tmp-1617822669.81-26017-270775847038048/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822669.81-26017-270775847038048/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822669.81-26017-270775847038048/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (96 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 5, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822675.16-26017-176395669497556 `" && echo ansible-tmp-1617822675.16-26017-176395669497556="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822675.16-26017-176395669497556 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822675.16-26017-176395669497556=/home/larsch/.ansible/tmp/ansible-tmp-1617822675.16-26017-176395669497556\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmp_W2dYq TO /home/larsch/.ansible/tmp/ansible-tmp-1617822675.16-26017-176395669497556/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmp_W2dYq /home/larsch/.ansible/tmp/ansible-tmp-1617822675.16-26017-176395669497556/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmp_W2dYq\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822675.16-26017-176395669497556/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17199\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17199 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822675.16-26017-176395669497556/ /home/larsch/.ansible/tmp/ansible-tmp-1617822675.16-26017-176395669497556/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822675.16-26017-176395669497556/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822675.16-26017-176395669497556/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (95 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 6, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822680.46-26017-228124067314134 `" && echo ansible-tmp-1617822680.46-26017-228124067314134="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822680.46-26017-228124067314134 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822680.46-26017-228124067314134=/home/larsch/.ansible/tmp/ansible-tmp-1617822680.46-26017-228124067314134\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpKPEhWl TO /home/larsch/.ansible/tmp/ansible-tmp-1617822680.46-26017-228124067314134/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpKPEhWl /home/larsch/.ansible/tmp/ansible-tmp-1617822680.46-26017-228124067314134/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpKPEhWl\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822680.46-26017-228124067314134/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17199\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17199 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822680.46-26017-228124067314134/ /home/larsch/.ansible/tmp/ansible-tmp-1617822680.46-26017-228124067314134/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822680.46-26017-228124067314134/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822680.46-26017-228124067314134/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (94 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 7, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822685.76-26017-270468382793933 `" && echo ansible-tmp-1617822685.76-26017-270468382793933="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822685.76-26017-270468382793933 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822685.76-26017-270468382793933=/home/larsch/.ansible/tmp/ansible-tmp-1617822685.76-26017-270468382793933\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpySQhJ6 TO /home/larsch/.ansible/tmp/ansible-tmp-1617822685.76-26017-270468382793933/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpySQhJ6 /home/larsch/.ansible/tmp/ansible-tmp-1617822685.76-26017-270468382793933/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpySQhJ6\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822685.76-26017-270468382793933/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17199\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17199 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822685.76-26017-270468382793933/ /home/larsch/.ansible/tmp/ansible-tmp-1617822685.76-26017-270468382793933/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822685.76-26017-270468382793933/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822685.76-26017-270468382793933/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (93 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 8, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822691.05-26017-220671853657968 `" && echo ansible-tmp-1617822691.05-26017-220671853657968="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822691.05-26017-220671853657968 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822691.05-26017-220671853657968=/home/larsch/.ansible/tmp/ansible-tmp-1617822691.05-26017-220671853657968\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpmMKuJD TO /home/larsch/.ansible/tmp/ansible-tmp-1617822691.05-26017-220671853657968/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpmMKuJD /home/larsch/.ansible/tmp/ansible-tmp-1617822691.05-26017-220671853657968/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpmMKuJD\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822691.05-26017-220671853657968/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17199\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17199 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822691.05-26017-220671853657968/ /home/larsch/.ansible/tmp/ansible-tmp-1617822691.05-26017-220671853657968/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822691.05-26017-220671853657968/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822691.05-26017-220671853657968/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (92 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 9, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822696.35-26017-163678271413799 `" && echo ansible-tmp-1617822696.35-26017-163678271413799="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822696.35-26017-163678271413799 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822696.35-26017-163678271413799=/home/larsch/.ansible/tmp/ansible-tmp-1617822696.35-26017-163678271413799\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpXA0Zwl TO /home/larsch/.ansible/tmp/ansible-tmp-1617822696.35-26017-163678271413799/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpXA0Zwl /home/larsch/.ansible/tmp/ansible-tmp-1617822696.35-26017-163678271413799/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpXA0Zwl\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822696.35-26017-163678271413799/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17199\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17199 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822696.35-26017-163678271413799/ /home/larsch/.ansible/tmp/ansible-tmp-1617822696.35-26017-163678271413799/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822696.35-26017-163678271413799/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822696.35-26017-163678271413799/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (91 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 10, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<bench-loc> (0, '/home/larsch\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/larsch/.ansible/tmp `"&& mkdir "` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822701.65-26017-45102674628212 `" && echo ansible-tmp-1617822701.65-26017-45102674628212="` echo /home/larsch/.ansible/tmp/ansible-tmp-1617822701.65-26017-45102674628212 `" ) && sleep 0'"'"''
<bench-loc> (0, 'ansible-tmp-1617822701.65-26017-45102674628212=/home/larsch/.ansible/tmp/ansible-tmp-1617822701.65-26017-45102674628212\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/utilities/logic/async_status.py
<bench-loc> PUT /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpKWRQY4 TO /home/larsch/.ansible/tmp/ansible-tmp-1617822701.65-26017-45102674628212/AnsiballZ_async_status.py
<bench-loc> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 '[bench-loc]'
<bench-loc> (0, 'sftp> put /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpKWRQY4 /home/larsch/.ansible/tmp/ansible-tmp-1617822701.65-26017-45102674628212/AnsiballZ_async_status.py\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Unrecognised server extension "lsetstat@openssh.com"\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/larsch size 0\r\ndebug3: Looking up /home/larsch/.ansible/tmp/ansible-local-25971Nb2kvu/tmpKWRQY4\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/larsch/.ansible/tmp/ansible-tmp-1617822701.65-26017-45102674628212/AnsiballZ_async_status.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:17198\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 17198 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'chmod u+x /home/larsch/.ansible/tmp/ansible-tmp-1617822701.65-26017-45102674628212/ /home/larsch/.ansible/tmp/ansible-tmp-1617822701.65-26017-45102674628212/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 -tt bench-loc '/bin/sh -c '"'"'/usr/bin/python3 /home/larsch/.ansible/tmp/ansible-tmp-1617822701.65-26017-45102674628212/AnsiballZ_async_status.py && sleep 0'"'"''
<bench-loc> (0, '\r\n{"started": 1, "finished": 0, "ansible_job_id": "973870053482.5418", "invocation": {"module_args": {"jid": "973870053482.5418", "_async_dir": "/home/larsch/.ansible_async", "mode": "status"}}}\r\n', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to bench-loc closed.\r\n')
<bench-loc> ESTABLISH SSH CONNECTION FOR USER: None
<bench-loc> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/larsch/.ansible/cp/8b4698beb6 bench-loc '/bin/sh -c '"'"'rm -f -r /home/larsch/.ansible/tmp/ansible-tmp-1617822701.65-26017-45102674628212/ > /dev/null 2>&1 && sleep 0'"'"''
<bench-loc> (0, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017\r\ndebug1: Reading configuration data /home/larsch/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 24439\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
FAILED - RETRYING: check status (90 retries left).Result was: {
    "ansible_job_id": "973870053482.5418", 
    "attempts": 11, 
    "changed": false, 
    "finished": 0, 
    "invocation": {
        "module_args": {
            "_async_dir": "/home/larsch/.ansible_async", 
            "jid": "973870053482.5418", 
            "mode": "status"
        }
    }, 
    "retries": 101, 
    "started": 1
}

Code of Conduct

I agree to follow the Ansible Code of Conduct

@ansibot
Copy link
Contributor

ansibot commented Apr 7, 2021

Files identified in the description:

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

@ansibot ansibot added affects_2.9 This issue/PR affects Ansible v2.9 bug This issue/PR relates to a bug. module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. python3 support:core This issue/PR relates to code supported by the Ansible Engineering Team. labels Apr 7, 2021
@samdoran samdoran added needs_verified This issue needs to be verified/reproduced by maintainer P3 Priority 3 - Approved, No Time Limitation and removed needs_triage Needs a first human triage before being processed. labels Apr 8, 2021
@nitzmahone
Copy link
Member

Looks like the result file write on timeout got lost at some point in async_wrapper.py- the Windows one properly records this, but the Python one currently does not...

@sivel sivel added verified This issue has been verified/reproduced by maintainer and removed needs_verified This issue needs to be verified/reproduced by maintainer labels Apr 8, 2021
@sivel
Copy link
Member

sivel commented Apr 8, 2021

I've verified this. When we kill the child process, we don't write a result out, so the file only contains the last result of the child. I just made a bunch of other changes here, but anyone can feel to pick this up and get it finished:

Click to expand diff...

diff --git a/lib/ansible/modules/async_wrapper.py b/lib/ansible/modules/async_wrapper.py
index 7ba8271ef6..6694cdf3b5 100644
--- a/lib/ansible/modules/async_wrapper.py
+++ b/lib/ansible/modules/async_wrapper.py
@@ -129,11 +129,9 @@ def _make_temp_dir(path):
 def _run_module(wrapped_cmd, jid, job_path):
 
     tmp_job_path = job_path + ".tmp"
-    jobfile = open(tmp_job_path, "w")
-    jobfile.write(json.dumps({"started": 1, "finished": 0, "ansible_job_id": jid}))
-    jobfile.close()
+    with open(tmp_job_path, "w") as jobfile:
+        json.dump({"started": 1, "finished": 0, "ansible_job_id": jid}, jobfile)
     os.rename(tmp_job_path, job_path)
-    jobfile = open(tmp_job_path, "w")
     result = {}
 
     # signal grandchild process started and isolated from being terminated
@@ -173,7 +171,8 @@ def _run_module(wrapped_cmd, jid, job_path):
 
         if stderr:
             result['stderr'] = stderr
-        jobfile.write(json.dumps(result))
+        with open(tmp_job_path, "w") as jobfile:
+            json.dump(result, jobfile)
 
     except (OSError, IOError):
         e = sys.exc_info()[1]
@@ -182,10 +181,11 @@ def _run_module(wrapped_cmd, jid, job_path):
             "cmd": wrapped_cmd,
             "msg": to_text(e),
             "outdata": outdata,  # temporary notice only
-            "stderr": stderr
+            "stderr": stderr,
+            "ansible_job_id": jid,
         }
-        result['ansible_job_id'] = jid
-        jobfile.write(json.dumps(result))
+        with open(tmp_job_path, "w") as jobfile:
+            json.dump(result, jobfile)
 
     except (ValueError, Exception):
         result = {
@@ -193,12 +193,12 @@ def _run_module(wrapped_cmd, jid, job_path):
             "cmd": wrapped_cmd,
             "data": outdata,  # temporary notice only
             "stderr": stderr,
-            "msg": traceback.format_exc()
+            "msg": traceback.format_exc(),
+            "ansible_job_id": jid,
         }
-        result['ansible_job_id'] = jid
-        jobfile.write(json.dumps(result))
+        with open(tmp_job_path, "w") as jobfile:
+            json.dump(result, jobfile)
 
-    jobfile.close()
     os.rename(tmp_job_path, job_path)
 
 
@@ -307,6 +307,17 @@ def main():
                     time.sleep(step)
                     remaining = remaining - step
                     if remaining <= 0:
+                        with open(job_path + ".tmp", "w") as jobfile:
+                            json.dump(
+                                {
+                                    "failed": 1,
+                                    "cmd": cmd,
+                                    "msg": "Killed async task, timed out.",
+                                    "timeout": True,
+                                },
+                                jobfile
+                            )
+                        os.rename(job_path + ".tmp", job_path)
                         notice("Now killing %s" % (sub_pid))
                         os.killpg(sub_pid, signal.SIGKILL)
                         notice("Sent kill to group %s " % sub_pid)

@sivel sivel added the easyfix This issue is considered easy to fix by aspiring contributors. label Apr 8, 2021
@eqrx eqrx self-assigned this Apr 9, 2021
@ansibot ansibot added the has_pr This issue has an associated PR. label May 10, 2021
@Akasurde
Copy link
Member

@schem-codes Could you please check if PR #74199 resolves your issue and let us know? Thanks.

needs_info

@ansibot ansibot added the needs_info This issue requires further information. Please answer any outstanding questions. label May 15, 2021
@ansibot
Copy link
Contributor

ansibot commented Jun 16, 2021

@schem-codes This issue is waiting for your response. Please respond or the issue will be closed.

click here for bot help

@ansibot
Copy link
Contributor

ansibot commented Jul 18, 2021

@schem-codes You have not responded to information requests in this issue so we will assume it no longer affects you. If you are still interested in this, please create a new issue with the requested information.

click here for bot help

@ansibot ansibot closed this as completed Jul 18, 2021
@ansible ansible locked and limited conversation to collaborators Aug 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
affects_2.9 This issue/PR affects Ansible v2.9 bug This issue/PR relates to a bug. easyfix This issue is considered easy to fix by aspiring contributors. has_pr This issue has an associated PR. module This issue/PR relates to a module. needs_info This issue requires further information. Please answer any outstanding questions. P3 Priority 3 - Approved, No Time Limitation python3 support:core This issue/PR relates to code supported by the Ansible Engineering Team. verified This issue has been verified/reproduced by maintainer
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants
@sivel @Akasurde @samdoran @ansibot @nitzmahone @eqrx @larsche and others