in 2.13 some modules fail apparently trying to use local python path instead of remote.

[Vladimir-csp]

Summary

Trying ansible 2.13 on FreeBSD control host, Linux target host. Some modules fail while apparently trying to use python path from local machine.

Issue Type

Bug Report

Component Name

core

Ansible Version

$ ansible --version
ansible [core 2.13.1]

Configuration

$ ansible-config dump --only-changed
ANSIBLE_NOCOWS(/usr/local/etc/ansible/ansible.cfg) = True
ANSIBLE_PIPELINING(/usr/local/etc/ansible/ansible.cfg) = True
CALLBACKS_ENABLED(/usr/local/etc/ansible/ansible.cfg) = ['profile_tasks']
DEFAULT_BECOME_FLAGS(/usr/local/etc/ansible/ansible.cfg) = -H -S -n -E
DEFAULT_FORKS(/usr/local/etc/ansible/ansible.cfg) = 30
DEFAULT_GATHER_SUBSET(/usr/local/etc/ansible/ansible.cfg) = ['!hardware', 'network', 'virtual']
DEFAULT_HASH_BEHAVIOUR(/usr/local/etc/ansible/ansible.cfg) = replace
DEFAULT_LOAD_CALLBACK_PLUGINS(/usr/local/etc/ansible/ansible.cfg) = False
DEFAULT_LOG_PATH(/usr/local/etc/ansible/ansible.cfg) = /data/log/ansible/ansible.log
DEFAULT_POLL_INTERVAL(/usr/local/etc/ansible/ansible.cfg) = 15
DEFAULT_REMOTE_USER(/usr/local/etc/ansible/ansible.cfg) = ansible
DEFAULT_STDOUT_CALLBACK(/usr/local/etc/ansible/ansible.cfg) = yaml
DEFAULT_TIMEOUT(/usr/local/etc/ansible/ansible.cfg) = 30
HOST_KEY_CHECKING(/usr/local/etc/ansible/ansible.cfg) = True
INJECT_FACTS_AS_VARS(/usr/local/etc/ansible/ansible.cfg) = False
INTERPRETER_PYTHON(/usr/local/etc/ansible/ansible.cfg) = auto_silent
PERSISTENT_CONNECT_TIMEOUT(/usr/local/etc/ansible/ansible.cfg) = 30
TRANSFORM_INVALID_GROUP_CHARS(/usr/local/etc/ansible/ansible.cfg) = ignore
USE_PERSISTENT_CONNECTIONS(/usr/local/etc/ansible/ansible.cfg) = True

OS / Environment

FreeBSD 13.1, Ubuntu 20.04

Steps to Reproduce

on FreeBSD control host use any of affected modules (timezone, mount, sysctl, ini_file, alternatives, selinux) to deploy to a linux host. (There may be others affected).
Among not affected: debug, copy, template, group_by, assert, shell, command, systemd, stat, file, service...

Expected Results

Modules work correctly using discovered python on remote host (/usr/bin/python)

Actual Results

TASK [set timezone] ******************************************************
fatal: [hostname.example.com]: FAILED! => changed=false 
  module_stderr: |-
    /bin/sh: 1: /usr/local/bin/python3.8: not found
  module_stdout: ''
  msg: |-
    The module failed to execute correctly, you probably need to set the interpreter.
    See stdout/stderr for the exact error
  rc: 127

# /usr/local/bin/python3.8 is local path on FreeBSD control host.

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description:
None

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[sivel]

At a minimum, we're going to need the full output with at least -vvv to make any sort of guesses here.

[bcoca]

This is probably an issue with the BSD port itself as they traditionally rewrite the 'python location' to match the BSD platfrom (it does not know about remotes).

[Vladimir-csp]

ansible-playbook [core 2.13.1]
  config file = /usr/local/etc/ansible/ansible.cfg
  configured module search path = ['/home/username/.ansible/plugins/modules', '/usr/local/share/py38-ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/username/.ansible/collections:/usr/local/share/py38-ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.8.13 (default, May 20 2022, 05:42:25) [Clang 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c
  jinja version = 3.0.1
  libyaml = True
Using /usr/local/etc/ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /home/username/svn/ansible/inventories/test as it did not pass its verify_file() method
script declined parsing /home/username/svn/ansible/inventories/test as it did not pass its verify_file() method
auto declined parsing /home/username/svn/ansible/inventories/test as it did not pass its verify_file() method
Parsed /home/username/svn/ansible/inventories/test inventory source with ini plugin
redirecting (type: modules) ansible.builtin.ini_file to community.general.ini_file
Loading collection community.general from /usr/local/lib/python3.8/site-packages/ansible_collections/community/general
redirecting (type: modules) community.general.ini_file to community.general.files.ini_file
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Loading callback plugin community.general.yaml of type stdout, v2.0 from /usr/local/lib/python3.8/site-packages/ansible_collections/community/general/plugins/callback/yaml.py
redirecting (type: callback) ansible.builtin.profile_tasks to ansible.posix.profile_tasks
Loading collection ansible.posix from /usr/local/lib/python3.8/site-packages/ansible_collections/ansible/posix
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
Loading callback plugin ansible.posix.profile_tasks of type aggregate, v2.0 from /usr/local/lib/python3.8/site-packages/ansible_collections/ansible/posix/plugins/callback/profile_tasks.py

PLAYBOOK: test.yaml ************************************************************
Positional arguments: unixadm/test.yaml
verbosity: 4
remote_user: ansible
connection: smart
timeout: 30
become_method: sudo
tags: ('all',)
inventory: ('/home/username/svn/ansible/inventories/test',)
forks: 30
1 plays in unixadm/test.yaml

PLAY [Just Testing] ************************************************************
META: ran handlers

TASK [drop ini file] ***********************************************************
task path: /home/username/svn/ansible/unixadm/test.yaml:8
Thursday 14 July 2022  15:18:04 +0300 (0:00:00.031)       0:00:00.031 ********* 
<hostname.example.com> Attempting python interpreter discovery
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ansible
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' hostname.example.com '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'python3.10'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<hostname.example.com> (0, b'PLATFORM\nLinux\nFOUND\n/usr/bin/python3.8\n/usr/bin/python3\n/usr/bin/python2.7\n/usr/bin/python\n/usr/bin/python\nENDFOUND\n', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ansible
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' hostname.example.com '/bin/sh -c '"'"'/usr/bin/python3.8 && sleep 0'"'"''
<hostname.example.com> (0, b'{"platform_dist_result": [], "osrelease_content": "NAME=\\"Ubuntu\\"\\nVERSION=\\"20.04.4 LTS (Focal Fossa)\\"\\nID=ubuntu\\nID_LIKE=debian\\nPRETTY_NAME=\\"Ubuntu 20.04.4 LTS\\"\\nVERSION_ID=\\"20.04\\"\\nHOME_URL=\\"https://www.ubuntu.com/\\"\\nSUPPORT_URL=\\"https://help.ubuntu.com/\\"\\nBUG_REPORT_URL=\\"https://bugs.launchpad.net/ubuntu/\\"\\nPRIVACY_POLICY_URL=\\"https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\\"\\nVERSION_CODENAME=focal\\nUBUNTU_CODENAME=focal\\n"}\n', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/local/lib/python3.8/site-packages/ansible/modules/file.py
Pipelining is enabled.
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ansible
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' hostname.example.com '/bin/sh -c '"'"'/usr/bin/python3 && sleep 0'"'"''
<hostname.example.com> (0, b'\n{"path": "/tmp/test.ini", "changed": true, "diff": {"before": {"path": "/tmp/test.ini", "state": "file"}, "after": {"path": "/tmp/test.ini", "state": "absent"}}, "state": "absent", "invocation": {"module_args": {"path": "/tmp/test.ini", "state": "absent", "recurse": false, "force": false, "follow": true, "modification_time_format": "%Y%m%d%H%M.%S", "access_time_format": "%Y%m%d%H%M.%S", "unsafe_writes": false, "_original_basename": null, "_diff_peek": null, "src": null, "modification_time": null, "access_time": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null}}}\n', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
changed: [hostname.example.com] => changed=true 
  ansible_facts:
    discovered_interpreter_python: /usr/bin/python3
  diff:
    after:
      path: /tmp/test.ini
      state: absent
    before:
      path: /tmp/test.ini
      state: file
  invocation:
    module_args:
      _diff_peek: null
      _original_basename: null
      access_time: null
      access_time_format: '%Y%m%d%H%M.%S'
      attributes: null
      follow: true
      force: false
      group: null
      mode: null
      modification_time: null
      modification_time_format: '%Y%m%d%H%M.%S'
      owner: null
      path: /tmp/test.ini
      recurse: false
      selevel: null
      serole: null
      setype: null
      seuser: null
      src: null
      state: absent
      unsafe_writes: false
  path: /tmp/test.ini
  state: absent

TASK [init ini file] ***********************************************************
task path: /home/username/svn/ansible/unixadm/test.yaml:12
Thursday 14 July 2022  15:18:04 +0300 (0:00:00.459)       0:00:00.490 ********* 
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ansible
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' hostname.example.com '/bin/sh -c '"'"'echo ~ansible && sleep 0'"'"''
<hostname.example.com> (0, b'/home/ansible\n', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ansible
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' hostname.example.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp `"&& mkdir "` echo /home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243 `" && echo ansible-tmp-1657801084.5421062-81941-210528627909243="` echo /home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243 `" ) && sleep 0'"'"''
<hostname.example.com> (0, b'ansible-tmp-1657801084.5421062-81941-210528627909243=/home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243\n', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/local/lib/python3.8/site-packages/ansible/modules/stat.py
Pipelining is enabled.
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ansible
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' hostname.example.com '/bin/sh -c '"'"'/usr/bin/python3 && sleep 0'"'"''
<hostname.example.com> (0, b'\n{"changed": false, "stat": {"exists": false}, "invocation": {"module_args": {"path": "/tmp/test.ini", "follow": false, "get_checksum": true, "checksum_algorithm": "sha1", "get_md5": false, "get_mime": true, "get_attributes": true}}}\n', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<hostname.example.com> PUT /home/username/.ansible/tmp/ansible-local-81932uh6p5l9x/tmpl1o_3m5b TO /home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243/source
<hostname.example.com> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' '[hostname.example.com]'
<hostname.example.com> (0, b'sftp> put /home/username/.ansible/tmp/ansible-local-81932uh6p5l9x/tmpl1o_3m5b /home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243/source\n', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug2: Server supports extension "lsetstat@openssh.com" revision 1\r\ndebug3: Sent message fd 3 T:16 I:1\r\ndebug3: SSH2_FXP_REALPATH . -> /home/ansible\r\ndebug3: Looking up /home/username/.ansible/tmp/ansible-local-81932uh6p5l9x/tmpl1o_3m5b\r\ndebug3: Sent message fd 3 T:17 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent dest message SSH2_FXP_OPEN I:3 P:/home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243/source M:0x001a\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:0 S:14\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 14 bytes at 0\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ansible
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' hostname.example.com '/bin/sh -c '"'"'chmod u+x /home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243/ /home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243/source && sleep 0'"'"''
<hostname.example.com> (0, b'', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/local/lib/python3.8/site-packages/ansible/modules/copy.py
Pipelining is enabled.
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ansible
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' hostname.example.com '/bin/sh -c '"'"'/usr/bin/python3 && sleep 0'"'"''
<hostname.example.com> (0, b'\n{"dest": "/tmp/test.ini", "src": "/home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243/source", "md5sum": "fa92f52e62ca391a02749c69eab7f158", "checksum": "709abf8fca55a7c5034c8b0d02a61005d114b72c", "changed": true, "uid": 1002, "gid": 1002, "owner": "ansible", "group": "ansible", "mode": "0664", "state": "file", "size": 14, "invocation": {"module_args": {"dest": "/tmp/test.ini", "src": "/home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243/source", "_original_basename": "tmpl1o_3m5b", "follow": false, "checksum": "709abf8fca55a7c5034c8b0d02a61005d114b72c", "backup": false, "force": true, "unsafe_writes": false, "content": null, "validate": null, "directory_mode": null, "remote_src": null, "local_follow": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null}}}\n', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ansible
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' hostname.example.com '/bin/sh -c '"'"'rm -f -r /home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243/ > /dev/null 2>&1 && sleep 0'"'"''
<hostname.example.com> (0, b'', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
changed: [hostname.example.com] => changed=true 
  checksum: 709abf8fca55a7c5034c8b0d02a61005d114b72c
  dest: /tmp/test.ini
  diff: []
  gid: 1002
  group: ansible
  invocation:
    module_args:
      _original_basename: tmpl1o_3m5b
      attributes: null
      backup: false
      checksum: 709abf8fca55a7c5034c8b0d02a61005d114b72c
      content: null
      dest: /tmp/test.ini
      directory_mode: null
      follow: false
      force: true
      group: null
      local_follow: null
      mode: null
      owner: null
      remote_src: null
      selevel: null
      serole: null
      setype: null
      seuser: null
      src: /home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243/source
      unsafe_writes: false
      validate: null
  md5sum: fa92f52e62ca391a02749c69eab7f158
  mode: '0664'
  owner: ansible
  size: 14
  src: /home/ansible/.ansible/tmp/ansible-tmp-1657801084.5421062-81941-210528627909243/source
  state: file
  uid: 1002

TASK [tweak ini file] **********************************************************
task path: /home/username/svn/ansible/unixadm/test.yaml:18
Thursday 14 July 2022  15:18:05 +0300 (0:00:00.813)       0:00:01.304 ********* 
redirecting (type: modules) ansible.builtin.ini_file to community.general.ini_file
redirecting (type: modules) community.general.ini_file to community.general.files.ini_file
redirecting (type: modules) ansible.builtin.ini_file to community.general.ini_file
redirecting (type: modules) community.general.ini_file to community.general.files.ini_file
Using module file /usr/local/lib/python3.8/site-packages/ansible_collections/community/general/plugins/modules/files/ini_file.py
Pipelining is enabled.
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ansible
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ansible"' -o ConnectTimeout=30 -o 'ControlPath="~/.ansible/cp/%C"' hostname.example.com '/bin/sh -c '"'"'/usr/local/bin/python3.8 && sleep 0'"'"''
<hostname.example.com> (127, b'', b'OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022\r\ndebug1: [ssh stuff redacted]debug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81908\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\n/bin/sh: 1: /usr/local/bin/python3.8: not found\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 127\r\n')
<hostname.example.com> Failed to connect to the host via ssh: OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022
[ssh stuff redacted]
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 81908
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
/bin/sh: 1: /usr/local/bin/python3.8: not found
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 127
fatal: [hostname.example.com]: FAILED! => changed=false 
  module_stderr: |-
    OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022
    debug1: [ssh stuff redacted]
    debug1: auto-mux: Trying existing master
    debug2: fd 3 setting O_NONBLOCK
    debug2: mux_client_hello_exchange: master version 4
    debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
    debug3: mux_client_request_session: entering
    debug3: mux_client_request_alive: entering
    debug3: mux_client_request_alive: done pid = 81908
    debug3: mux_client_request_session: session request sent
    debug1: mux_client_request_session: master session id: 2
    /bin/sh: 1: /usr/local/bin/python3.8: not found
    debug3: mux_client_read_packet: read header failed: Broken pipe
    debug2: Received exit status from master 127
  module_stdout: ''
  msg: |-
    The module failed to execute correctly, you probably need to set the interpreter.
    See stdout/stderr for the exact error
  rc: 127

PLAY RECAP *********************************************************************
hostname.example.com : ok=2    changed=2    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

3 modules acting on one file: file, copy, ini_file. last one seems to be trying wrong python right from the start.

[sivel]

Yeah, this is as we assumed. This is an issue with the FreeBSD ansible (e.g. py38-ansible) package:

root@freebsd:~/ansible$ head -1 /usr/local/lib/python3.8/site-packages/ansible_collections/community/general/plugins/modules/files/ini_file.py
#!/usr/local/bin/python3.8

Because they have modified the shebang in the module file, our shebang replacer doesn't work properly. Specifically, due to the addition of 3.8 in the shebang, ansible_python_interpreter doesn't match.

In order for it to match, you would have to be able to set an ansible_python3.8_interpreter instead.

Since this is not an ansible problem, but a problem with FreeBSD packaging, I'm going to close this. You may want to open a bug report with FreeBSD.

If you have further questions please stop by IRC or the mailing list:

• IRC: #ansible on irc.libera.chat
• mailing list: https://groups.google.com/forum/#!forum/ansible-project

[Vladimir-csp]

2.12 port has the same shebang, but works. Was something changed in 2.13 that now requires special handling of shebangs in modules?

[sivel]

2.13 addressed a bug where the shebang of the module was not being properly consulted when doing shebang replacements via ansible_python_interpreter:

#76677

This was an expected change, and the prior behavior was incorrect.

This is still a bug with FreeBSD packaging.

[Vladimir-csp]

Do I understand correctly that shebangs in modules should now be either unversioned or absent entirely?

[sivel]

They should preferably be left untouched with a value of #!/usr/bin/python which is meant to be a replacer string, and not used as the actual shebang, or at a minimum it needs to be an unversioned path ending in exactly /python.

The shebang is replaced during runtime execution, with either the result of interpreter discovery, or the explicitly set ansible_python_interpreter variable.