AWX Receptor Node: IPv6 flapping (working, broken)

[discostur]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

I deployed AWX (v24.5.0) in k8s via helm. I added a receptor node (execution). If the server which i want to provision has an ipv4 address everything is working normal.

If the server i want to run my playbooks against as an ipv6 address, sometimes the job finishes successful and sometimes it failes. It is exactly the same job template. If i just click on re-run it sometimes works and sometimes not. Cannot see any pattern when it works and when not ...

AWX version

24.5.0

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

Debian

Web browser

Firefox

Steps to reproduce

• deploy awx in k8s via helm
• add debian (11.9) receptor execution node
• provision servers with ipv6 address

Expected results

Working

Identity added: /runner/artifacts/1145/ssh_key_data (/runner/artifacts/1145/ssh_key_data)
ansible-playbook [core 2.15.12]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Loading callback plugin awx_display of type stdout, v2.0 from /runner/artifacts/1145/callback/awx_display.py
Skipping callback 'awx_display', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: CloudCheckSSH.yml ****************************************************
Positional arguments: CloudCheckSSH/CloudCheckSSH.yml
verbosity: 4
remote_user: shell
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/runner/inventory',)
subset: 2001:4178:6:1416:0000:000b:a:14
extra_vars: ('@/runner/env/extravars',)
forks: 5
1 plays in CloudCheckSSH/CloudCheckSSH.yml

PLAY [--> CloudCheckSSH] *******************************************************

TASK [Gathering Facts] *********************************************************
task path: /runner/project/CloudCheckSSH/CloudCheckSSH.yml:4
<2001:4178:6:1416:0000:000b:a:14> ESTABLISH SSH CONNECTION FOR USER: root
<2001:4178:6:1416:0000:000b:a:14> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/d04603209b"' 2001:4178:6:1416:0000:000b:a:14 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<2001:4178:6:1416:0000:000b:a:14> (0, b'/root\\n', b'OpenSSH_8.7p1, OpenSSL 3.2.1 30 Jan 2024\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for \\'final all\\' host 2001:4178:6:1416:0000:000b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched \\'final\\'\\r\\ndebug2: match not found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug1: configuration requests final Match pass\\r\\ndebug2: resolve_canonicalize: hostname 2001:4178:6:1416:0000:000b:a:14 is address\\r\\ndebug2: resolve_canonicalize: canonicalised address "2001:4178:6:1416:0000:000b:a:14" => "2001:4178:6:1416:0:b:a:14"\\r\\ndebug1: re-parsing configuration\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for \\'final all\\' host 2001:4178:6:1416:0:b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched \\'final\\'\\r\\ndebug2: match found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug3: expanded UserKnownHostsFile \\'~/.ssh/known_hosts\\' -> \\'/root/.ssh/known_hosts\\'\\r\\ndebug3: expanded UserKnownHostsFile \\'~/.ssh/known_hosts2\\' -> \\'/root/.ssh/known_hosts2\\'\\r\\ndebug1: auto-mux: Trying existing master\\r\\ndebug1: Control socket "/runner/cp/d04603209b" does not exist\\r\\ndebug3: ssh_connect_direct: entering\\r\\ndebug1: Connecting to 2001:4178:6:1416:0:b:a:14 [2001:4178:6:1416:0:b:a:14] port 22.\\r\\ndebug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x48\\r\\ndebug2: fd 3 setting O_NONBLOCK\\r\\ndebug1: fd 3 clearing O_NONBLOCK\\r\\ndebug1: Connection established.\\r\\ndebug3: timeout: 9997 ms remain after connect\\r\\ndebug1: identity file /root/.ssh/id_rsa type -1\\r\\ndebug1: identity file /root/.ssh/id_rsa-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_dsa type -1\\r\\ndebug1: identity file /root/.ssh/id_dsa-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_ecdsa type -1\\r\\ndebug1: identity file /root/.ssh/id_ecdsa-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_ecdsa_sk type -1\\r\\ndebug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_ed25519 type -1\\r\\ndebug1: identity file /root/.ssh/id_ed25519-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_ed25519_sk type -1\\r\\ndebug1: identity file /root/.ssh/id_ed25519_sk-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_xmss type -1\\r\\ndebug1: identity file /root/.ssh/id_xmss-cert type -1\\r\\ndebug1: Local version string SSH-2.0-OpenSSH_8.7\\r\\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u2\\r\\ndebug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u2 pat OpenSSH* compat 0x04000000\\r\\ndebug2: fd 3 setting O_NONBLOCK\\r\\ndebug1: Authenticating to 2001:4178:6:1416:0:b:a:14:22 as \\'root\\'\\r\\ndebug1: load_hostkeys: fopen /root/.ssh/known_hosts: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory\\r\\ndebug3: order_hostkeyalgs: no algorithms matched; accept original\\r\\ndebug3: send packet: type 20\\r\\ndebug1: SSH2_MSG_KEXINIT sent\\r\\ndebug3: receive packet: type 20\\r\\ndebug1: SSH2_MSG_KEXINIT received\\r\\ndebug2: local client KEXINIT proposal\\r\\ndebug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c,kex-strict-c-v00@openssh.com\\r\\ndebug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256\\r\\ndebug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr\\r\\ndebug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr\\r\\ndebug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512\\r\\ndebug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512\\r\\ndebug2: compression ctos: zlib@openssh.com,zlib,none\\r\\ndebug2: compression stoc: zlib@openssh.com,zlib,none\\r\\ndebug2: languages ctos: \\r\\ndebug2: languages stoc: \\r\\ndebug2: first_kex_follows 0 \\r\\ndebug2: reserved 0 \\r\\ndebug2: peer server KEXINIT proposal\\r\\ndebug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,kex-strict-s-v00@openssh.com\\r\\ndebug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519\\r\\ndebug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\\r\\ndebug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\\r\\ndebug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\\r\\ndebug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\\r\\ndebug2: compression ctos: none,zlib@openssh.com\\r\\ndebug2: compression stoc: none,zlib@openssh.com\\r\\ndebug2: languages ctos: \\r\\ndebug2: languages stoc: \\r\\ndebug2: first_kex_follows 0 \\r\\ndebug2: reserved 0 \\r\\ndebug3: kex_choose_conf: will use strict KEX ordering\\r\\ndebug1: kex: algorithm: curve25519-sha256\\r\\ndebug1: kex: host key algorithm: ssh-ed25519\\r\\ndebug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: zlib@openssh.com\\r\\ndebug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: zlib@openssh.com\\r\\ndebug1: kex: curve25519-sha256 need=32 dh_need=32\\r\\ndebug1: kex: curve25519-sha256 need=32 dh_need=32\\r\\ndebug3: send packet: type 30\\r\\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\\r\\ndebug3: receive packet: type 31\\r\\ndebug1: SSH2_MSG_KEX_ECDH_REPLY received\\r\\ndebug1: Server host key: ssh-ed25519 SHA256:atqPBuFfxO6Sc9gPTvfkVqgaxu03/rGssvmMhSUg00U\\r\\ndebug1: load_hostkeys: fopen /root/.ssh/known_hosts: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory\\r\\nWarning: Permanently added \\'2001:4178:6:1416:0:b:a:14\\' (ED25519) to the list of known hosts.\\r\\ndebug1: check_host_key: hostkey not known or explicitly trusted: disabling UpdateHostkeys\\r\\ndebug3: send packet: type 21\\r\\ndebug1: ssh_packet_send2_wrapped: resetting send seqnr 3\\r\\ndebug2: set_newkeys: mode 1\\r\\ndebug1: rekey out after 4294967296 blocks\\r\\ndebug1: SSH2_MSG_NEWKEYS sent\\r\\ndebug1: expecting SSH2_MSG_NEWKEYS\\r\\ndebug3: receive packet: type 21\\r\\ndebug1: ssh_packet_read_poll2: resetting read seqnr 3\\r\\ndebug1: SSH2_MSG_NEWKEYS received\\r\\ndebug2: set_newkeys: mode 0\\r\\ndebug1: rekey in after 4294967296 blocks\\r\\ndebug1: Will attempt key: /runner/artifacts/1145/ssh_key_data RSA SHA256:+mnZxEhj4lNVhNfzvE860S9Yz7I5nMkYNBJe8HaGOnI agent\\r\\ndebug1: Will attempt key: /root/.ssh/id_rsa \\r\\ndebug1: Will attempt key: /root/.ssh/id_dsa \\r\\ndebug1: Will attempt key: /root/.ssh/id_ecdsa \\r\\ndebug1: Will attempt key: /root/.ssh/id_ecdsa_sk \\r\\ndebug1: Will attempt key: /root/.ssh/id_ed25519 \\r\\ndebug1: Will attempt key: /root/.ssh/id_ed25519_sk \\r\\ndebug1: Will attempt key: /root/.ssh/id_xmss \\r\\ndebug2: pubkey_prepare: done\\r\\ndebug3: send packet: type 5\\r\\ndebug3: receive packet: type 7\\r\\ndebug1: SSH2_MSG_EXT_INFO received\\r\\ndebug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>\\r\\ndebug1: kex_input_ext_info: publickey-hostbound@openssh.com (unrecognised)\\r\\ndebug3: receive packet: type 6\\r\\ndebug2: service_accept: ssh-userauth\\r\\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\\r\\ndebug3: send packet: type 50\\r\\ndebug3: receive packet: type 51\\r\\ndebug1: Authentications that can continue: publickey,password\\r\\ndebug3: start over, passed a different list publickey,password\\r\\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\\r\\ndebug3: authmethod_lookup publickey\\r\\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\\r\\ndebug3: authmethod_is_enabled publickey\\r\\ndebug1: Next authentication method: publickey\\r\\ndebug1: Offering public key: /runner/artifacts/1145/ssh_key_data RSA SHA256:+mnZxEhj4lNVhNfzvE860S9Yz7I5nMkYNBJe8HaGOnI agent\\r\\ndebug3: send packet: type 50\\r\\ndebug2: we sent a publickey packet, wait for reply\\r\\ndebug3: receive packet: type 60\\r\\ndebug1: Server accepts key: /runner/artifacts/1145/ssh_key_data RSA SHA256:+mnZxEhj4lNVhNfzvE860S9Yz7I5nMkYNBJe8HaGOnI agent\\r\\ndebug3: sign_and_send_pubkey: RSA SHA256:+mnZxEhj4lNVhNfzvE860S9Yz7I5nMkYNBJe8HaGOnI\\r\\ndebug3: sign_and_send_pubkey: signing using rsa-sha2-256 SHA256:+mnZxEhj4lNVhNfzvE860S9Yz7I5nMkYNBJe8HaGOnI\\r\\ndebug3: send packet: type 50\\r\\ndebug3: receive packet: type 52\\r\\ndebug1: Enabling compression at level 6.\\r\\nAuthenticated to 2001:4178:6:1416:0:b:a:14 ([2001:4178:6:1416:0:b:a:14]:22) using "publickey".\\r\\ndebug1: pkcs11_del_provider: called, provider_id = (null)\\r\\ndebug1: setting up multiplex master socket\\r\\ndebug3: muxserver_listen: temporary control path /runner/cp/d04603209b.Q9xNMF20HvKiTMbY\\r\\ndebug2: fd 4 setting O_NONBLOCK\\r\\ndebug3: fd 4 is O_NONBLOCK\\r\\ndebug3: fd 4 is O_NONBLOCK\\r\\ndebug1: channel 0: new [/runner/cp/d04603209b]\\r\\ndebug3: muxserver_listen: mux listener channel 0 fd 4\\r\\ndebug2: fd 3 setting TCP_NODELAY\\r\\ndebug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x20\\r\\ndebug1: control_persist_detach: backgrounding master process\\r\\ndebug2: control_persist_detach: background process is 22\\r\\ndebug2: fd 4 setting O_NONBLOCK\\r\\ndebug1: forking to background\\r\\ndebug1: Entering interactive session.\\r\\ndebug1: pledge: id\\r\\ndebug2: set_control_persist_exit_time: schedule exit in 60 seconds\\r\\ndebug1: multiplexing control connection\\r\\ndebug2: fd 5 setting O_NONBLOCK\\r\\ndebug3: fd 5 is O_NONBLOCK\\r\\ndebug1: channel 1: new [mux-control]\\r\\ndebug3: channel_post_mux_listener: new mux channel 1 fd 5\\r\\ndebug3: mux_master_read_cb: channel 1: hello sent\\r\\ndebug2: set_control_persist_exit_time: cancel scheduled exit\\r\\ndebug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4\\r\\ndebug2: mux_master_process_hello: channel 1 client version 4\\r\\ndebug2: mux_client_hello_exchange: master version 4\\r\\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\\r\\ndebug3: mux_client_request_session: entering\\r\\ndebug3: mux_client_request_alive: entering\\r\\ndebug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4\\r\\ndebug2: mux_master_process_alive_check: channel 1: alive check\\r\\ndebug3: mux_client_request_alive: done pid = 24\\r\\ndebug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 75\\r\\ndebug2: mux_master_process_new_session: channel 1: request tty 0, X 0, agent 0, subsys 0, term "xterm", cmd "/bin/sh -c \\'echo ~root && sleep 0\\'", env 0\\r\\ndebug3: mux_client_request_session: session request sent\\r\\ndebug3: mux_master_process_new_session: got fds stdin 6, stdout 7, stderr 8\\r\\ndebug1: channel 2: new [client-session]\\r\\ndebug2: mux_master_process_new_session: channel_new: 2 linked to control channel 1\\r\\ndebug2: channel 2: send open\\r\\ndebug3: send packet: type 90\\r\\ndebug3: receive packet: type 80\\r\\ndebug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0\\r\\ndebug3: receive packet: type 4\\r\\ndebug1: Remote: /root/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding\\r\\ndebug3: receive packet: type 4\\r\\ndebug1: Remote: /root/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding\\r\\ndebug3: receive packet: type 91\\r\\ndebug2: channel_input_open_confirmation: channel 2: callback start\\r\\ndebug2: client_session2_setup: id 2\\r\\ndebug1: Sending command: /bin/sh -c \\'echo ~root && sleep 0\\'\\r\\ndebug2: channel 2: request exec confirm 1\\r\\ndebug3: send packet: type 98\\r\\ndebug3: mux_session_confirm: sending success reply\\r\\ndebug2: channel_input_open_confirmation: channel 2: callback done\\r\\ndebug2: channel 2: open confirm rwindow 0 rmax 32768\\r\\ndebug1: mux_client_request_session: master session id: 2\\r\\ndebug2: channel 2: rcvd adjust 2097152\\r\\ndebug3: receive packet: type 99\\r\\ndebug2: channel_input_status_confirm: type 99 id 2\\r\\ndebug2: exec request accepted on channel 2\\r\\ndebug3: receive packet: type 96\\r\\ndebug2: channel 2: rcvd eof\\r\\ndebug2: channel 2: output open -> drain\\r\\ndebug2: channel 2: obuf empty\\r\\ndebug2: chan_shutdown_write: channel 2: (i0 o1 sock -1 wfd 7 efd 8 [write])\\r\\ndebug2: channel 2: output drain -> closed\\r\\ndebug3: receive packet: type 98\\r\\ndebug1: client_input_channel_req: channel 2 rtype exit-status reply 0\\r\\ndebug3: mux_exit_message: channel 2: exit message, exitval 0\\r\\ndebug3: receive packet: type 98\\r\\ndebug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0\\r\\ndebug2: channel 2: rcvd eow\\r\\ndebug2: chan_shutdown_read: channel 2: (i0 o3 sock -1 wfd 6 efd 8 [write])\\r\\ndebug2: channel 2: input open -> closed\\r\\ndebug3: receive packet: type 97\\r\\ndebug2: channel 2: rcvd close\\r\\ndebug3: channel 2: will not send data after close\\r\\ndebug2: channel 2: send close\\r\\ndebug3: send packet: type 97\\r\\ndebug2: channel 2: is dead\\r\\ndebug2: channel 2: gc: notify user\\r\\ndebug3: mux_master_session_cleanup_cb: entering for channel 2\\r\\ndebug2: channel 1: rcvd close\\r\\ndebug2: channel 1: output open -> drain\\r\\ndebug2: chan_shutdown_read: channel 1: (i0 o1 sock 5 wfd 5 efd -1 [closed])\\r\\ndebug2: channel 1: input open -> closed\\r\\ndebug2: channel 2: gc: user detached\\r\\ndebug2: channel 2: is dead\\r\\ndebug2: channel 2: garbage collecting\\r\\ndebug1: channel 2: free: client-session, nchannels 3\\r\\ndebug3: channel 2: status: The following connections are open:\\r\\n  #1 mux-control (t16 nr0 i3/0 o1/16 e[closed]/0 fd 5/5/-1 sock 5 cc -1)\\r\\n  #2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1)\\r\\n\\r\\ndebug2: channel 1: obuf empty\\r\\ndebug2: chan_shutdown_write: channel 1: (i3 o1 sock 5 wfd 5 efd -1 [closed])\\r\\ndebug2: channel 1: output drain -> closed\\r\\ndebug2: channel 1: is dead (local)\\r\\ndebug2: channel 1: gc: notify user\\r\\ndebug3: mux_master_control_cleanup_cb: entering for channel 1\\r\\ndebug2: channel 1: gc: user detached\\r\\ndebug2: channel 1: is dead (local)\\r\\ndebug3: mux_client_read_packet: read header failed: Broken pipe\\r\\ndebug2: channel 1: garbage collecting\\r\\ndebug1: channel 1: free: mux-control, nchannels 2\\r\\ndebug2: Received exit status from master 0\\r\\ndebug3: channel 1: status: The following connections are open:\\r\\n  #1 mux-control (t16 nr0 i3/0 o3/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1)\\r\\n\\r\\ndebug2: set_control_persist_exit_time: schedule exit in 60 seconds\\r\\n')
<2001:4178:6:1416:0000:000b:a:14> ESTABLISH SSH CONNECTION FOR USER: root
<2001:4178:6:1416:0000:000b:a:14> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/d04603209b"' 2001:4178:6:1416:0000:000b:a:14 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1718316863.213383-19-63963168782651 `" && echo ansible-tmp-1718316863.213383-19-63963168782651="` echo /root/.ansible/tmp/ansible-tmp-1718316863.213383-19-63963168782651 `" ) && sleep 0'"'"''

Actual results

Broken / Network unreachable

Identity added: /runner/artifacts/1144/ssh_key_data (/runner/artifacts/1144/ssh_key_data)
ansible-playbook [core 2.15.12]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Loading callback plugin awx_display of type stdout, v2.0 from /runner/artifacts/1144/callback/awx_display.py
Skipping callback 'awx_display', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: CloudCheckSSH.yml ****************************************************
Positional arguments: CloudCheckSSH/CloudCheckSSH.yml
verbosity: 4
remote_user: shell
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/runner/inventory',)
subset: 2001:4178:6:1416:0000:000b:a:14
extra_vars: ('@/runner/env/extravars',)
forks: 5
1 plays in CloudCheckSSH/CloudCheckSSH.yml

PLAY [--> CloudCheckSSH] *******************************************************

TASK [Gathering Facts] *********************************************************
task path: /runner/project/CloudCheckSSH/CloudCheckSSH.yml:4
<2001:4178:6:1416:0000:000b:a:14> ESTABLISH SSH CONNECTION FOR USER: root
<2001:4178:6:1416:0000:000b:a:14> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/d04603209b"' 2001:4178:6:1416:0000:000b:a:14 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<2001:4178:6:1416:0000:000b:a:14> (255, b'', b'OpenSSH_8.7p1, OpenSSL 3.2.1 30 Jan 2024\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for \\'final all\\' host 2001:4178:6:1416:0000:000b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched \\'final\\'\\r\\ndebug2: match not found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug1: configuration requests final Match pass\\r\\ndebug2: resolve_canonicalize: hostname 2001:4178:6:1416:0000:000b:a:14 is address\\r\\ndebug2: resolve_canonicalize: canonicalised address "2001:4178:6:1416:0000:000b:a:14" => "2001:4178:6:1416:0:b:a:14"\\r\\ndebug1: re-parsing configuration\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for \\'final all\\' host 2001:4178:6:1416:0:b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched \\'final\\'\\r\\ndebug2: match found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug3: expanded UserKnownHostsFile \\'~/.ssh/known_hosts\\' -> \\'/root/.ssh/known_hosts\\'\\r\\ndebug3: expanded UserKnownHostsFile \\'~/.ssh/known_hosts2\\' -> \\'/root/.ssh/known_hosts2\\'\\r\\ndebug1: auto-mux: Trying existing master\\r\\ndebug1: Control socket "/runner/cp/d04603209b" does not exist\\r\\ndebug3: ssh_connect_direct: entering\\r\\ndebug1: Connecting to 2001:4178:6:1416:0:b:a:14 [2001:4178:6:1416:0:b:a:14] port 22.\\r\\ndebug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x48\\r\\ndebug2: fd 3 setting O_NONBLOCK\\r\\ndebug1: connect to address 2001:4178:6:1416:0:b:a:14 port 22: Network is unreachable\\r\\nssh: connect to host 2001:4178:6:1416:0:b:a:14 port 22: Network is unreachable\\r\\n')
fatal: [2001:4178:6:1416:0000:000b:a:14]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: OpenSSH_8.7p1, OpenSSL 3.2.1 30 Jan 2024\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for 'final all' host 2001:4178:6:1416:0000:000b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final'\\r\\ndebug2: match not found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug1: configuration requests final Match pass\\r\\ndebug2: resolve_canonicalize: hostname 2001:4178:6:1416:0000:000b:a:14 is address\\r\\ndebug2: resolve_canonicalize: canonicalised address \\"2001:4178:6:1416:0000:000b:a:14\\" => \\"2001:4178:6:1416:0:b:a:14\\"\\r\\ndebug1: re-parsing configuration\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for 'final all' host 2001:4178:6:1416:0:b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final'\\r\\ndebug2: match found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'\\r\\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'\\r\\ndebug1: auto-mux: Trying existing master\\r\\ndebug1: Control socket \\"/runner/cp/d04603209b\\" does not exist\\r\\ndebug3: ssh_connect_direct: entering\\r\\ndebug1: Connecting to 2001:4178:6:1416:0:b:a:14 [2001:4178:6:1416:0:b:a:14] port 22.\\r\\ndebug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x48\\r\\ndebug2: fd 3 setting O_NONBLOCK\\r\\ndebug1: connect to address 2001:4178:6:1416:0:b:a:14 port 22: Network is unreachable\\r\\nssh: connect to host 2001:4178:6:1416:0:b:a:14 port 22: Network is unreachable",
    "unreachable": true
}

PLAY RECAP *********************************************************************
2001:4178:6:1416:0000:000b:a:14 : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0 

Additional information

I even did a tcpdump on the interface on my receptor node and i can see that when awx says Network is unreachable it DOES NOT try to connect. I can not see any tcp packet on the interface.

When it is working i see normal tcp packets on the interface.

I did some ICMP / manual SSH connection attempts and it always works. I run a ICMP test for about 60 minutes from the receptor node to the target server and no single package got dropped. Network is table and running without any issues.

[discostur]

Ok debugged it the last days and it seems to be related to slirp4netns. Maybe its a race condition or something like that. However, with podman 3 and slirp4netns 1.0.1 there seems to be many ipv6 issues (when you scroll through the github issues). Since my receptor node (and also podman) is running on debian 11 i won't get any newer versions with bugfixes.

However i found a workaround to disable slirp4netns and enable host networking. At the moment this cannot be configured on a job or template based option - you can just set it global via job settings (in awx ui):

old: 

[
  "--network",
  "slirp4netns:enable_ipv6=true"
]

new: 
[
  "--network=host"
]

I found someone who was able to set it via extra_settings but that didn't work for me:

https://forum.ansible.com/t/awx-instance-receptor-podman-bug/2599/5
https://github.com/ansible/awx-operator/blob/devel/docs/user-guide/advanced-configuration/extra-settings.md

Podman then was always running a command like

/usr/bin/podman run --rm --tty --interactive --workdir /runner/project -v /tmp/awx_1193_ycvbr77j/:/runner/:Z --env-file /tmp/awx_1193_ycvbr77j/artifacts/1193/env.list --quiet --name ansible_runner_1193 --user=root - - n e t w o r k = h o s t

which seems to be a syntax issue. However in the configmap on awx it look correct.

It would also be nice to configure such podman run options on a job or template base level and not just globally. It seems there are some tickets already open but nothing happend:

#11552
#12339

So for the moment it works for me via global job configuration (network=host).