AWX fails to start on IPv6 disabled environments

[digifuchsi]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Hi all,

I need to run AWX in an minikube environment with IPv6 disabled.

I can successfully install with awx-operator version <=0.23.0, but starting with version 0.24.0 it fails.
The awx-web container throws an error saying nginx cannot bind to IPv6 (Address family not supported by protocol).

As I cannot find any other issue in the logs I am certain this change is causing the issue. On other IPv6 disabled environments it is showing the same behaviour.

Is it possible to introduce a disable_ipv6 switch so that nginx does not listen to IPv6?
I can try creating a PR on my own if nobody has time to spare and the project would not mind having such a switch.

Thanks

Chris

AWX Operator version

0.24.0

AWX version

21.2.0

Kubernetes platform

minikube

Kubernetes/Platform version

minikube version: v1.25.2 / Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3"}

Modifications

no

Steps to reproduce

1. Disable IPv6 on host
2. Install minikube
3. Install AWX using operator > v0.23.0

Expected results

AWX is running after some time

Actual results

awx-web container keeps crashing and getting restarted throwing an error that nginx cannot bind to IPv6

Additional information

 # kubectl -n chris-tst logs awx-686c5f6c8c-bkts8 awx-web
[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1 of 30
2022-08-11 13:56:25,333 INFO RPC interface 'supervisor' initialized
2022-08-11 13:56:25,333 INFO RPC interface 'supervisor' initialized
2022-08-11 13:56:25,333 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-08-11 13:56:25,333 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-08-11 13:56:25,333 INFO supervisord started with pid 6
2022-08-11 13:56:25,333 INFO supervisord started with pid 6
2022-08-11 13:56:26,336 INFO spawned: 'superwatcher' with pid 25
2022-08-11 13:56:26,336 INFO spawned: 'superwatcher' with pid 25
2022-08-11 13:56:26,338 INFO spawned: 'nginx' with pid 26
2022-08-11 13:56:26,338 INFO spawned: 'nginx' with pid 26
2022-08-11 13:56:26,340 INFO spawned: 'uwsgi' with pid 27
2022-08-11 13:56:26,340 INFO spawned: 'uwsgi' with pid 27
2022-08-11 13:56:26,342 INFO spawned: 'daphne' with pid 28
2022-08-11 13:56:26,342 INFO spawned: 'daphne' with pid 28
2022-08-11 13:56:26,344 INFO spawned: 'wsbroadcast' with pid 29
2022-08-11 13:56:26,344 INFO spawned: 'wsbroadcast' with pid 29
2022-08-11 13:56:26,346 INFO spawned: 'awx-rsyslogd' with pid 30
2022-08-11 13:56:26,346 INFO spawned: 'awx-rsyslogd' with pid 30
READY
2022-08-11 13:56:26,352 INFO exited: nginx (exit status 1; not expected)
2022-08-11 13:56:26,352 INFO exited: nginx (exit status 1; not expected)
[uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 11 13:56:26 2022] ***
compiled with version: 11.3.1 20220421 (Red Hat 11.3.1-2) on 12 July 2022 16:57:29
os: Linux-4.12.14-197.105-default #1 SMP Mon Jan 31 17:51:58 UTC 2022 (eaeeffc)
nodename: awx-686c5f6c8c-bkts8
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /var/lib/awx
detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8050 fd 3
Python version: 3.9.13 (main, Jun  9 2022, 00:00:00)  [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)]
2022/08/11 13:56:26 [emerg] 26#26: socket() [::]:8052 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x19815d0
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 27)
spawned uWSGI worker 1 (pid: 31, cores: 1)
spawned uWSGI worker 2 (pid: 32, cores: 1)
spawned uWSGI worker 3 (pid: 33, cores: 1)
spawned uWSGI worker 4 (pid: 34, cores: 1)
spawned uWSGI worker 5 (pid: 35, cores: 1)
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
2022-08-11 13:56:27,419 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-08-11 13:56:27,419 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-08-11 13:56:27,421 INFO spawned: 'nginx' with pid 36
2022-08-11 13:56:27,421 INFO spawned: 'nginx' with pid 36
2022/08/11 13:56:27 [emerg] 36#36: socket() [::]:8052 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)
2022-08-11 13:56:27,502 INFO exited: nginx (exit status 1; not expected)
2022-08-11 13:56:27,502 INFO exited: nginx (exit status 1; not expected)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x19815d0 pid: 31 (default app)
2022-08-11 13:56:28,828 INFO     [-] daphne.cli Starting server at tcp:port=8051:interface=127.0.0.1
2022-08-11 13:56:28,828 INFO     Starting server at tcp:port=8051:interface=127.0.0.1
2022-08-11 13:56:28,829 INFO     [-] daphne.server HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2022-08-11 13:56:28,829 INFO     HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2022-08-11 13:56:28,829 INFO     [-] daphne.server Configuring endpoint tcp:port=8051:interface=127.0.0.1
2022-08-11 13:56:28,829 INFO     Configuring endpoint tcp:port=8051:interface=127.0.0.1
2022-08-11 13:56:28,831 INFO     [-] daphne.server Listening on TCP address 127.0.0.1:8051
2022-08-11 13:56:28,831 INFO     Listening on TCP address 127.0.0.1:8051
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x19815d0 pid: 32 (default app)
2022-08-11 13:56:29,563 INFO spawned: 'nginx' with pid 43
2022-08-11 13:56:29,563 INFO spawned: 'nginx' with pid 43
2022/08/11 13:56:29 [emerg] 43#43: socket() [::]:8052 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)
2022-08-11 13:56:29,583 INFO exited: nginx (exit status 1; not expected)
2022-08-11 13:56:29,583 INFO exited: nginx (exit status 1; not expected)
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x19815d0 pid: 33 (default app)
WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x19815d0 pid: 35 (default app)
WSGI app 0 (mountpoint='/') ready in 4 seconds on interpreter 0x19815d0 pid: 34 (default app)
2022-08-11 13:56:30,608 INFO     [-] awx.main.wsbroadcast Active instance with hostname awx-686c5f6c8c-bkts8 is registered.
2022-08-11 13:56:32,613 INFO spawned: 'nginx' with pid 53
2022-08-11 13:56:32,613 INFO spawned: 'nginx' with pid 53
2022/08/11 13:56:32 [emerg] 53#53: socket() [::]:8052 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)
2022-08-11 13:56:32,624 INFO exited: nginx (exit status 1; not expected)
2022-08-11 13:56:32,624 INFO exited: nginx (exit status 1; not expected)
2022-08-11 13:56:33,625 INFO gave up: nginx entered FATAL state, too many start retries too quickly
2022-08-11 13:56:33,625 INFO gave up: nginx entered FATAL state, too many start retries too quickly
Processing Event: ver:3.0 server:supervisor serial:0 pool:superwatcher poolserial:0 eventname:PROCESS_STATE_FATAL len:62
2022-08-11 13:56:34,627 WARN received SIGQUIT indicating exit request
2022-08-11 13:56:34,627 WARN received SIGQUIT indicating exit request
2022-08-11 13:56:34,627 INFO waiting for superwatcher, uwsgi, daphne, wsbroadcast, awx-rsyslogd to die
2022-08-11 13:56:34,627 INFO waiting for superwatcher, uwsgi, daphne, wsbroadcast, awx-rsyslogd to die
...brutally killing workers...
2022-08-11 13:56:34,628 INFO     [-] daphne.server Killed 0 pending application instances
2022-08-11 13:56:34,628 INFO     Killed 0 pending application instances
2022-08-11 13:56:34,653 INFO stopped: wsbroadcast (terminated by SIGTERM)
2022-08-11 13:56:34,653 INFO stopped: wsbroadcast (terminated by SIGTERM)
2022-08-11 13:56:34,653 INFO stopped: awx-rsyslogd (exit status 0)
2022-08-11 13:56:34,653 INFO stopped: awx-rsyslogd (exit status 0)
2022-08-11 13:56:35,318 INFO stopped: daphne (exit status 0)
2022-08-11 13:56:35,318 INFO stopped: daphne (exit status 0)
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
binary reloading uWSGI...
chdir() to /var/lib/awx
closing all non-uwsgi socket fds > 2 (max_fd = 1048576)...
found fd 3 mapped to socket 0 (127.0.0.1:8050)
running /var/lib/awx/venv/awx/bin/uwsgi
[uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 11 13:56:36 2022] ***
compiled with version: 11.3.1 20220421 (Red Hat 11.3.1-2) on 12 July 2022 16:57:29
os: Linux-4.12.14-197.105-default #1 SMP Mon Jan 31 17:51:58 UTC 2022 (eaeeffc)
nodename: awx-686c5f6c8c-bkts8
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /var/lib/awx
detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited INET address 127.0.0.1:8050 fd 3
Python version: 3.9.13 (main, Jun  9 2022, 00:00:00)  [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)]
2022-08-11 13:56:37,970 INFO waiting for superwatcher, uwsgi to die
2022-08-11 13:56:37,970 INFO waiting for superwatcher, uwsgi to die
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0xc69ea0
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 27)
spawned uWSGI worker 1 (pid: 54, cores: 1)
spawned uWSGI worker 2 (pid: 55, cores: 1)
spawned uWSGI worker 3 (pid: 56, cores: 1)
spawned uWSGI worker 4 (pid: 57, cores: 1)
spawned uWSGI worker 5 (pid: 58, cores: 1)
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
WSGI app 0 (mountpoint='/') ready in 1 seconds on interpreter 0xc69ea0 pid: 58 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0xc69ea0 pid: 55 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0xc69ea0 pid: 56 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0xc69ea0 pid: 54 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0xc69ea0 pid: 57 (default app)
2022-08-11 13:56:41,796 INFO waiting for superwatcher, uwsgi to die
2022-08-11 13:56:41,796 INFO waiting for superwatcher, uwsgi to die
2022-08-11 13:56:44,799 WARN killing 'uwsgi' (27) with SIGKILL
2022-08-11 13:56:44,799 WARN killing 'uwsgi' (27) with SIGKILL
2022-08-11 13:56:44,799 INFO waiting for superwatcher, uwsgi to die
2022-08-11 13:56:44,799 INFO waiting for superwatcher, uwsgi to die
2022-08-11 13:56:44,814 INFO stopped: uwsgi (terminated by SIGKILL)
2022-08-11 13:56:44,814 INFO stopped: uwsgi (terminated by SIGKILL)
2022-08-11 13:56:44,815 INFO stopped: superwatcher (terminated by SIGTERM)
2022-08-11 13:56:44,815 INFO stopped: superwatcher (terminated by SIGTERM)

Operator Logs

No response

[shanemcd]

Probably broke in #950

If someone can open a PR adding variables to enable / disable ipv4/ipv6 we can get that reviewed and tested.

[digifuchsi]

@shanemcd I gave it a try. Could you have a quick look at the change, please? I am not sure if I just need to update the template in the role or if I have to add this new variable to some other file as well.
devel...digifuchsi:awx-operator:devel

[shanemcd]

Sweet - can your open a PR and we can discuss there?

[digifuchsi]

Sure. Created #1024

[derhoeppi]

Hi i have the same issue. We need to run AWX on kubernetes host with disabled IPv6.
I take a look into the PR of @digifuchsi . It's a good solution but it is also possible to set this param in makefile (global)?

[digifuchsi]

@derhoeppi Sry but I have no clue how to add this. If you like you can add the needed code to the PR or create another PR to add those parameters.

[koshrf]

I can confirm too, can't deploy 0.24+ (up to 0.28/latest) on a K8s without IPv6 making AWX imposible to upgrade or install.

[amirm-hrl]

Our upgrade from 0.21 to 0.28 also failed from the same reason

[mklitzn]

As a quick workarround you could create a configmap "awx-awx-configmap-modified" with the modified https://github.com/ansible/awx-operator/blob/0.29.0/roles/installer/templates/configmaps/config.yaml.j2 and overwrite it within the operator via

spec:
  volumes:
    - name: configmap-modified
      configMap:
        name: awx-awx-configmap-modified
  containers:
    - name: awx-manager
      volumeMounts:
        - name: configmap-modified
          mountPath: /opt/ansible/roles/installer/templates/configmaps

[derhoeppi]

@mklitzn it looks like you have experience in kubernetes and the deployment of awx. Do you think it is possible to control IPv6 enable / disable in kustomization.yaml?

[SamuelHornsey]

Used the workaround mentioned by @mklitzn to fix our deployment. What is the status of the PR from @digifuchsi? That IPv6 change seems to have affected a fair few users and the error is difficult to debug and not immediately obvious. Thanks.

[kenichimilo]

+1 broken upgrade, as mentioned.. workaround didn't seem to work, 4 me..
A disable ipv6 could come in handy in the CRD

[rooftopcellist]

@kenichimilo It is now possible to disable IPv6 in the nginx config from the AWX CR's spec thanks to this PR:

• Added flag to disable ipv6 listener #1135

This is currently available in the devel branch, and will be available in the next released version of the awx-operator as well.

[TechBrain64]

While I appreciate everyone's efforts bringing this project to life and keeping it current, I would be remised if I didn't point out my heartache with this issue. I too came across the IPv6 disabled where awx-web container continually crashes. I eventually (days later) found issue posted and docs. However, the remediation steps of this issue is not clearly defined. How does one implement "Spec: ipv6_disabled: true"?