Ingress controller crash randomly with exit code 111

[mdecalf]

Hello,
I run haproxy ingress controller ad Daemonset (4 replicas) on rke2.
I upgrade from helm release 1.39.4 (v1.11) which working well to 1.4X (I tried all v3.X) and the pod are crashing randomly and terminate with exit code 111.
I tried to run it with hostNetwork and without to see, same problem.
Tried also to run it as root and non root same problem.

The only thing that I can found in the logs except the access log and health check is

xxx:24088 [30/Jan/2025:09:52:21.788] https/v4: SSL handshake failure
xxx:14690 [30/Jan/2025:09:52:21.788] https/v4: SSL handshake failure
xxx:39793 [30/Jan/2025:09:52:21.788] https/v4: SSL handshake failure
xxx:42447 [30/Jan/2025:09:52:21.788] https/v4: SSL handshake failure
xxx:35800 [30/Jan/2025:09:52:21.788] https/v4: SSL handshake failure
xxx:39367 [30/Jan/2025:09:52:21.789] https/v4: SSL handshake failure
xxx:10664 [30/Jan/2025:09:52:21.789] https/v4: SSL handshake failure
xxx:40857 [30/Jan/2025:09:52:21.789] https/v4: SSL handshake failure
xxx:52486 [30/Jan/2025:09:52:21.789] https/v4: SSL handshake failure
xxx:58673 [30/Jan/2025:09:52:21.789] https/v4: SSL handshake failure
xxx:33828 [30/Jan/2025:09:52:21.789] https/v4: SSL handshake failure
xxx:38355 [30/Jan/2025:09:52:21.789] https/v4: SSL handshake failure
xxx:32023 [30/Jan/2025:09:52:21.789] https/v4: SSL handshake failure
xxx:59652 [30/Jan/2025:09:52:21.789] https/v4: SSL handshake failure
xxx:29675 [30/Jan/2025:09:52:21.789] https/v4: SSL handshake failure

and the termination log

s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service ingress-controller: stopping
Ingress Controller exited with code 0, restarting...
s6-rc: info: service ingress-controller successfully stopped
s6-rc: info: service haproxy: stopping
[WARNING]  (69) : Exiting Master process...
[ALERT]    (69) : Current worker (213) exited with code 143 (Terminated)
[WARNING]  (69) : All workers exited. Exiting... (0)
s6-rc: info: service haproxy successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service sigusr1: stopping
s6-rc: info: service aux-cfg: stopping
s6-rc: info: service sigusr1 successfully stopped
s6-rc: info: service aux-cfg successfully stopped
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

The parameters of the chart :

controller:
  kind: DaemonSet
  nodeSelector:
    role.xxx.com/haproxy: "true"
  extraArgs:
    - --healthz-bind-port=1042
    - --stats-bind-port=1024
    - --localpeer-port=10000
    - --prometheus
  containerPort:
    http: 80
    https: 443
    stat: 1024
    healthz: 1042
    prometheus: 6060
  ingressClass: haproxy
  ingressClassResource:
    enabled: false
    name: haproxy
    default: false
  defaultTLSSecret:
    enabled: false
  service:
    enabled: true
    type: ClusterIP
    ports:
      http: 8080
      https: 8443
      stat: 1024
      healthz: 1042
    prometheus: 6060
  daemonset:
    useHostPort: true
    hostPorts:
      http: 8080
      https: 8443
      stat: 1024
      healthz: 1042
    prometheus: 6060
  logging:
    level: warning
    traffic:
      address: stdout
      format: raw
      facility: daemon
  startupProbe:
    periodSeconds: 10
    timeoutSeconds: 5

Describe of the pod :

    State:          Running
      Started:      Thu, 30 Jan 2025 10:52:30 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    111
      Started:      Wed, 29 Jan 2025 18:13:01 +0100
      Finished:     Thu, 30 Jan 2025 10:52:29 +0100
    Ready:          True
    Restart Count:  10
    Requests:
      cpu:      250m
      memory:   400Mi
    Liveness:   http-get http://:1042/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:1042/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Startup:    http-get http://:1042/healthz delay=0s timeout=5s period=10s #success=1 #failure=20

The only specific thing that we have are snippet to have backup on backend

haproxy.org/backend-config-snippet: |
        option allbackups

Do you have an idea about what could be the issue?

[mdecalf]

Found ...

Default configuration on 3.x

pid = 244 (process #1, nbproc = 1, nbthread = 64)uptime = 0d 0h08m58s; warnings = 249system limits: memmax = 171728 MB; ulimit-n = 5847maxsock = 5847; maxconn = 100; reached = 0; maxpipes = 0current conns = 9; current pipes = 0/0; conn rate = 94/sec; bit rate = 24.556 MbpsRunning tasks: 0/5880 (0 niced); idle = 99 %

Default configuration on 2.8.x

pid = 223 (process #1, nbproc = 1, nbthread = 48)uptime = 2d 21h08m23s; warnings = 766system limits: memmax = 171596 MB; ulimit-n = 1044649maxsock = 1044649; maxconn = 520000; reached = 0; maxpipes = 0current conns = 10642; current pipes = 0/0; conn rate = 501/sec; bit rate = 51.994 MbpsRunning tasks: 2/16216; idle = 93 %

... it was reported in the changelog? I didn't found anything related to the change regarding the default maxconn

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.