Open
Description
Description of the problem
When during an HAProxy reload, another configuration change happens, it can happen that dynamically applied changes may not be applied to the latest HAProxy process when the changes are processed while a newly reloaded HAProxy process is still in startup. This seems more likely, the longer HAProxy takes to reload a configuration change.
The instance is running in master-worker mode.
With the below logs we observed that
- the two pods (haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r and haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6) that started the reload at 22:43:34 were the ones that did not receive the updated backend. The health check of the backends was failing on these pods and the client connections were routed to a default backend.
- the pod (haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88) that started the reload only at 22:44:01 was working and serving traffic to the updated backend successfully
| Time | Pod | Log |
|---|---|---|
| 2025-04-19 22:44:18.19207424 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | [NOTICE] (89) : Loading success. |
| 2025-04-19 22:44:01.343866624 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | [NOTICE] (89) : Reloading HAProxy |
| 2025-04-19 22:43:51.181850368 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | [NOTICE] (88) : Loading success. |
| 2025-04-19 22:43:51.22416128 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | [NOTICE] (89) : Loading success. |
| 2025-04-19 22:43:42.631933184 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 22:43:42.631609 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
| 2025-04-19 22:43:42.556745984 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 22:43:42.556610 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
| 2025-04-19 22:43:42.322117120 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | W0419 22:43:42.321955 10 dynupdate.go:458] unrecognized response adding/updating endpoint default_backend-coordinator_12345/srv001: No such backend. |
| 2025-04-19 22:43:42.294593792 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | W0419 22:43:42.294516 10 dynupdate.go:458] unrecognized response adding/updating endpoint default_backend-coordinator_13456/srv001: No such backend. |
| 2025-04-19 22:43:41.875889664 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 22:43:41.875586 10 converters.go:67] applying 4 change notifications: [update/Endpoints:default/backend-0 update/Endpoints:default/backend-discovery update/Endpoints:default/backend-subdomain update/Endpoints:default/backend-coordinator] |
| 2025-04-19 22:43:41.871669504 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 22:43:41.871438 10 converters.go:67] applying 4 change notifications: [update/Endpoints:default/backend-0 update/Endpoints:default/backend-discovery update/Endpoints:default/backend-subdomain update/Endpoints:default/backend-coordinator] |
| 2025-04-19 22:43:41.862155008 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 22:43:41.861942 10 converters.go:67] applying 4 change notifications: [update/Endpoints:default/backend-0 update/Endpoints:default/backend-discovery update/Endpoints:default/backend-subdomain update/Endpoints:default/backend-coordinator] |
| 2025-04-19 22:43:34.307413760 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | [NOTICE] (89) : Reloading HAProxy |
| 2025-04-19 22:43:34.219276288 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | [NOTICE] (88) : Reloading HAProxy |
| 2025-04-19 22:43:17.943481856 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | [NOTICE] (89) : Loading success. |
Once the backend pod was restarted (killed), haproxy was again updated without needing to reload. Only at this time the backend became healthy and connections were successful again.
| Time | Pod | Log |
|---|---|---|
| 2025-04-19 23:08:30.314232576 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 23:08:30.314125 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
| 2025-04-19 23:08:30.311860480 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 23:08:30.311563 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
| 2025-04-19 23:08:30.271827712 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 23:08:30.271598 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
| 2025-04-19 23:08:29.597730560 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 23:08:29.597615 10 converters.go:67] applying 4 change notifications: [update/Endpoints:default/backend-discovery update/Endpoints:default/backend-0 update/Endpoints:default/backend-coordinator update/Endpoints:default/backend-subdomain] |
| 2025-04-19 23:08:29.586635520 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 23:08:29.586533 10 converters.go:67] applying 4 change notifications: [update/Endpoints:default/backend-discovery update/Endpoints:default/backend-0 update/Endpoints:default/backend-coordinator update/Endpoints:default/backend-subdomain] |
| 2025-04-19 23:08:29.583913728 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 23:08:29.583604 10 converters.go:67] applying 4 change notifications: [update/Endpoints:default/backend-discovery update/Endpoints:default/backend-0 update/Endpoints:default/backend-coordinator update/Endpoints:default/backend-subdomain] |
| 2025-04-19 23:08:28.341615616 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 23:08:28.341331 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
| 2025-04-19 23:08:28.295460096 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 23:08:28.295159 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
| 2025-04-19 23:08:28.287066880 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 23:08:28.286968 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
| 2025-04-19 23:08:28.104987136 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 23:08:28.104907 10 dynupdate.go:438] disabled endpoint '100.97.100.90:12345' on backend/server 'default_backend-coordinator_12345/srv001' |
| 2025-04-19 23:08:28.83690496 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 23:08:28.083612 10 dynupdate.go:438] disabled endpoint '100.97.100.90:12345' on backend/server 'default_backend-0_12345/backend-d5fxx' |
| 2025-04-19 23:08:28.70128640 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 23:08:28.070056 10 dynupdate.go:438] disabled endpoint '100.97.100.90:13456' on backend/server 'default_backend-coordinator_13456/srv001' |
| 2025-04-19 23:08:28.56912384 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 23:08:28.056865 10 dynupdate.go:438] disabled endpoint '100.97.100.90:13456' on backend/server 'default_backend-coordinator_13456/srv001' |
| 2025-04-19 23:08:28.44797696 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 23:08:28.044731 10 dynupdate.go:438] disabled endpoint '100.97.100.90:12345' on backend/server 'default_backend-0_12345/backend-d5fxx' |
| 2025-04-19 23:08:28.21609216 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 23:08:28.021543 10 dynupdate.go:438] disabled endpoint '100.97.100.90:12345' on backend/server 'default_backend-coordinator_12345/srv001' |
| 2025-04-19 23:08:28.20207872 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 23:08:28.020174 10 dynupdate.go:438] disabled endpoint '100.97.100.90:12345' on backend/server 'default_backend-0_12345/backend-d5fxx' |
| 2025-04-19 23:08:28.4992768 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 23:08:28.004938 10 dynupdate.go:438] disabled endpoint '100.97.100.90:13456' on backend/server 'default_backend-coordinator_13456/srv001' |
| 2025-04-19 23:08:27.999251712 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 23:08:27.999211 10 dynupdate.go:438] disabled endpoint '100.97.100.90:12345' on backend/server 'default_backend-coordinator_12345/srv001' |
| 2025-04-19 23:08:27.599577088 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 23:08:27.599349 10 converters.go:67] applying 7 change notifications: [update/Endpoints:default/backend-discovery update/Endpoints:default/backend-subdomain update/Endpoints:default/backend-0 update/Endpoints:default/backend-coordinator update/Pod:default/backend update/Pod:default/backend update/Pod:default/backend] |
| 2025-04-19 23:08:27.584164608 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 23:08:27.584078 10 converters.go:67] applying 7 change notifications: [update/Endpoints:default/backend-discovery update/Endpoints:default/backend-subdomain update/Endpoints:default/backend-0 update/Endpoints:default/backend-coordinator update/Pod:default/backend update/Pod:default/backend update/Pod:default/backend] |
| 2025-04-19 23:08:27.582267904 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 23:08:27.582061 10 converters.go:67] applying 7 change notifications: [update/Endpoints:default/backend-discovery update/Endpoints:default/backend-subdomain update/Endpoints:default/backend-0 update/Endpoints:default/backend-coordinator update/Pod:default/backend update/Pod:default/backend update/Pod:default/backend] |
| 2025-04-19 23:08:02.569538304 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 23:08:02.569450 10 converters.go:69] applying 1 change notification: [update/Pod:default/backend] |
| 2025-04-19 23:08:02.562761216 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 23:08:02.562620 10 converters.go:69] applying 1 change notification: [update/Pod:default/backend] |
| 2025-04-19 23:08:02.557552384 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 23:08:02.557345 10 converters.go:69] applying 1 change notification: [update/Pod:default/backend] |
| 2025-04-19 22:58:35.283097600 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 22:58:35.282750 10 converters.go:69] applying 1 change notification: [update/Pod:default/backup-lcyc-backend-xlmjf] |
| 2025-04-19 22:58:35.275385088 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 22:58:35.275181 10 converters.go:69] applying 1 change notification: [update/Pod:default/backup-lcyc-backend-xlmjf] |
| 2025-04-19 22:58:35.271483392 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 22:58:35.271405 10 converters.go:69] applying 1 change notification: [update/Pod:default/backup-lcyc-backend-xlmjf] |
| 2025-04-19 22:46:47.844442112 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-jl9v6 | I0419 22:46:47.844309 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
| 2025-04-19 22:46:47.737763328 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-sll7r | I0419 22:46:47.737521 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
| 2025-04-19 22:46:47.731740672 UTC | haproxy-ingress-nlb-blue-7p4-5c9f8f9899-hjk88 | I0419 22:46:47.731330 10 instance.go:355] haproxy updated without needing to reload. Commands sent: 9 |
Expected behavior
The latest haproxy process should always use the latest available config.
Steps to reproduce the problem
Difficult to reproduce manually, probably easier with an integration test:
- Trigger HAProxy reload
- While HAProxy reloads, trigger a configuration change (endpoint change)
Environment information
HAProxy Ingress version: v0.14.6