OCPBUGS-21803: haproxy-template: Add 'no strict-limits' to address HAProxy 2.6 issue #527
Conversation
In HAProxy 2.6, the 'strict limits' setting, which enforces resource
limits (e.g., the number of open files), became the default behaviour.
Consequently, any failure to configure resource limits now results in
fatal errors. This change introduces 'no strict-limits' in the
template to revert to the behaviour of HAProxy 2.2. This is necessary
to ensure the router starts successfully when a maxconn setting cannot
be satisfied.
The HAProxy documentation for 'strict-limits' states:
Makes process fail at startup when a setrlimit fails. HAProxy
tries to set the best setrlimit according to what has been
calculated. If it fails, it will emit a warning. This option is
here to guarantee an explicit failure of HAProxy when those limits
fail. It is enabled by default. It may still be forcibly disabled
by prefixing it with the "no" keyword.
For example, and without this change, if you tune the
IngressController to have maxConnections: 2000000, and if the
requirement for 2000000 connections cannot be satisfied at runtime,
then the router pod will log the following message, and the HAProxy
process will exit fatally with:
sh-4.4$ ../reload-haproxy
[NOTICE] (62) : haproxy version is 2.6.13-234aa6d
[NOTICE] (62) : path to executable is /usr/sbin/haproxy
[ALERT] (62) : [/usr/sbin/haproxy.main()] Cannot raise FD limit to 4000237, limit is 1048576.
Setting 'no strict-limits' changes this to:
sh-4.4$ ../reload-haproxy
[NOTICE] (50) : haproxy version is 2.6.13-234aa6d
[NOTICE] (50) : path to executable is /usr/sbin/haproxy
[WARNING] (50) : [/usr/sbin/haproxy.main()] Cannot raise FD limit to 4000237, limit is 1048576.
[ALERT] (50) : [/usr/sbin/haproxy.main()] FD limit (1048576) too low for maxconn=2000000/maxsock=4000237. Please raise 'ulimit-n' to 4000237 or more to avoid any trouble.
- Checking http://localhost:80 ...
- Health check ok : 0 retry attempt(s).
And, despite the ALERT, the haproxy process will not fatally exit.
I always recommend using -1 (or 'auto') for the maxConnections setting
if you want to increase the limit beyond the current default. This
choice results in HAProxy dynamically calculating the maximum value
based on the available resource limits in the running container.
Importantly, when using the 'auto' setting, there are no warnings
emitted, regardless of whether 'no strict-limits' is used or not.
This solution will be re-evaluated after the 4.14.0 release.
Fixes: https://issues.redhat.com/browse/OCPBUGS-21803
frobware
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@frobware: This pull request references Jira Issue OCPBUGS-21803, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@frobware: This pull request references Jira Issue OCPBUGS-21803, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@frobware: This pull request references Jira Issue OCPBUGS-21803, which is valid. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This commit adds a test case to verify that the maxConnections setting
can be configured with the absolute upper limit of 2000000, as defined
by the API
% oc explain ingresscontroller.spec.tuningOptions.maxConnections
KIND: IngressController
VERSION: operator.openshift.io/v1
FIELD: maxConnections <integer>
DESCRIPTION:
maxConnections defines the maximum number of simultaneous
connections that can be established per HAProxy process.
Increasing this value allows each ingress controller pod to
handle more connections but at the cost of additional system
resources being consumed. Permitted values are: empty, 0, -1, and
the range 2000-2000000.
This change requires openshift/router#527.
Fixes: https://issues.redhat.com/browse/OCPBUGS-21803.
|
Added test case for 2,000,000 max connections in openshift/cluster-ingress-operator#983. |
|
Ran the case successfully: % ./bin/extended-platform-tests run all --dry-run | grep 50926 | ./bin/extended-platform-tests run -f - 1 pass, 0 skip (12m6s) /qe-approved |
|
@frobware: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
This commit adds a test case to verify that the maxConnections setting
can be configured with the absolute upper limit of 2000000, as defined
by the API
% oc explain ingresscontroller.spec.tuningOptions.maxConnections
KIND: IngressController
VERSION: operator.openshift.io/v1
FIELD: maxConnections <integer>
DESCRIPTION:
maxConnections defines the maximum number of simultaneous
connections that can be established per HAProxy process.
Increasing this value allows each ingress controller pod to
handle more connections but at the cost of additional system
resources being consumed. Permitted values are: empty, 0, -1, and
the range 2000-2000000.
This change requires openshift/router#527.
Fixes: https://issues.redhat.com/browse/OCPBUGS-21803.
|
Thanks! |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Miciah The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
@frobware: Jira Issue OCPBUGS-21803: Some pull requests linked via external trackers have merged: The following pull requests linked via external trackers have not merged: These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-21803 has not been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cherry-pick release-4.14 |
|
@frobware: new pull request created: #528 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This commit adds a test case to verify that the maxConnections setting
can be configured with the absolute upper limit of 2000000, as defined
by the API
% oc explain ingresscontroller.spec.tuningOptions.maxConnections
KIND: IngressController
VERSION: operator.openshift.io/v1
FIELD: maxConnections <integer>
DESCRIPTION:
maxConnections defines the maximum number of simultaneous
connections that can be established per HAProxy process.
Increasing this value allows each ingress controller pod to
handle more connections but at the cost of additional system
resources being consumed. Permitted values are: empty, 0, -1, and
the range 2000-2000000.
This change requires openshift/router#527.
Fixes: https://issues.redhat.com/browse/OCPBUGS-21803.
|
Fix included in accepted release 4.15.0-0.nightly-2023-10-24-230302 |
frobware
deleted the
OCPBUGS-21803-Ingress-stuck-in-progressing-when-maxConnections-increased-to-2000000
branch
In HAProxy 2.6, the 'strict limits' setting, which enforces resource
limits (e.g., the number of open files), became the default behaviour.
Consequently, any failure to configure resource limits now results in
fatal errors. This change introduces 'no strict-limits' in the
template to revert to the behaviour of HAProxy 2.2. This is necessary
to ensure the router starts successfully when a maxconn setting cannot
be satisfied.
The HAProxy documentation for 'strict-limits' states:
For example, and without this change, if you tune the
IngressController to have maxConnections: 2000000, and if the
requirement for 2000000 connections cannot be satisfied at runtime,
then the router pod will log the following message, and the HAProxy
process will exit with:
Setting 'no strict-limits' changes this to:
And, despite the ALERT, the haproxy process will not exit.
I always recommend using -1 (or 'auto') for the maxConnections setting
if you want to increase the limit beyond the current default. This
choice results in HAProxy dynamically calculating the maximum value
based on the available resource limits in the running container.
Importantly, when using the 'auto' setting, there are no warnings
emitted, regardless of whether 'no strict-limits' is used or not.
This solution will be re-evaluated after the 4.14.0 release.
Fixes: https://issues.redhat.com/browse/OCPBUGS-21803