OCPBUGS-23544: Increase concurrent reconciles to 10

[JoelSpeed]

Approving CSRs should not need to be in sequence, we should be able to process in parallel. Kubelet will create CSRs when it starts and these should be approved in a timely manner. On large scale events, we are slow to process the CSRs and this is causing kubelet to create duplicate CSRs.

Creating duplicates leads to us exhausting our limit which stops approval altogether.

The only time where we may have an issue is if 2 concurrent reconciles approve client CSRs for the same node at the same time, this today has a check in place such that we don't approve client CSRs when nodes already exist. In theory, this should never happen if we are approving the client CSRs promptly anyway.

I also noticed while working on this, that we are accounting for Multus CSRs when calculating the pending number of CSRs. As these CSRs are networking related, approved by a different controller and not related to our intended operation here, I'm excluding them from being considered as part of the pending CSR count.

[openshift-ci-robot]

@JoelSpeed: This pull request references Jira Issue OCPBUGS-23544, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sunzhaohua2

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Approving CSRs should not need to be in sequence, we should be able to process in parallel. Kubelet will create CSRs when it starts and these should be approved in a timely manner. On large scale events, we are slow to process the CSRs and this is causing kubelet to create duplicate CSRs.

Creating duplicates leads to us exhausting our limit which stops approval altogether.

The only time where we may have an issue is if 2 concurrent reconciles approve client CSRs for the same node at the same time, this today has a check in place such that we don't approve client CSRs when nodes already exist. In theory, this should never happen if we are approving the client CSRs promptly anyway.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[JoelSpeed]

/test all

[openshift-ci]

@JoelSpeed: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[theobarberbany]

/lgtm

[JoelSpeed]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JoelSpeed]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@JoelSpeed: Jira Issue OCPBUGS-23544: Some pull requests linked via external trackers have merged:

The following pull requests linked via external trackers have not merged:

• openshift/machine-api-provider-azure#87 is open

These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-23544 has not been moved to the MODIFIED state.

In response to this:

Approving CSRs should not need to be in sequence, we should be able to process in parallel. Kubelet will create CSRs when it starts and these should be approved in a timely manner. On large scale events, we are slow to process the CSRs and this is causing kubelet to create duplicate CSRs.

Creating duplicates leads to us exhausting our limit which stops approval altogether.

The only time where we may have an issue is if 2 concurrent reconciles approve client CSRs for the same node at the same time, this today has a check in place such that we don't approve client CSRs when nodes already exist. In theory, this should never happen if we are approving the client CSRs promptly anyway.

I also noticed while working on this, that we are accounting for Multus CSRs when calculating the pending number of CSRs. As these CSRs are networking related, approved by a different controller and not related to our intended operation here, I'm excluding them from being considered as part of the pending CSR count.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-cluster-machine-approver-container-v4.16.0-202401161431.p0.g2f95a70.assembly.stream for distgit ose-cluster-machine-approver.
All builds following this will include this PR.

[openshift-merge-robot]

Fix included in accepted release 4.16.0-0.nightly-2024-01-18-111941

[JoelSpeed]

/cherry-pick release-4.15

[openshift-cherrypick-robot]

@JoelSpeed: new pull request created: #226

In response to this:

/cherry-pick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.