Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stop throttling KKP controllers #12764

Merged
merged 1 commit into from
Oct 20, 2023

Conversation

xrstf
Copy link
Contributor

@xrstf xrstf commented Oct 20, 2023

What this PR does / why we need it:
The resource constraints for KKP seed/master controllers are ancient. I (apparently) set them in #3012, a time when there wasn't even a master-controller-manager, but only an "rbac generator". "100m" might have been a useful limit for something that generates RBAC only, but the master-ctrl-mgr nowadays does way more.

We have observed it hitting the 100m CPU limit very often, to a point where even

I1020 11:42:42.896768 1 trace.go:236] Trace[1992949100]: "DeltaFIFO Pop Process" ID:cluster-lrvfphkcvb/front-proxy-ca,Depth:91,Reason:slow event handlers blocking the queue (20-Oct-2023 11:42:42.695) (total time: 201ms):

warnings were logged.

This PR raises the limit considerably, which should make KKP reconciliations much smoother.

What type of PR is this?
/kind bug

Does this PR introduce a user-facing change? Then add your Release Note here:

Increase default CPU limits for KKP API/seed/master-controller-managers to prevent general slowness.

Documentation:

NONE

@kubermatic-bot kubermatic-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. docs/none Denotes a PR that doesn't need documentation (changes). kind/bug Categorizes issue or PR as related to a bug. dco-signoff: yes Denotes that all commits in the pull request have the valid DCO signoff message. do-not-merge/code-freeze Indicates that a PR should not merge because it has not been approved for code freeze yet. labels Oct 20, 2023
@xrstf xrstf self-assigned this Oct 20, 2023
@kubermatic-bot kubermatic-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 20, 2023
@embik
Copy link
Member

embik commented Oct 20, 2023

Nit: Let's add information to what new limits we are updating in the release note.

@xrstf xrstf requested a review from embik October 20, 2023 12:33
Copy link
Member

@embik embik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@kubermatic-bot kubermatic-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 20, 2023
@kubermatic-bot
Copy link
Contributor

LGTM label has been added.

Git tree hash: d4f74e850b254e5b2f50f4689e52844b21606a4f

@kubermatic-bot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: embik

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubermatic-bot kubermatic-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 20, 2023
@embik embik added the code-freeze-approved Indicates a PR has been approved by release managers during code freeze. label Oct 20, 2023
@kubermatic-bot kubermatic-bot removed the do-not-merge/code-freeze Indicates that a PR should not merge because it has not been approved for code freeze yet. label Oct 20, 2023
@kubermatic-triage-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs

Review the full test history

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@xrstf
Copy link
Contributor Author

xrstf commented Oct 20, 2023

/override pre-kubermatic-test-integration

@kubermatic-bot
Copy link
Contributor

@xrstf: Overrode contexts on behalf of xrstf: pre-kubermatic-test-integration

In response to this:

/override pre-kubermatic-test-integration

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@xrstf xrstf added this to the KKP 2.24 milestone Oct 20, 2023
@kubermatic-triage-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs

Review the full test history

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@kubermatic-triage-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs

Review the full test history

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@kubermatic-bot kubermatic-bot merged commit f856f54 into kubermatic:main Oct 20, 2023
16 checks passed
@xrstf xrstf deleted the stop-throttling-kkp branch October 21, 2023 09:12
@xrstf
Copy link
Contributor Author

xrstf commented Oct 21, 2023

/cherrypick release/v2.23

@xrstf
Copy link
Contributor Author

xrstf commented Oct 21, 2023

/cherrypick release/v2.22

@xrstf
Copy link
Contributor Author

xrstf commented Oct 21, 2023

/cherrypick release/v2.21

@kubermatic-bot
Copy link
Contributor

@xrstf: new pull request created: #12769

In response to this:

/cherrypick release/v2.23

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kubermatic-bot
Copy link
Contributor

@xrstf: new pull request created: #12770

In response to this:

/cherrypick release/v2.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kubermatic-bot
Copy link
Contributor

@xrstf: #12764 failed to apply on top of branch "release/v2.21":

Applying: stop throttling KKP controllers
Using index info to reconstruct a base tree...
A	docs/zz_generated.kubermaticConfiguration.ce.yaml
A	docs/zz_generated.kubermaticConfiguration.ee.yaml
A	pkg/defaulting/configuration.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/controller/operator/defaults/defaults.go
Auto-merging docs/zz_generated.kubermaticConfiguration.yaml
CONFLICT (modify/delete): docs/zz_generated.kubermaticConfiguration.ee.yaml deleted in HEAD and modified in stop throttling KKP controllers. Version stop throttling KKP controllers of docs/zz_generated.kubermaticConfiguration.ee.yaml left in tree.
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 stop throttling KKP controllers
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherrypick release/v2.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. code-freeze-approved Indicates a PR has been approved by release managers during code freeze. dco-signoff: yes Denotes that all commits in the pull request have the valid DCO signoff message. docs/none Denotes a PR that doesn't need documentation (changes). kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants