Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1860889: decrease CMO log verbosity from 3 to 2 #913

Merged
merged 1 commit into from Aug 25, 2020
Merged

Bug 1860889: decrease CMO log verbosity from 3 to 2 #913

merged 1 commit into from Aug 25, 2020

Conversation

simonpasquier
Copy link
Contributor

@simonpasquier simonpasquier commented Aug 21, 2020 •
edited

With -v=3, the logs are spammed by client-go telling that client requests have been throttled. Looking at the Kubernetes code, we won't lose anything useful by lowering the verbosity level.

The change also increases the QPS and burst settings (respectively 5->20 and 10->40) because the default values aren't suited for CMO that runs lots of tasks in parallel. With QPS=100 and Burst=200, the throttling logs disappeared but we felt that these values were too high for the API server.

  • I added CHANGELOG entry for this change.
  • No user facing changes, so no entry in CHANGELOG was needed.

@openshift-ci-robot openshift-ci-robot added bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Aug 21, 2020
@openshift-ci-robot
Copy link
Contributor

@simonpasquier: This pull request references Bugzilla bug 1860889, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.6.0) matches configured target release for branch (4.6.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1860889: increase QPS and burst settings for clients

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link
Contributor

@simonpasquier: This pull request references Bugzilla bug 1860889, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.6.0) matches configured target release for branch (4.6.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1860889: increase QPS and burst settings for clients

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 21, 2020
@simonpasquier simonpasquier force-pushed the bz-1860889 branch 5 times, most recently from c65de50 to 6c5496b Compare August 24, 2020 09:30
s-urbaniak
s-urbaniak reviewed Aug 24, 2020
View reviewed changes
pkg/client/client.go
@@ -716,7 +716,7 @@ func (c *Client) WaitForDeploymentRollout(dep *appsv1.Deployment) error {
}
if d.Status.UnavailableReplicas != 0 {
lastErr = errors.Errorf("got %d unavailable replicas",
d.Status.UpdatedReplicas)
d.Status.UnavailableReplicas)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch 👍

s-urbaniak
s-urbaniak reviewed Aug 24, 2020
View reviewed changes
cmd/operator/main.go Outdated
@@ -155,6 +155,10 @@ func Main() int {
return 1
}

// CMO runs many tasks in parallel and the default values for rate limiting are too low.
config.QPS = 50
config.Burst = 100
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we comment on how we came up with these numbers?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still experimenting :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still seeing throttling logs with QPS = 50 and Burst = 100 though less frequently.

@simonpasquier
Copy link
Contributor Author

/test e2e-aws-operator

@simonpasquier simonpasquier force-pushed the bz-1860889 branch 2 times, most recently from dd1e7fc to bb7799b Compare August 24, 2020 13:28
@simonpasquier
Bug 1860889: decrease CMO log verbosity from 3 to 2
1fa5e80 
With `-v=3`, the logs are spammed by client-go telling that client
requests have been throttled. Looking at the Kubernetes code, we won't
lose anything useful by lowering the verbosity level.

The change also increases the QPS and burst settings (respectively 5->20
and 10->40) because the default values aren't suited for CMO that runs
lots of tasks in parallel. With QPS=100 and Burst=200, the throttling
logs disappeared but we felt that these values were too high for the API
server.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
@simonpasquier simonpasquier changed the title Bug 1860889: increase QPS and burst settings for clients Bug 1860889: decrease CMO log verbosity from 3 to 2 Aug 24, 2020
@openshift-ci-robot
Copy link
Contributor

@simonpasquier: This pull request references Bugzilla bug 1860889, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.6.0) matches configured target release for branch (4.6.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1860889: decrease CMO log verbosity from 3 to 2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@simonpasquier
Copy link
Contributor Author

With QPS=100 and Burst=200, the throttling logs disappeared but it would be too demanding for the API server. Instead of optimizing these settings, lets decrease the log verbosity of CMO from 3 to 2: we shouldn't lose any useful information in the logs.
QPS and burst settings still get an increase because the default values aren't suited in the context of CMO.

@s-urbaniak
Copy link
Contributor

/lgtm

@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: s-urbaniak, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [s-urbaniak,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 24, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

3 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

6 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 1cdc504 into openshift:master Aug 25, 2020
@openshift-ci-robot
Copy link
Contributor

@simonpasquier: All pull requests linked via external trackers have merged:

Bugzilla bug 1860889 has been moved to the MODIFIED state.

In response to this:

Bug 1860889: decrease CMO log verbosity from 3 to 2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@simonpasquier simonpasquier deleted the bz-1860889 branch October 15, 2020 07:35
@simonpasquier
Copy link
Contributor Author

/cherry-pick release-4.5

@openshift-cherrypick-robot
Copy link

@simonpasquier: #913 failed to apply on top of branch "release-4.5":

Applying: Bug 1860889: decrease CMO log verbosity from 3 to 2
Using index info to reconstruct a base tree...
M	OWNERS
M	cmd/operator/main.go
M	manifests/0000_50_cluster_monitoring_operator_05-deployment.yaml
M	pkg/client/client.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/client/client.go
Auto-merging manifests/0000_50_cluster_monitoring_operator_05-deployment.yaml
Auto-merging cmd/operator/main.go
CONFLICT (content): Merge conflict in cmd/operator/main.go
Auto-merging OWNERS
CONFLICT (content): Merge conflict in OWNERS
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Bug 1860889: decrease CMO log verbosity from 3 to 2
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-4.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@simonpasquier simonpasquier mentioned this pull request Oct 15, 2020
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@s-urbaniak s-urbaniak s-urbaniak left review comments

@paulfantom paulfantom Awaiting requested review from paulfantom

Assignees

@s-urbaniak s-urbaniak

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@simonpasquier @openshift-ci-robot @s-urbaniak @openshift-bot @openshift-cherrypick-robot @openshift-merge-robot