Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1972076: jsonnet: Disable cpufreq collector in node_exporter #1229

Merged
merged 1 commit into from Jul 2, 2021

Conversation

bison
Copy link
Contributor

@bison bison commented Jun 17, 2021

The node_exporter cpufreq collector seems to be causing high load on
some machines with lots of cores. This disables it temporarily as a
workaround until the issue can be addressed upstream. We don't seem
to rely on these metrics anywhere.

See: https://bugzilla.redhat.com/show_bug.cgi?id=1972076

  • I added CHANGELOG entry for this change.
  • No user facing changes, so no entry in CHANGELOG was needed.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 17, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 17, 2021

@bison: Bugzilla bug 1972076 is in a bug group that is not in the allowed groups for this repo.
Allowed groups for this repo are:

  • qe_staff
  • redhat

In response to this:

WIP: Bug 1972076: jsonnet: Disable cpufreq collector in node_exporter

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 17, 2021
The node_exporter cpufreq collector seems to be causing high load on
some machines with lots of cores.  This disables it temporarily as a
workaround until the issue can be addressed upstream.  We don't seem
to rely on these metrics anywhere.

See: https://bugzilla.redhat.com/show_bug.cgi?id=1972076
@simonpasquier
Copy link
Contributor

simonpasquier commented Jul 1, 2021

/lgtm

even though we're not sure that node_exporter is the root cause for the kernel soft locks, it might amplify the issue. The procfs library (used to gather the CPU frequency metrics) reads the values from all CPUs in parallel [1] which creates undesired CPU spikes on nodes with many CPUs (prometheus/node_exporter#1880).

[1] https://github.com/prometheus/procfs/blob/5162bec877a860b5ff140b5d13db31ebb0643dd3/sysfs/system_cpu.go#L163-L174

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 1, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 1, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bison, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [bison,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@simonpasquier
Copy link
Contributor

/retitle Bug 1972076: jsonnet: Disable cpufreq collector in node_exporter

@openshift-ci openshift-ci bot changed the title WIP: Bug 1972076: jsonnet: Disable cpufreq collector in node_exporter Bug 1972076: jsonnet: Disable cpufreq collector in node_exporter Jul 2, 2021
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 2, 2021
@simonpasquier
Copy link
Contributor

/retest

@simonpasquier
Copy link
Contributor

/bugzilla refresh

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 2, 2021

@simonpasquier: Bugzilla bug 1972076 is in a bug group that is not in the allowed groups for this repo.
Allowed groups for this repo are:

  • qe_staff
  • redhat

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 2, 2021

@bison: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-aws-single-node ed1d1f6 link /test e2e-aws-single-node

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit a1ce745 into openshift:master Jul 2, 2021
@bison bison deleted the no-cpufreq branch July 5, 2021 09:49
@bison
Copy link
Contributor Author

bison commented Jul 9, 2021

/cherry-pick release-4.8

@openshift-cherrypick-robot

@bison: new pull request created: #1272

In response to this:

/cherry-pick release-4.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants