Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release-4.4] Bug 1811713: Specify cgroups in kubelet.conf so cAdvisor stats will be tracked #1546

Merged
merged 1 commit into from Mar 11, 2020
Merged

[release-4.4] Bug 1811713: Specify cgroups in kubelet.conf so cAdvisor stats will be tracked #1546

merged 1 commit into from Mar 11, 2020

Conversation

openshift-cherrypick-robot
Copy link

This is an automated cherry-pick of #1540

/assign joelsmith

@joelsmith
Specify cgroups in kubelet.conf so cAdvisor stats will be tracked
0cb2380 
Specifying runtimeCgroups will cause the Kubelet to monitor
statistics for the specified cgroup so that we have statistics
for crio.

Specifying systemCgroups will cause the Kubelet to monitor
statistics for system processes and make that data available
in our metrics pipeline.

Specifying cgroupRoot explicitly is required when specifying
systemCgroups
@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Mar 9, 2020
Merged
@openshift-ci-robot openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Mar 9, 2020
@openshift-ci-robot
Copy link
Contributor

@openshift-cherrypick-robot: This pull request references Bugzilla bug 1806027, which is invalid:

  • expected the bug to target the "4.4.0" release, but it targets "4.5.0" instead
  • expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is ON_QA instead
  • expected Bugzilla bug 1806027 to depend on a bug targeting the "4.5.0" release and in one of the following states: MODIFIED, ON_QA, VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

[release-4.4] Bug 1806027: Specify cgroups in kubelet.conf so cAdvisor stats will be tracked

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Mar 9, 2020
@joelsmith
Copy link
Contributor

/retitle [release-4.4] Bug 1811713: Specify cgroups in kubelet.conf so cAdvisor stats will be tracked

@openshift-ci-robot openshift-ci-robot changed the title [release-4.4] Bug 1806027: Specify cgroups in kubelet.conf so cAdvisor stats will be tracked [release-4.4] Bug 1811713: Specify cgroups in kubelet.conf so cAdvisor stats will be tracked Mar 9, 2020
@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Mar 9, 2020
@openshift-ci-robot
Copy link
Contributor

@openshift-cherrypick-robot: This pull request references Bugzilla bug 1811713, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.4.0) matches configured target release for branch (4.4.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 1806027 is in the state ON_QA, which is one of the valid states (MODIFIED, ON_QA, VERIFIED, RELEASE_PENDING, CLOSED (ERRATA))
  • dependent Bugzilla bug 1806027 targets the "4.5.0" release, matching the expected (4.5.0) release
  • bug has dependents

In response to this:

[release-4.4] Bug 1811713: Specify cgroups in kubelet.conf so cAdvisor stats will be tracked

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Mar 9, 2020
@joelsmith
Copy link
Contributor

/retest

@smarterclayton
Copy link
Contributor

/hold

I don't think this actually fixed anything, after looking at the 4.5 latest.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 9, 2020
@joelsmith
Copy link
Contributor

joelsmith commented Mar 9, 2020 •
edited

@smarterclayton the CI tests were broken for most of the weekend and the master PR only merged early AM Sunday. Are you sure that image builds happened in time for the version you tested on? Are you seeing the cgroup changes in your nodes' kubelet.conf files?

$ grep -i cgroup /etc/kubernetes/kubelet.conf 
cgroupDriver: systemd
cgroupRoot: /
runtimeCgroups: /system.slice/crio.service
systemCgroups: /system.slice

I am seeing them like that on the 4.5 cluster that I just now installed.

Are you not able to see info from cadvisor? I am:

$ oc get --raw /api/v1/nodes/jsmith-libvirt-n2nnw-master-0/proxy/metrics/cadvisor | grep sshd.service
container_cpu_load_average_10s{container="",id="/system.slice/sshd.service",image="",name="",namespace="",pod=""} 0 1583793792528
container_cpu_system_seconds_total{container="",id="/system.slice/sshd.service",image="",name="",namespace="",pod=""} 0.11 1583793792528
container_cpu_usage_seconds_total{container="",cpu="total",id="/system.slice/sshd.service",image="",name="",namespace="",pod=""} 0.212079904 1583793792528
... snip ...

@smarterclayton
Copy link
Contributor

/hold cancel

We had a race in builds that prevented the latest version from getting pushed until the next build ran. Steve and I will track that down. This PR was manually verified and it has the data I expect.

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 10, 2020
@rphillips
Copy link
Contributor

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 10, 2020
@rphillips
Copy link
Contributor

/approve

@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: openshift-cherrypick-robot, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 10, 2020
@mrunalp mrunalp added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Mar 10, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

3 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 0f12a6b into openshift:release-4.4 Mar 11, 2020
@openshift-ci-robot
Copy link
Contributor

@openshift-cherrypick-robot: All pull requests linked via external trackers have merged. Bugzilla bug 1811713 has been moved to the MODIFIED state.

In response to this:

[release-4.4] Bug 1811713: Specify cgroups in kubelet.conf so cAdvisor stats will be tracked

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mrunalp mrunalp Awaiting requested review from mrunalp

@rphillips rphillips Awaiting requested review from rphillips

Assignees

@rphillips rphillips

@joelsmith joelsmith

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
8 participants
@openshift-cherrypick-robot @openshift-ci-robot @joelsmith @smarterclayton @rphillips @openshift-bot @mrunalp @openshift-merge-robot