Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add go runtime scheduler latency metrics #24745

Merged
merged 1 commit into from Apr 12, 2023
Merged

Add go runtime scheduler latency metrics #24745

merged 1 commit into from Apr 12, 2023

Conversation

derailed
Copy link
Contributor

@derailed derailed commented Apr 4, 2023 •
edited

The Agent/Controller largely depends on various goroutines being scheduled on time to perform critical control plane tasks (i.e. like controllers, etc...)

We want to be able to detect/alert goroutine scheduling latency, specifically when CPU contention is so great that things may not be running on time.

  • Added GOR scheduler latency metric to the default GO metrics collector

Signed-off-by: Fernand Galiana fernand.galiana@isovalent.com

Please ensure your pull request adheres to the following guidelines:

  • For first time contributors, read Submitting a pull request
  • All code is covered by unit and/or runtime tests where feasible.
  • All commits contain a well written commit description including a title,
    description and a Fixes: #XXX line if the commit addresses a particular
    GitHub issue.
  • If your commit description contains a Fixes: <commit-id> tag, then
    please add the commit author[s] as reviewer[s] to this issue.
  • All commits are signed off. See the section Developer’s Certificate of Origin
  • Provide a title or release-note blurb suitable for the release notes.
  • Are you a user of Cilium? Please add yourself to the Users doc
  • Thanks for contributing!
Expose Cilium agent go runtime scheduler latency prometheus metric `go_sched_latencies_seconds`

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Apr 4, 2023
@derailed derailed marked this pull request as ready for review April 5, 2023 13:39
@derailed derailed requested a review from a team as a code owner April 5, 2023 13:39
@derailed derailed force-pushed the feat/go_rt_latency branch 2 times, most recently from 17bb2fb to a1915f7 Compare April 5, 2023 19:47
@tommyp1ckles tommyp1ckles added area/metrics Impacts statistics / metrics gathering, eg via Prometheus. release-note/minor This PR changes functionality that users may find relevant to operating Cilium. labels Apr 5, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Apr 5, 2023
@christarazi christarazi added the area/daemon Impacts operation of the Cilium daemon. label Apr 5, 2023
christarazi
christarazi requested changes Apr 5, 2023
View reviewed changes
Copy link
Member

@christarazi christarazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

It looks good to me. There are a couple of things that need to be addressed

  • The checkpatch job in CI is failing on a "bad sign-off". I think this can be fixed by amending your commit by doing git commit --amend --signoff.
  • We should add a note in the upgrade guide for v1.14 that a new metric was added. You can add this inside Documentation/operations/upgrade.rst under the "Added Metrics" section.

@derailed derailed requested a review from a team as a code owner April 5, 2023 21:57
@christarazi
Copy link
Member

FYI @derailed in case you didn't notice, but the sign-off is from your personal email.

zacharysarah
zacharysarah approved these changes Apr 5, 2023
View reviewed changes
Copy link
Contributor

@zacharysarah zacharysarah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from a (limited) docs perspective.

@derailed derailed force-pushed the feat/go_rt_latency branch 3 times, most recently from ffcfe93 to 18ee432 Compare April 6, 2023 15:04
@tommyp1ckles
Copy link
Contributor

tommyp1ckles commented Apr 7, 2023 •
edited by maintainer-s-little-helper bot

/test

Job 'Cilium-PR-K8s-1.25-kernel-4.19' failed:

Click to show.

Test Name

K8sDatapathConfig Host firewall With VXLAN and endpoint routes

Failure Output

FAIL: Error deleting resource /home/jenkins/workspace/Cilium-PR-K8s-1.25-kernel-4.19/src/github.com/cilium/cilium/test/k8s/manifests/host-policies.yaml: Cannot retrieve "cilium-w7s5n"'s policy revision: cannot get policy revision: ""

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.25-kernel-4.19/1669/

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.25-kernel-4.19 so I can create one.

@tommyp1ckles
Copy link
Contributor

/test-1.25-4.19

@tommyp1ckles
Copy link
Contributor

@derailed Some possible flakes on the tests, rerunning failed suites.

@christarazi
Copy link
Member

christarazi commented Apr 11, 2023 •
edited by maintainer-s-little-helper bot

/test

Job 'Cilium-PR-K8s-1.16-kernel-4.19' hit: #24697 (88.40% similarity)

@maintainer-s-little-helper maintainer-s-little-helper bot mentioned this pull request Apr 11, 2023
Closed
@derailed
Add go runtime scheduler latency metrics
a682929 
The Agent/Controller largely depends on various goroutines being scheduled on
time to perform critical control plane tasks (i.e. like controllers, etc...)

We want to be able to detect/alert goroutine scheduling latency, specifically
when CPU contention is so great that things may not be running on time.

- Added GOR scheduler latency metric to the default GO metrics collector

Signed-off-by: derailed <fernand.galiana@isovalent.com>
@christarazi
Copy link
Member

/test

@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Apr 12, 2023
@tommyp1ckles
Copy link
Contributor

Tests look good 🙏

@christarazi christarazi merged commit 702cc00 into cilium:master Apr 12, 2023
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zacharysarah zacharysarah zacharysarah approved these changes

@christarazi christarazi christarazi approved these changes

@tommyp1ckles tommyp1ckles tommyp1ckles approved these changes

Assignees
No one assigned
Labels
area/daemon Impacts operation of the Cilium daemon. area/metrics Impacts statistics / metrics gathering, eg via Prometheus. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/minor This PR changes functionality that users may find relevant to operating Cilium.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@derailed @christarazi @tommyp1ckles @zacharysarah