Performance problems

[zuzzas]

NPD Version

v0.6.3

Kernel version

4.15.0

What happened?

Kernel log got hammered with a lot of OOM kills. Totaling to about 4500 in a minute. This led to an extremely high CPU usage by node-problem-detector.

At a cursory glance, most of the time is spent in the kmsg's watchLoop. I've used Linux's perf utility, because I had no time to debug the issue properly with go tool pprof.

I'll try to find some time for the benchmarks, just leaving this here for now.

[xueweiz]

/assign @xueweiz
Thanks for filing the issue! I wanted to do some performance testing & improvement on NPD as well.

I think we should make sure NPD's resource consumption (cpu, mem, IO) always get bounded.
i.e. If there is a million problem going on, NPD should restrain from using too much resource detecting & reporting these problems. It's better to detect and report the issues slowly, rather than further punish the system by taking more resource away.

We could start with putting a CPU usage capacity on NPD. I think we could either use cgroup/cpulimit to limit NPD's CPU usage. We could add a systemd unit for starting NPD with recommended configuration under node-problem-detector/deployment/systemd/

@wangzhen127 WDYT?

@zuzzas Hi, I wonder if you are running NPD as a standalone daemon or as a Kubernetes DaemonSet? And if you are running it as a standalone daemon, I wonder if your OS has system? Thanks!

[k8s-ci-robot]

@xueweiz: GitHub didn't allow me to assign the following users: xueweiz.

Note that only kubernetes members and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @xueweiz
Thanks for filing the issue! I wanted to do some performance testing & improvement on NPD as well.

I think we should make sure NPD's resource consumption (cpu, mem, IO) always get bounded.
i.e. If there is a million problem going on, NPD should restrain from using too much resource detecting & reporting these problems. It's better to detect and report the issues slowly, rather than further punish the system by taking more resource away.

We could start with putting a CPU usage capacity on NPD. I think we could either use cgroup/cpulimit to limit NPD's CPU usage. We could add a systemd unit for starting NPD with recommended configuration under node-problem-detector/deployment/systemd/

@wangzhen127 WDYT?

@zuzzas Hi, I wonder if you are running NPD as a standalone daemon or as a Kubernetes DaemonSet? And if you are running it as a standalone daemon, I wonder if your OS has system? Thanks!

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wangzhen127]

I think we should make sure NPD's resource consumption (cpu, mem, IO) always get bounded.
i.e. If there is a million problem going on, NPD should restrain from using too much resource detecting & reporting these problems. It's better to detect and report the issues slowly, rather than further punish the system by taking more resource away.

Totally agree.

We could start with putting a CPU usage capacity on NPD. I think we could either use cgroup/cpulimit to limit NPD's CPU usage. We could add a systemd unit for starting NPD with recommended configuration under node-problem-detector/deployment/systemd/

This can be the starting point. I think the problem is that NPD still needs to that much amount of work, it just becomes slower.

I think a better (and long term) way is to define the priority of the problems. When NPD found itself approaching the resource usage limit, it should try to complete the high priority problem detentions first, and drop low priority things. Or have a "degrade" mode, where NPD can drop log entries in order to catch up with the log production within CPU limit.

cc @Random-Liu @andyxning @dchen1107 for thoughts.

[zuzzas]

@xueweiz
As a DaemonSet. CPU Limits, obviously, worked, but the container got throttled into oblivion.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[wangzhen127]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[zuzzas]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wangzhen127]

/reopen

[k8s-ci-robot]

@wangzhen127: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wangzhen127]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.