kube-proxy constantly syncing/restoring iptables rules, consuming CPU resources #1158
Comments
|
This is most likely due to this issue with kube-proxy. The fix hasn't yet been released in the latest non-beta version of kubernetes. Unfortunately trying to force |
|
@eden kubernetes/kubernetes#26637 seems to be an issue only with multiple schedulers and controller-managers so could it be something else? |
|
@dimpavloff Initially, I thought the same thing, but after increasing the verbosity of minikube's logs to 10, I see this in localkubes logs nearly continously: There's a lot going on in kubernetes/kubernetes#26637, but this fix in particular looks like it might help: kubernetes/kubernetes#41223 |
|
@eden kubernetes has fixed this issue in v1.5.4, so when will minikube enable this version? |
|
You should be able to get minikube to start using 1.5.4 by starting it with the You can use 1.6, too, if you're ok with a beta, and when I did, I noticed the iptables issue is gone. However, I noticed something new that seems to cause a small amount of continuous CPU use (although anecdotally not as much). I had to switch back because I wasn't ready for some other changes that 1.6 introduced. |
|
Two points
|
|
Confirming this issue still exists with Minikube running kubernetes 1.6.0, on a fresh install:
|
|
Same CPU issue, though not sure if related or not to the original one. VM Driver: xhyve Kubernetes: Logs: EDIT No more issue in the logs, but still high CPU. |
|
For what its worth, I was able to confirm that the iptables churn is no longer an issue with minikube 0.18.0, which includes kubernetes/kubernetes#41223. I can also confirm that the fix only cut CPU usage by about half (I was hovering around 25-30% on the osx host, now down to 10-15%), but its not iptables, as evidenced by the lack of iptables-related output from strace-ing localkube. |
|
Fresh installed minikube & kubectl on Ubuntu 17.04 with VirtualBox 5.1.18_Ubuntur114002:
Nothing started, nothing configured, just 'minikube start' and got ~30% CPU consuming at Core i7 |
|
Happens with these versions: Just did a minikube start and it's taking about 15-18% of my CPU. |
|
It got much worse with minikube v0.20.0 and kubernetes 1.7.0: $ minikube start --memory=6144 --vm-driver=xhyve --kubernetes-version v1.7.0
...
$ minikube version
minikube version: v0.20.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-30T09:51:01Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-30T10:17:58Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
$ minikube ssh
$ top
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
3208 root 20 0 10.829g 404.7m 50.7 6.8 2:30.70 S localkubeMy mac's activity monitor claims docker-machine-driver-xhyve is using about 110% CPU. The numbers for top is 50% of two CPUs. |
|
With 0.19.1 here is a snippet from my logs: |
|
Also seeing very high system CPU time on 1.7.0 / MacOS / VirtualBox. With 2 CPUs allocated, each is at 50% System CPU. VBox on the mac is chewing up 150% CPU. |
|
MacOS / VBox / Kube 1.6.4. |
|
MacOS / VBox / Kube 1.7.0. |
|
I don't think this makes much of a difference either but I see the same results with qemu on Debian 9 as well. Just wanted to give an idea that this appears to be minikube/localkube related and not MacOS related. |
|
@zacheryph great research :) There is a KubeProxyIPTablesConfiguration.MinSyncPeriod setting, see https://godoc.org/k8s.io/kubernetes/pkg/apis/componentconfig#KubeProxyIPTablesConfiguration
When I type The --extra-config=proxy.IPTables.SyncPeriod=1m --extra-config=proxy.IPTables.MinSyncPeriod=1min order to do the check every minute instead of continuously. However the CPU usage stays high even with that extra option for me :( I do see the option being passed on to localkube though. I copied an strace binary into the minikube VM, running it against the localkube process and using the -f flag shows it's continuously forking off iptables. Examples: It also performs quite a bit of HTTP GET requests various APIs: |
|
I tried to set most of the duration-settings described at https://godoc.org/k8s.io/kubernetes/pkg/apis/componentconfig to 1 minute, but no success in reducing CPU usage. |
|
Setting the options to 5 seconds by specifying it in nanoseconds helped: |
|
@oliverbestmann huge thanks! That one worked for me too, the laptop fan is now quiet again. According to https://kubernetes.io/docs/admin/kube-proxy/ the default SyncPeriod is meant to be 30 seconds, so set it to that in nanoseconds. Localcube now consumes "just" 14% according to "ps uaxS" (S includes the child processes), similar to the original bug description. |
|
I'll set it to the upstream defaults. This regression probably happened because of the new way of configuring kube-proxy, there is no longer a default config struct that we can init. I'll make sure the other defaults are properly set also. |
Set some kube-proxy defaults that got unset through the new way of configuring kube-proxy. Add more delay to the ip tables syncing reduces idle CPU load a lot. See kubernetes#1158 (comment)
|
I've sent #1699 to make those kube-proxy options the default for minikube. Once that is merged, I would like to set some benchmarks that we would like to hit to make this issue more concrete. Something like With default options:
I'm not sure exactly how we could set limits on the driver binary or minikube, since those will probably fluctuate a lot more running on different platforms. |
|
@r2d4 Thanks! |
|
This workaround doesn't seem to work for me: minikube start --kubernetes-version v1.7.0 --vm-driver xhyve --extra-config=proxy.IPTables.SyncPeriod.Duration=5000000000 --extra-config=proxy.IPTables.MinSyncPeriod.Duration=3000000000 Does this need to be done with more recent code than v0.20.0? I'm still seeing ~60-80%, ~30%, ~30% CPU usage on the 3 VBoxHeadless processes I have. |
|
@daveoconnor I'm not sure, but perhaps this requires a newer ISO than the current release uses by default. This $ minikube config view | grep iso-url
- iso-url: https://storage.googleapis.com/minikube/iso/minikube-v0.22.0.iso |
|
@alanbrent Thanks for the response.
I tried using As per the documentation at https://kubernetes.io/docs/getting-started-guides/minikube/#using-rkt-container-engine and the output I got was: No message of downloading a new ISO so I'm guessing it's already using that. After trying The startup output was the same, and I can't be sure the CPU load is any different. Not spiking quite as high but I'm not sure that's not a coincidence. |
|
@daveoconnor I think at least that I had to run |
|
@stela Thanks, good idea. I'm not seeing much change. Here's how that went: EDIT: removed xhyve driver section because I realised I shouldn't be using it. Sorry if that caused confusion. CPU processes still at 40-70%, 20-30%, 20-30%. |
|
FWIW, I am seeing significantly lower (from ~100% CPU utilization down to 20%) with minikube v0.21.0 (and no custom settings). Killing all of the localkube and k8s services brings VirtualBox down to 8% or so. |
|
It's been 8 months, anything happening with this? I'm not seeing any difference with the above flags (I've confirmed that |
|
Using around 15 to 20 percent CPU is after those flags was applied as a default. A little bit of tracing the minikube binary reveals most of the time spend in cgroup/cadvisor stats for getting the CPU usage of the containers and for querying various APIs internally by localkube or one of its components. |
|
I'm not sure if this is a workable solution for everyone, but I've solved this problem by switching to the Please note that in order for this to be effective (I believe) you'll need to |
|
@alanbrent just tried this with minikube 0.22.3 and the vm is still consuming around 18%-20% on the host. There's no localkube to blame anymore, but I see various kubernetes components collectively consuming around 5-20% cpu on the vm itself. It does not appear that kubeadm is helping things here. |
|
Issues go stale after 90d of inactivity. Prevent issues from auto-closing with an If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or |
k8s-ci-robot
added
the
lifecycle/stale
|
/remove-lifecycle stale |
k8s-ci-robot
removed
the
lifecycle/stale
|
Attaching a strace of localkube (localkube.strace.gz); looks like it's constantly polling stuff inside |
|
I am affected by this issue as well. Running minikube using virtualbox driver consistently drivers the CPU well over 100%. |
|
So I'm looking at 30% thrashing of hyperkit at idle state on a Mac Pro. When I I've been using the mac "Activity Monitor" when judging the behaviour of the hyperkit process. Using htop paints a different picture, this gives what I expect the timing reports to be with the cpu idling on average at 2%. When I was doing systems programming "Activity Monitor" was off about the virtual memory metrics -- so I can completely believe it is just flat out wrong. Could one of the systems programmers here shed some light on the discrepancies between "Activity monitor" and A simple guide on correctly profiling and interpreting the vm process and the processes inside the vm would help a lot in diagnosing and providing feedback on such issues. |
|
disregard the last one, |
|
Same here. Using Ubuntu, minikube v0.25.2, kubernetes 1.7.5, VirtualBox. It's idling at 40% CPU with nothing installed. |
tstromberg
added
co/xhyve
bootstrapper/localkube
area/performance
tstromberg
changed the title
|
The iptables issues were fixed long ago. Please re-open other performance issues as new bugs. |
and others
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT
Minikube version (use
minikube version): 0.16.0Environment:
cat ~/.minikube/machines/minikube/config.json | grep DriverName):xhyvecat ~/.minikube/machines/minikube/config.json | grep ISO):minikube-v1.0.6.isoingressaddon enabled.What happened:
localkubeis consuming about 12% CPU constantly, even though no container is actively running. On host,docker-machine-driver-xhyveis consuming ~30% CPU.Nothing much is actually running:
Verified with
psthat no container is using that CPU: It's alllocalkube. According tops,localkubehas also allocated 10GB of virtual memory.Here is a gist with
psoutput, plus output from runningjournalctl-ffor half a minute or so. It mostly shows it asking about some containers that no longer exists.Nothing is being emitted to any container logs.
Problem re-occurs if I kill the process.
Here's an strace log (
-fF -s10000 -tt). Looks like it's constantly spawningiptables,iptables-saveandiptables-restore.What you expected to happen:
I expected
localkubenot to use much CPU when the system idle.How to reproduce it (as minimally and precisely as possible):
No idea. I did install Helm (the
tillercontroller), but I uninstalled it.The text was updated successfully, but these errors were encountered: