New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-proxy ipvs mode hangs after a few hours, needs manual restart #71071

Open
gjcarneiro opened this Issue Nov 15, 2018 · 126 comments

Comments

Projects
None yet
@gjcarneiro
Copy link

gjcarneiro commented Nov 15, 2018

What happened: upgraded from v1.11.0 to v1.12.2, now kube-proxy hangs, usually in less than a day

What you expected to happen: it should continue working

How to reproduce it (as minimally and precisely as possible): just have a normal kube-proxy running in ipvs mode. It gets stuck usually in less than a day.

Anything else we need to know?:

I've sent SIGABRT to the kube-proxy process to get a stack trace, and the only thing interesting is that when the bug happens goroutine 1 is reading from netlink, whereas when I send SIGABRT and the process is not stuck it is not reading from netlink.

Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: goroutine 1 [syscall]:
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.Syscall6(0x2d, 0x3, 0xc42081c000, 0x1000, 0x0, 0xc42083deb0, 0xc42083dea4, 0x40fb86, 0x7f2ab9788aa8, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.recvfrom(0x3, 0xc42081c000, 0x1000, 0x1000, 0x0, 0xc42083deb0, 0xc42083dea4, 0x101ffffffffffff, 0x0, 0x1000)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1665 +0xa6
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.Recvfrom(0x3, 0xc42081c000, 0x1000, 0x1000, 0x0, 0x1000, 0x0, 0x0, 0x16ad4c0, 0x1f4d508)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/syscall_unix.go:252 +0xaf
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xc4204d3440, 0x0, 0x0, 0x0, 0x16ad4c0, 0x1f4d508)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:613 +0x9b
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.execute(0xc4204d3440, 0xc42083e150, 0x0, 0x1, 0x2, 0xc420423740, 0x1, 0x2)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:219 +0xd3
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).doCmdwithResponse(0xc420573990, 0xc420990750, 0x0, 0xc42003d501, 0xa, 0x10, 0xc420990701, 0xc42083e230, 0x118dd6b)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:140 +0x20b
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).doCmd(0xc420573990, 0xc420990750, 0x0, 0x1, 0x60000020fc6e0, 0xc420990750)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:149 +0x48
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).NewService(0xc420573990, 0xc420990750, 0x0, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/ipvs.go:117 +0x43
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/ipvs.(*runner).AddVirtualServer(0xc42052a340, 0xc420b0e550, 0x1e, 0xc42083e3b8)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/ipvs/ipvs_linux.go:61 +0x60
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).syncService(0xc4201b3e00, 0xc420d53620, 0x21, 0xc420b0e550, 0xc420357901, 0x5, 0xc420d53620)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:1454 +0x635
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).syncProxyRules(0xc4201b3e00)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:803 +0x1081
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).(k8s.io/kubernetes/pkg/proxy/ipvs.syncProxyRules)-fm()
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:395 +0x2a
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/async.(*BoundedFrequencyRunner).tryRun(0xc42042e3f0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/async/bounded_frequency_runner.go:217 +0xb6
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/async.(*BoundedFrequencyRunner).Loop(0xc42042e3f0, 0xc4200b0120)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/async/bounded_frequency_runner.go:179 +0x211
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).SyncLoop(0xc4201b3e00)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:589 +0x4e
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.(*ProxyServer).Run(0xc4207f2000, 0xc4207f2000, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:568 +0x3dd
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.(*Options).Run(0xc4201082c0, 0xc42063e230, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:238 +0x69
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.NewProxyCommand.func1(0xc4200f2500, 0xc42063e230, 0x0, 0x7)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:360 +0x156
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200f2500, 0xc4200c4010, 0x7, 0x7, 0xc4200f2500, 0xc4200c4010)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200f2500, 0xc4205dc090, 0x16076a0, 0xc4207a7ee8)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200f2500, 0x1608b88, 0x20dcf40)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: main.main()

The last lines that it logged were:

Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392942    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.241:4059/TCP/10.125.2.26:4059
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392952    5389 graceful_termination.go:160] Trying to delete rs: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: E1115 07:48:52.392978    5389 graceful_termination.go:89] Try delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432" err: Failed to delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432", can't 
find the real server
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392985    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: E1115 07:48:52.392992    5389 graceful_termination.go:183] Try flush graceful termination list err
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393147    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.156:8125/TCP/10.125.2.21:8125
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393260    5389 graceful_termination.go:89] Try delete rs "10.128.33.156:8125/TCP/10.125.2.21:8125" err: Failed to delete rs "10.128.33.156:8125/TCP/10.125.2.21:8125", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393272    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.156:8125/TCP/10.125.2.21:8125
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393281    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.142:6379/TCP/10.125.6.13:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393310    5389 graceful_termination.go:89] Try delete rs "10.128.33.142:6379/TCP/10.125.6.13:6379" err: Failed to delete rs "10.128.33.142:6379/TCP/10.125.6.13:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393317    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.142:6379/TCP/10.125.6.13:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393327    5389 graceful_termination.go:160] Trying to delete rs: 10.128.34.237:6379/TCP/10.125.6.11:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393355    5389 graceful_termination.go:89] Try delete rs "10.128.34.237:6379/TCP/10.125.6.11:6379" err: Failed to delete rs "10.128.34.237:6379/TCP/10.125.6.11:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393362    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.34.237:6379/TCP/10.125.6.11:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393371    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.200:5432/TCP/10.125.6.27:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393397    5389 graceful_termination.go:89] Try delete rs "10.128.33.200:5432/TCP/10.125.6.27:6432" err: Failed to delete rs "10.128.33.200:5432/TCP/10.125.6.27:6432", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393404    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.200:5432/TCP/10.125.6.27:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393412    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.228:6379/TCP/10.125.2.63:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393440    5389 graceful_termination.go:89] Try delete rs "10.128.33.228:6379/TCP/10.125.2.63:6379" err: Failed to delete rs "10.128.33.228:6379/TCP/10.125.2.63:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393447    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.228:6379/TCP/10.125.2.63:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393455    5389 graceful_termination.go:160] Trying to delete rs: 10.128.34.247:5672/TCP/10.125.2.6:5672
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393482    5389 graceful_termination.go:89] Try delete rs "10.128.34.247:5672/TCP/10.125.2.6:5672" err: Failed to delete rs "10.128.34.247:5672/TCP/10.125.2.6:5672", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393489    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.34.247:5672/TCP/10.125.2.6:5672
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393497    5389 graceful_termination.go:160] Trying to delete rs: 10.128.35.47:4007/TCP/10.125.2.15:4007
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393523    5389 graceful_termination.go:89] Try delete rs "10.128.35.47:4007/TCP/10.125.2.15:4007" err: Failed to delete rs "10.128.35.47:4007/TCP/10.125.2.15:4007", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393530    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.35.47:4007/TCP/10.125.2.15:4007
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393538    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.241:4059/TCP/10.125.2.26:4059
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393563    5389 graceful_termination.go:89] Try delete rs "10.128.33.241:4059/TCP/10.125.2.26:4059" err: Failed to delete rs "10.128.33.241:4059/TCP/10.125.2.26:4059", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393571    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.241:4059/TCP/10.125.2.26:4059
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393578    5389 graceful_termination.go:160] Trying to delete rs: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393605    5389 graceful_termination.go:89] Try delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432" err: Failed to delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393613    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393621    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.252:6379/TCP/10.125.6.26:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393648    5389 graceful_termination.go:89] Try delete rs "10.128.33.252:6379/TCP/10.125.6.26:6379" err: Failed to delete rs "10.128.33.252:6379/TCP/10.125.6.26:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393656    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.252:6379/TCP/10.125.6.26:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393664    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.237:6379/TCP/10.125.6.34:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393690    5389 graceful_termination.go:89] Try delete rs "10.128.33.237:6379/TCP/10.125.6.34:6379" err: Failed to delete rs "10.128.33.237:6379/TCP/10.125.6.34:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393697    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.237:6379/TCP/10.125.6.34:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393704    5389 graceful_termination.go:183] Try flush graceful termination list err
Nov 15 07:50:52 hex-48b-pm kube-proxy[5389]: I1115 07:50:52.393838    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.156:8125/TCP/10.125.2.21:8125
Nov 15 07:50:52 hex-48b-pm kube-proxy[5389]: E1115 07:50:52.393940    5389 graceful_termination.go:89] Try delete rs "10.128.33.156:8125/TCP/10.125.2.21:8125" err: device or resource busy
Nov 15 07:50:52 hex-48b-pm kube-proxy[5389]: I1115 07:50:52.393957    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.142:6379/TCP/10.125.6.13:6379
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420734    5389 proxier.go:1496] Failed to list IPVS destinations, error: invalid argument
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420763    5389 proxier.go:809] Failed to sync endpoint for service: 10.128.33.156:8125/TCP, err: invalid argument
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420889    5389 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420903    5389 proxier.go:809] Failed to sync endpoint for service: 10.128.35.150:6379/TCP, err: Expected only one service obtained=0
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420950    5389 proxier.go:1455] Failed to add IPVS service "monitoring/extradata-inserter:": file exists
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420961    5389 proxier.go:812] Failed to sync service: 10.128.34.54:80/TCP, err: file exists
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.421886    5389 proxier.go:1544] Failed to add destination: 10.125.27.66:80, error: file exists

If I restart kube-proxy (manually, ugh), it all recovers and works fine. But it will get stuck again, eventually.

Environment:

/kind bug

@gjcarneiro

This comment has been minimized.

Copy link

gjcarneiro commented Nov 15, 2018

/sig network

@k8s-ci-robot k8s-ci-robot added sig/network and removed needs-sig labels Nov 15, 2018

@berlinsaint

This comment has been minimized.

Copy link

berlinsaint commented Nov 19, 2018

the same error. upgrade from 1.11.1. Everything become good except kube-proxy becomse not stable, some time pod can't access svc though their endpoints are ok .. And the same as the name resolution. so i think it may exist an important bug here.

@berlinsaint

This comment has been minimized.

Copy link

berlinsaint commented Nov 23, 2018

/area ipvs

@berlinsaint

This comment has been minimized.

Copy link

berlinsaint commented Nov 23, 2018

@gjcarneiro do you resolve this problem?

@gjcarneiro

This comment has been minimized.

Copy link

gjcarneiro commented Nov 23, 2018

Well, I switched the cluster back to the default (iptables) mode, appears to be pretty stable since then...

@berlinsaint

This comment has been minimized.

Copy link

berlinsaint commented Nov 26, 2018

@kubernetes/sig-network-bugs

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 26, 2018

@berlinsaint: Reiterating the mentions to trigger a notification:
@kubernetes/sig-network-bugs

In response to this:

@kubernetes/sig-network-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@annProg

This comment has been minimized.

Copy link

annProg commented Nov 26, 2018

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

@berlinsaint

This comment has been minimized.

Copy link

berlinsaint commented Nov 27, 2018

@gjcarneiro do you mean that in kernel 4.15 , it has the same problem?

@berlinsaint

This comment has been minimized.

Copy link

berlinsaint commented Nov 27, 2018

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊,,认识下。。少年

@annProg

This comment has been minimized.

Copy link

annProg commented Nov 27, 2018

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊,,认识下。。少年

点我头像就看到了

@aleks-mariusz

This comment has been minimized.

Copy link

aleks-mariusz commented Nov 27, 2018

I am also seeing this repeatedly in my centralized logging system:

  | Time | sys_name | log_lvl | tag | log_msg
  | November 27th 2018, 17:56:23.501 | k8lab2bs | 6 | kube.kube-system.kube-proxy | lw: remote out of the list: 10.12.12.1:443/TCP/10.12.100.133:6443
  | November 27th 2018, 17:56:23.500 | k8lab2bs | 6 | kube.kube-system.kube-proxy | Deleting rs: 10.12.12.1:443/TCP/10.12.100.133:6443
  | November 27th 2018, 17:56:23.497 | k8lab2bs | 6 | kube.kube-system.kube-proxy | Trying to delete rs: 10.12.12.1:443/TCP/10.12.100.133:6443

I'm running v1.12.3 (fresh install, not an upgrade).

@Lion-Wei

This comment has been minimized.

Copy link
Contributor

Lion-Wei commented Nov 29, 2018

Hi, all. Thanks for the report, I'll make a research and try to figure out what happened, any progress I'll report here.

@gjcarneiro

This comment has been minimized.

Copy link

gjcarneiro commented Nov 29, 2018

Besides what happened, which is good to find out, I don't feel confident about IPVS until there is code in place that detects and resets a broken netlink socket. Because from what I can guess from the logs and traces it is that the netlink socket gets into a broken state and the task that reads/writes from/to it will hang forever. I think kube-proxy needs to have code that detects this problem, and if it happens create a new netlink socket from scratch, and maybe restart the task that handles it. But, no, I'm not volunteering, I don't even know Go, sorry.

@m1093782566

This comment has been minimized.

Copy link
Member

m1093782566 commented Nov 29, 2018

Because from what I can guess from the logs and traces it is that the netlink socket gets into a broken state and the task that reads/writes from/to it will hang forever.

We upgraded the go netlink version some days ago for supporting graceful termination. The netlink library was running pretty stable before. I am not sure if it's the cause.

@berlinsaint

This comment has been minimized.

Copy link

berlinsaint commented Nov 30, 2018

1.12.3 with the same error.
After filter the INFO log,
execute kubectl -n kube-system logs kube-proxy-29zh8|egrep ^E; kubectl -n kube-system logs kube-proxy-2pqdn|egrep ^E; kubectl -n kube-system logs kube-proxy-4xw8q|egrep ^E; kubectl -n kube-system logs kube-proxy-6j4bc|egrep ^E; kubectl -n kube-system logs kube-proxy-brbjb|egrep ^E; kubectl -n kube-system logs kube-proxy-r6cg2|egrep ^E; kubectl -n kube-system logs kube-proxy-rpl6t|egrep ^E;
and all the proxy node log is

E1129 01:01:10.862180       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 21:09:11.422553       1 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
E1129 21:09:11.422699       1 proxier.go:1116] Failed to sync endpoint for service: 172.17.0.1:31800/TCP, err: Expected only one service obtained=0
E1129 01:01:11.618125       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:11.905626       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:01:41.771596       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:02:11.923872       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:02:42.060658       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:03:12.326762       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:03:42.718417       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:04:12.859933       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:04:42.963527       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:05:13.078486       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:05:43.185141       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:06:13.303090       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:06:43.408582       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:07:13.550358       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:07:43.686235       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:08:13.786326       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:08:43.981764       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:09:14.188053       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:09:44.316316       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:10:14.440466       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:10:44.550092       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:11:14.658190       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:11:44.765396       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:12:14.975724       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:12:45.085632       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:13:15.317679       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:13:45.432469       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:14:15.689681       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:14:45.813565       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:15:33.434931       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:16:03.578017       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:16:33.709744       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:17:03.823325       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:17:33.935191       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:18:04.053183       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:18:34.248994       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:04.375946       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:34.552779       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:36.596182       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:47.433846       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:20:17.534206       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:20:47.682350       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:21:17.825175       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:21:47.942548       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:22:18.055820       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:22:48.149494       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:23:18.564985       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:23:48.678462       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:24:18.836217       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:24:48.991344       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:25:19.091032       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:25:49.187178       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:26:19.301770       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:26:49.425812       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:27:19.606711       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:27:49.721224       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:28:20.109117       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:28:50.211574       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:29:20.405007       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:29:50.507766       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:30:20.605471       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:30:50.702798       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:31:20.807400       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:31:50.903049       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:32:21.005618       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:32:51.106378       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:33:21.224447       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:33:51.354916       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:34:21.467898       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:34:51.583216       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:35:21.691099       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:35:51.797151       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:36:21.921057       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:36:52.083821       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:02.445737       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:07.427811       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:37.546362       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:38:07.651769       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:38:37.797858       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:39:08.064112       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:39:38.181089       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:40:08.282892       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:40:38.390167       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:41:08.508549       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:41:38.617921       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:42:08.728384       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:42:38.844074       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:43:08.973517       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:43:39.092095       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:44:09.202329       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:44:39.363721       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:45:09.476240       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:45:39.764944       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:46:09.894587       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:46:40.013388       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:47:10.127894       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:47:40.236119       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:48:10.397111       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:48:40.600084       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:49:10.877034       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:49:41.023593       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:50:11.149894       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:50:41.262320       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:51:11.410407       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:51:41.547057       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:52:11.609974       1 proxier.go:1544] Failed to add destination: 172.31.200.9:443, error: file exists
E1129 01:52:11.610088       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: invalid argument
E1129 01:52:11.616690       1 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
E1129 01:52:11.616767       1 proxier.go:1116] Failed to sync endpoint for service: 172.18.0.1:31645/TCP, err: Expected only one service obtained=0
E1129 01:01:12.181112       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:22.278474       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:21.459378       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:17.442316       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 20:10:17.955872       1 proxier.go:1455] Failed to add IPVS service "dev-fffrf/mysql-testportal:mysql": file exists
E1129 20:10:17.955863       1 graceful_termination.go:89] Try delete rs "10.100.217.68:15004/TCP/192.100.143.225:15004" err: Failed to delete rs "10.100.217.68:15004/TCP/192.100.143.225:15004", can't find the real server
E1129 20:10:17.955892       1 proxier.go:1119] Failed to sync service: 172.17.0.1:31645/TCP, err: file exists
E1129 20:10:17.956412       1 proxier.go:1485] Failed to get IPVS service, error: invalid argument
E1129 20:10:17.956417       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: Failed to delete rs "10.96.0.1:443/TCP/172.31.10.38:6443", can't find the real server
E1129 20:10:17.956426       1 proxier.go:1116] Failed to sync endpoint for service: 172.31.200.67:31645/TCP, err: invalid argument
E1129 20:10:17.956448       1 graceful_termination.go:89] Try delete rs "10.101.1.98:5672/TCP/192.100.29.180:5672" err: invalid argument
E1129 20:10:17.956547       1 graceful_termination.go:183] Try flush graceful termination list err
E1129 01:01:17.484679       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 02:52:17.498135       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: invalid argument
E1129 02:52:17.498396       1 graceful_termination.go:183] Try flush graceful termination list err
E1129 06:28:17.607394       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: device or resource busy
E1129 06:28:17.607513       1 graceful_termination.go:183] Try flush graceful termination list err
E1129 10:02:17.718458       1 proxier.go:1544] Failed to add destination: 192.100.183.139:3306, error: file exists
E1129 10:02:17.718606       1 proxier.go:1544] Failed to add destination: 192.100.183.152:8080, error: invalid argument```

@berlinsaint

This comment has been minimized.

Copy link

berlinsaint commented Nov 30, 2018

BTW. from above my logs , kube-proxy seems like still have many problems with iptables-restore . all my node are UBUNTU 16.04 which have iptable's version 1.6.0.
@m1093782566 @Lion-Wei

@rtreffer

This comment has been minimized.

Copy link
Contributor

rtreffer commented Dec 5, 2018

We see this happening when upgrading from 1.12.1. it must thus be in the delta of 1.12.1 and 1.12.2.

@m1093782566

This comment has been minimized.

Copy link
Member

m1093782566 commented Dec 5, 2018

The issue is graceful termination related and I think we have fixed the bug in v1.13.1.

@matthiasr

This comment has been minimized.

Copy link
Member

matthiasr commented Dec 5, 2018

@m1093782566 will the fix be backported to the 1.12 line? right now this is blocking the 1.12.3 security fix for us.

@adidenko

This comment has been minimized.

Copy link

adidenko commented Dec 6, 2018

Got the same with upgrade from 1.11.2 to 1.11.5.

@ChiefAlexander

This comment has been minimized.

Copy link

ChiefAlexander commented Dec 6, 2018

@m1093782566 can you provide more information on the fix? I would be interested to look over the commit.

@Quentin-M

This comment has been minimized.

Copy link
Contributor

Quentin-M commented Dec 7, 2018

This is pretty insane, we got bitten really hard earlier today.

@Quentin-M

This comment has been minimized.

Copy link
Contributor

Quentin-M commented Dec 7, 2018

We see this happening when upgrading from 1.12.1. it must thus be in the delta of 1.12.1 and 1.12.2.

Got the same with upgrade from 1.11.2 to 1.11.5.

Was the bug chery-picked?

@dims dims reopened this Dec 27, 2018

@VincentFF

This comment has been minimized.

Copy link

VincentFF commented Dec 28, 2018

I have the same problem of Kube-proxy ipvs, version 1.12.3.
The real servers are wrong and Kube-proxy can't refresh the ipvs rules.

Here are last logs and it stuck here, didn't print new logs anymore.

27 23:31:24.113927       1 graceful_termination.go:93] lw: remote out of the list: 10.96.0.1:443/TCP/10.6.40.32:6443
E1227 23:31:24.113935       1 graceful_termination.go:183] Try flush graceful termination list err
I1227 23:32:24.114182       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.32:6443
E1227 23:32:24.114316       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/10.6.40.32:6443" err: Failed to delete rs "10.96.0.1:443/TCP/10.6.40.32:6443", can't find the real server
I1227 23:32:24.114332       1 graceful_termination.go:93] lw: remote out of the list: 10.96.0.1:443/TCP/10.6.40.32:6443
I1227 23:32:24.114348       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.33:6443
I1227 23:32:24.114397       1 graceful_termination.go:160] Trying to delete rs: 10.6.40.25:32444/TCP/10.244.20.41:32444
E1227 23:32:24.114446       1 graceful_termination.go:183] Try flush graceful termination list err
I1227 23:33:24.114853       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.32:6443
E1227 23:33:24.115043       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/10.6.40.32:6443" err: Failed to delete rs "10.96.0.1:443/TCP/10.6.40.32:6443", can't find the real server
I1227 23:33:24.115106       1 graceful_termination.go:93] lw: remote out of the list: 10.96.0.1:443/TCP/10.6.40.32:6443
I1227 23:33:24.115128       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.33:6443
@lbernail

This comment has been minimized.

Copy link
Contributor

lbernail commented Dec 28, 2018

@VincentFF : there are two issues:

  • failed deletions: fixed in 1.12.4
  • kube-proxy getting stuck due to a deadlock (which is the issue you are describing): hopefully fixed with the PR from yesterday. We are waiting for feedback from @emptywee and @bjornryden who are currently testing the patch. As soon as we have confirmation that the issue is not present anymore I'll create the cherrypicks for 1.13, 1.12 and 1.11 (if the patch works, it will probably be merged in all branches at the beginning of next week and will be part of the next patch releases)
@matthiasr

This comment has been minimized.

Copy link
Member

matthiasr commented Dec 28, 2018

I'm also re-running my experiment #71071 (comment) again, with the patch back-ported onto 1.12.4. Will report back!

@emptywee

This comment has been minimized.

Copy link

emptywee commented Dec 28, 2018

We haven't had any issues with accessing services while kube-proxy was(and is) running in the IPVS mode and we've had quite a few deployments and changes to the services. So far I'd say the patch is working well for us.

@lbernail Great job on fixing it! Much appreciated!

@lbernail

This comment has been minimized.

Copy link
Contributor

lbernail commented Dec 28, 2018

Ok let's wait until monday so we also get feedback from @matthiasr and @bjornryden (I want to avoid rushing the cherry-picks and make sure that we have a clean fix this time)

@VincentFF

This comment has been minimized.

Copy link

VincentFF commented Dec 28, 2018

@lbernail Good news, thanks to your reply.

@matthiasr

This comment has been minimized.

Copy link
Member

matthiasr commented Dec 28, 2018

No issues after 12+ hours on 1.12.4 with DataDog@9121f01 – without that I got a lockup in 1.5 hours, so this looks very good!

@VincentFF

This comment has been minimized.

Copy link

VincentFF commented Dec 29, 2018

@lbernail I have another question, please. If the next patch is version 1.12.5?
If I just upgrade kube-proxy to 1.12.5, other units(apiserver, kubelet, controller-manager, scheduler ...) keep 1.12.3, is it ok? Or I have to upgrade all units?

@lbernail

This comment has been minimized.

Copy link
Contributor

lbernail commented Dec 29, 2018

@VincentFF Hopefully it will make it to 1.12.5 yes. You can run kube-proxy 1.12.5 with other units running 1.12.3, no problem

@matthiasr

This comment has been minimized.

Copy link
Member

matthiasr commented Dec 29, 2018

Update: after 24 hours, still no issues, I'm stopping my experiment now & consider this a full success. Looking forward to 1.12.5!

@lbernail

This comment has been minimized.

Copy link
Contributor

lbernail commented Dec 29, 2018

Ok this is very good news. I'm going to create the cherry-pick for all branches in order to get the fix in official releases as soon as possible.
Many thanks for your involvement in helping solve the issue and testing!

@lbernail

This comment has been minimized.

Copy link
Contributor

lbernail commented Dec 29, 2018

Cherry-picks:
1.13: #72426
1.12: #72429
1.11: #72430

@bjornryden

This comment has been minimized.

Copy link

bjornryden commented Dec 31, 2018

I can join the choir of happy testers. 4 days of no trouble happening.

Thanks a lot to all involved, especially @lbernail. Happy New Year!

@VincentFF

This comment has been minimized.

Copy link

VincentFF commented Jan 1, 2019

I build it and deploy to test env, over 24 hours have no issues. Happy new year.

@daigong

This comment has been minimized.

Copy link

daigong commented Jan 1, 2019

Hello, My env is
Ubuntu 16.04.4 LTS
Kubernetes version (use kubectl version): v1.13.0
Cloud provider or hardware configuration: bare metal, 64 bit linux
Kernel 4.4.0-75-generic

same bug:

I0101 22:20:09.742898 267576 graceful_termination.go:160] Trying to delete rs: 10.200.198.70:9090/TCP/172.200.101.32:9090
E0101 22:20:09.742975 267576 proxier.go:1519] Failed to get IPVS service, error: Expected only one service obtained=0
E0101 22:20:09.743036 267576 proxier.go:843] Failed to sync endpoint for service: 10.200.36.192:80/TCP, err: Expected only one service obtained=0

I need restart kube-proxy fix it

@FrostyLeaf

This comment has been minimized.

Copy link

FrostyLeaf commented Jan 2, 2019

@daigong v1.13.1 have fixed this. In my case, no issues after more than one week.

@lbernail

This comment has been minimized.

Copy link
Contributor

lbernail commented Jan 2, 2019

@daigong : the full fix will actually be in 1.13.2 (it depends on how fast #72426 gets merged)

@daigong

This comment has been minimized.

Copy link

daigong commented Jan 2, 2019

@lbernail @FrostyLeaf
thanks , I use #72426 to test this bug

Anything is ok now

rjosephwright added a commit to cloudboss/keights that referenced this issue Jan 6, 2019

Revert "Switch to iptables mode for kube-proxy"
This reverts commit 4ae9635,
since v1.11.6 is now released with a fix for
kubernetes/kubernetes#71071.
@lbernail

This comment has been minimized.

Copy link
Contributor

lbernail commented Jan 7, 2019

The PR has been merged in all release branches, so it will be present in:

  • 1.11.7
  • 1.12.5
  • 1.13.2
@grpubr

This comment has been minimized.

Copy link

grpubr commented Jan 13, 2019

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊,,认识下。。少年

一样一样

@Deshke

This comment has been minimized.

Copy link

Deshke commented Jan 15, 2019

any chance for a new build in v1.11.x ?

@verwilst

This comment has been minimized.

Copy link

verwilst commented Jan 15, 2019

I've been refreshing the releases pages pretty much every hour for the last couple of days in the hope 1.12.5 will appear so i can patch this bug on multiple clusters. :-)

@emptywee

This comment has been minimized.

Copy link

emptywee commented Jan 15, 2019

@verwilst haha, same, can't wait for v1.11.7 to get released to upgrade current prod clusters. And v1.12.5 too to start upgrading in lower envs.

@mrrandrade

This comment has been minimized.

Copy link

mrrandrade commented Jan 17, 2019

I've been refreshing the releases pages pretty much every hour for the last couple of days in the hope 1.12.5 will appear so i can patch this bug on multiple clusters. :-)

Its here :-P

@emptywee

This comment has been minimized.

Copy link

emptywee commented Jan 17, 2019

I've been refreshing the releases pages pretty much every hour for the last couple of days in the hope 1.12.5 will appear so i can patch this bug on multiple clusters. :-)

Its here :-P

Why is it not there for 1.11? :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment