kube-proxy ipvs mode hangs after a few hours, needs manual restart #71071

gjcarneiro · Nov 15, 2018

What happened: upgraded from v1.11.0 to v1.12.2, now kube-proxy hangs, usually in less than a day

What you expected to happen: it should continue working

How to reproduce it (as minimally and precisely as possible): just have a normal kube-proxy running in ipvs mode. It gets stuck usually in less than a day.

Anything else we need to know?:

I've sent SIGABRT to the kube-proxy process to get a stack trace, and the only thing interesting is that when the bug happens goroutine 1 is reading from netlink, whereas when I send SIGABRT and the process is not stuck it is not reading from netlink.

Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: goroutine 1 [syscall]:
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.Syscall6(0x2d, 0x3, 0xc42081c000, 0x1000, 0x0, 0xc42083deb0, 0xc42083dea4, 0x40fb86, 0x7f2ab9788aa8, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.recvfrom(0x3, 0xc42081c000, 0x1000, 0x1000, 0x0, 0xc42083deb0, 0xc42083dea4, 0x101ffffffffffff, 0x0, 0x1000)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1665 +0xa6
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.Recvfrom(0x3, 0xc42081c000, 0x1000, 0x1000, 0x0, 0x1000, 0x0, 0x0, 0x16ad4c0, 0x1f4d508)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/syscall_unix.go:252 +0xaf
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xc4204d3440, 0x0, 0x0, 0x0, 0x16ad4c0, 0x1f4d508)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:613 +0x9b
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.execute(0xc4204d3440, 0xc42083e150, 0x0, 0x1, 0x2, 0xc420423740, 0x1, 0x2)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:219 +0xd3
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).doCmdwithResponse(0xc420573990, 0xc420990750, 0x0, 0xc42003d501, 0xa, 0x10, 0xc420990701, 0xc42083e230, 0x118dd6b)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:140 +0x20b
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).doCmd(0xc420573990, 0xc420990750, 0x0, 0x1, 0x60000020fc6e0, 0xc420990750)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:149 +0x48
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).NewService(0xc420573990, 0xc420990750, 0x0, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/ipvs.go:117 +0x43
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/ipvs.(*runner).AddVirtualServer(0xc42052a340, 0xc420b0e550, 0x1e, 0xc42083e3b8)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/ipvs/ipvs_linux.go:61 +0x60
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).syncService(0xc4201b3e00, 0xc420d53620, 0x21, 0xc420b0e550, 0xc420357901, 0x5, 0xc420d53620)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:1454 +0x635
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).syncProxyRules(0xc4201b3e00)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:803 +0x1081
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).(k8s.io/kubernetes/pkg/proxy/ipvs.syncProxyRules)-fm()
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:395 +0x2a
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/async.(*BoundedFrequencyRunner).tryRun(0xc42042e3f0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/async/bounded_frequency_runner.go:217 +0xb6
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/async.(*BoundedFrequencyRunner).Loop(0xc42042e3f0, 0xc4200b0120)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/async/bounded_frequency_runner.go:179 +0x211
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).SyncLoop(0xc4201b3e00)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:589 +0x4e
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.(*ProxyServer).Run(0xc4207f2000, 0xc4207f2000, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:568 +0x3dd
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.(*Options).Run(0xc4201082c0, 0xc42063e230, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:238 +0x69
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.NewProxyCommand.func1(0xc4200f2500, 0xc42063e230, 0x0, 0x7)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:360 +0x156
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200f2500, 0xc4200c4010, 0x7, 0x7, 0xc4200f2500, 0xc4200c4010)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200f2500, 0xc4205dc090, 0x16076a0, 0xc4207a7ee8)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200f2500, 0x1608b88, 0x20dcf40)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: main.main()

The last lines that it logged were:

Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392942    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.241:4059/TCP/10.125.2.26:4059
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392952    5389 graceful_termination.go:160] Trying to delete rs: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: E1115 07:48:52.392978    5389 graceful_termination.go:89] Try delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432" err: Failed to delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432", can't 
find the real server
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392985    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: E1115 07:48:52.392992    5389 graceful_termination.go:183] Try flush graceful termination list err
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393147    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.156:8125/TCP/10.125.2.21:8125
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393260    5389 graceful_termination.go:89] Try delete rs "10.128.33.156:8125/TCP/10.125.2.21:8125" err: Failed to delete rs "10.128.33.156:8125/TCP/10.125.2.21:8125", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393272    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.156:8125/TCP/10.125.2.21:8125
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393281    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.142:6379/TCP/10.125.6.13:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393310    5389 graceful_termination.go:89] Try delete rs "10.128.33.142:6379/TCP/10.125.6.13:6379" err: Failed to delete rs "10.128.33.142:6379/TCP/10.125.6.13:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393317    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.142:6379/TCP/10.125.6.13:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393327    5389 graceful_termination.go:160] Trying to delete rs: 10.128.34.237:6379/TCP/10.125.6.11:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393355    5389 graceful_termination.go:89] Try delete rs "10.128.34.237:6379/TCP/10.125.6.11:6379" err: Failed to delete rs "10.128.34.237:6379/TCP/10.125.6.11:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393362    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.34.237:6379/TCP/10.125.6.11:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393371    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.200:5432/TCP/10.125.6.27:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393397    5389 graceful_termination.go:89] Try delete rs "10.128.33.200:5432/TCP/10.125.6.27:6432" err: Failed to delete rs "10.128.33.200:5432/TCP/10.125.6.27:6432", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393404    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.200:5432/TCP/10.125.6.27:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393412    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.228:6379/TCP/10.125.2.63:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393440    5389 graceful_termination.go:89] Try delete rs "10.128.33.228:6379/TCP/10.125.2.63:6379" err: Failed to delete rs "10.128.33.228:6379/TCP/10.125.2.63:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393447    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.228:6379/TCP/10.125.2.63:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393455    5389 graceful_termination.go:160] Trying to delete rs: 10.128.34.247:5672/TCP/10.125.2.6:5672
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393482    5389 graceful_termination.go:89] Try delete rs "10.128.34.247:5672/TCP/10.125.2.6:5672" err: Failed to delete rs "10.128.34.247:5672/TCP/10.125.2.6:5672", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393489    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.34.247:5672/TCP/10.125.2.6:5672
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393497    5389 graceful_termination.go:160] Trying to delete rs: 10.128.35.47:4007/TCP/10.125.2.15:4007
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393523    5389 graceful_termination.go:89] Try delete rs "10.128.35.47:4007/TCP/10.125.2.15:4007" err: Failed to delete rs "10.128.35.47:4007/TCP/10.125.2.15:4007", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393530    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.35.47:4007/TCP/10.125.2.15:4007
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393538    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.241:4059/TCP/10.125.2.26:4059
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393563    5389 graceful_termination.go:89] Try delete rs "10.128.33.241:4059/TCP/10.125.2.26:4059" err: Failed to delete rs "10.128.33.241:4059/TCP/10.125.2.26:4059", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393571    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.241:4059/TCP/10.125.2.26:4059
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393578    5389 graceful_termination.go:160] Trying to delete rs: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393605    5389 graceful_termination.go:89] Try delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432" err: Failed to delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393613    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393621    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.252:6379/TCP/10.125.6.26:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393648    5389 graceful_termination.go:89] Try delete rs "10.128.33.252:6379/TCP/10.125.6.26:6379" err: Failed to delete rs "10.128.33.252:6379/TCP/10.125.6.26:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393656    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.252:6379/TCP/10.125.6.26:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393664    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.237:6379/TCP/10.125.6.34:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393690    5389 graceful_termination.go:89] Try delete rs "10.128.33.237:6379/TCP/10.125.6.34:6379" err: Failed to delete rs "10.128.33.237:6379/TCP/10.125.6.34:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393697    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.237:6379/TCP/10.125.6.34:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393704    5389 graceful_termination.go:183] Try flush graceful termination list err
Nov 15 07:50:52 hex-48b-pm kube-proxy[5389]: I1115 07:50:52.393838    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.156:8125/TCP/10.125.2.21:8125
Nov 15 07:50:52 hex-48b-pm kube-proxy[5389]: E1115 07:50:52.393940    5389 graceful_termination.go:89] Try delete rs "10.128.33.156:8125/TCP/10.125.2.21:8125" err: device or resource busy
Nov 15 07:50:52 hex-48b-pm kube-proxy[5389]: I1115 07:50:52.393957    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.142:6379/TCP/10.125.6.13:6379
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420734    5389 proxier.go:1496] Failed to list IPVS destinations, error: invalid argument
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420763    5389 proxier.go:809] Failed to sync endpoint for service: 10.128.33.156:8125/TCP, err: invalid argument
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420889    5389 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420903    5389 proxier.go:809] Failed to sync endpoint for service: 10.128.35.150:6379/TCP, err: Expected only one service obtained=0
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420950    5389 proxier.go:1455] Failed to add IPVS service "monitoring/extradata-inserter:": file exists
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420961    5389 proxier.go:812] Failed to sync service: 10.128.34.54:80/TCP, err: file exists
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.421886    5389 proxier.go:1544] Failed to add destination: 10.125.27.66:80, error: file exists

If I restart kube-proxy (manually, ugh), it all recovers and works fine. But it will get stuck again, eventually.

Environment:

Kubernetes version (use kubectl version): v1.12.2
Cloud provider or hardware configuration: bare metal, 64 bit linux
OS (e.g. from /etc/os-release): Ubuntu 16.04.5 LTS
Kernel (e.g. uname -a): 4.15.0-39-generic (but get it on the original 4.4 kernel as well)
Install tools: binary from https://storage.googleapis.com/kubernetes-release/release/v1.12.2/kubernetes-server-linux-amd64.tar.gz
Others:

/kind bug

gjcarneiro · Nov 15, 2018

/sig network

berlinsaint · Nov 19, 2018

the same error. upgrade from 1.11.1. Everything become good except kube-proxy becomse not stable, some time pod can't access svc though their endpoints are ok .. And the same as the name resolution. so i think it may exist an important bug here.

berlinsaint · Nov 23, 2018

/area ipvs

berlinsaint · Nov 23, 2018

@gjcarneiro do you resolve this problem?

gjcarneiro · Nov 23, 2018

Well, I switched the cluster back to the default (iptables) mode, appears to be pretty stable since then...

berlinsaint · Nov 26, 2018

@kubernetes/sig-network-bugs

k8s-ci-robot · Nov 26, 2018

@berlinsaint: Reiterating the mentions to trigger a notification:
@kubernetes/sig-network-bugs

In response to this:

@kubernetes/sig-network-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

annProg · Nov 26, 2018

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

berlinsaint · Nov 27, 2018

@gjcarneiro do you mean that in kernel 4.15 , it has the same problem?

berlinsaint · Nov 27, 2018

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊，，认识下。。少年

annProg · Nov 27, 2018

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊，，认识下。。少年

点我头像就看到了

aleks-mariusz · Nov 27, 2018

I am also seeing this repeatedly in my centralized logging system:

  | Time | sys_name | log_lvl | tag | log_msg
  | November 27th 2018, 17:56:23.501 | k8lab2bs | 6 | kube.kube-system.kube-proxy | lw: remote out of the list: 10.12.12.1:443/TCP/10.12.100.133:6443
  | November 27th 2018, 17:56:23.500 | k8lab2bs | 6 | kube.kube-system.kube-proxy | Deleting rs: 10.12.12.1:443/TCP/10.12.100.133:6443
  | November 27th 2018, 17:56:23.497 | k8lab2bs | 6 | kube.kube-system.kube-proxy | Trying to delete rs: 10.12.12.1:443/TCP/10.12.100.133:6443

I'm running v1.12.3 (fresh install, not an upgrade).

Lion-Wei · Nov 29, 2018

Hi, all. Thanks for the report, I'll make a research and try to figure out what happened, any progress I'll report here.

gjcarneiro · Nov 29, 2018

Besides what happened, which is good to find out, I don't feel confident about IPVS until there is code in place that detects and resets a broken netlink socket. Because from what I can guess from the logs and traces it is that the netlink socket gets into a broken state and the task that reads/writes from/to it will hang forever. I think kube-proxy needs to have code that detects this problem, and if it happens create a new netlink socket from scratch, and maybe restart the task that handles it. But, no, I'm not volunteering, I don't even know Go, sorry.

m1093782566 · Nov 29, 2018

Because from what I can guess from the logs and traces it is that the netlink socket gets into a broken state and the task that reads/writes from/to it will hang forever.

We upgraded the go netlink version some days ago for supporting graceful termination. The netlink library was running pretty stable before. I am not sure if it's the cause.

berlinsaint · Nov 30, 2018

1.12.3 with the same error.
After filter the INFO log,
execute kubectl -n kube-system logs kube-proxy-29zh8|egrep ^E; kubectl -n kube-system logs kube-proxy-2pqdn|egrep ^E; kubectl -n kube-system logs kube-proxy-4xw8q|egrep ^E; kubectl -n kube-system logs kube-proxy-6j4bc|egrep ^E; kubectl -n kube-system logs kube-proxy-brbjb|egrep ^E; kubectl -n kube-system logs kube-proxy-r6cg2|egrep ^E; kubectl -n kube-system logs kube-proxy-rpl6t|egrep ^E;
and all the proxy node log is

E1129 01:01:10.862180       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 21:09:11.422553       1 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
E1129 21:09:11.422699       1 proxier.go:1116] Failed to sync endpoint for service: 172.17.0.1:31800/TCP, err: Expected only one service obtained=0
E1129 01:01:11.618125       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:11.905626       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:01:41.771596       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:02:11.923872       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:02:42.060658       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:03:12.326762       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:03:42.718417       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:04:12.859933       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:04:42.963527       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:05:13.078486       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:05:43.185141       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:06:13.303090       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:06:43.408582       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:07:13.550358       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:07:43.686235       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:08:13.786326       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:08:43.981764       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:09:14.188053       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:09:44.316316       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:10:14.440466       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:10:44.550092       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:11:14.658190       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:11:44.765396       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:12:14.975724       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:12:45.085632       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:13:15.317679       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:13:45.432469       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:14:15.689681       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:14:45.813565       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:15:33.434931       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:16:03.578017       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:16:33.709744       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:17:03.823325       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:17:33.935191       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:18:04.053183       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:18:34.248994       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:04.375946       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:34.552779       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:36.596182       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:47.433846       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:20:17.534206       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:20:47.682350       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:21:17.825175       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:21:47.942548       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:22:18.055820       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:22:48.149494       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:23:18.564985       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:23:48.678462       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:24:18.836217       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:24:48.991344       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:25:19.091032       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:25:49.187178       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:26:19.301770       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:26:49.425812       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:27:19.606711       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:27:49.721224       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:28:20.109117       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:28:50.211574       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:29:20.405007       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:29:50.507766       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:30:20.605471       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:30:50.702798       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:31:20.807400       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:31:50.903049       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:32:21.005618       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:32:51.106378       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:33:21.224447       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:33:51.354916       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:34:21.467898       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:34:51.583216       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:35:21.691099       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:35:51.797151       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:36:21.921057       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:36:52.083821       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:02.445737       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:07.427811       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:37.546362       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:38:07.651769       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:38:37.797858       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:39:08.064112       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:39:38.181089       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:40:08.282892       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:40:38.390167       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:41:08.508549       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:41:38.617921       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:42:08.728384       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:42:38.844074       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:43:08.973517       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:43:39.092095       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:44:09.202329       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:44:39.363721       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:45:09.476240       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:45:39.764944       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:46:09.894587       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:46:40.013388       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:47:10.127894       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:47:40.236119       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:48:10.397111       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:48:40.600084       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:49:10.877034       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:49:41.023593       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:50:11.149894       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:50:41.262320       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:51:11.410407       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:51:41.547057       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:52:11.609974       1 proxier.go:1544] Failed to add destination: 172.31.200.9:443, error: file exists
E1129 01:52:11.610088       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: invalid argument
E1129 01:52:11.616690       1 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
E1129 01:52:11.616767       1 proxier.go:1116] Failed to sync endpoint for service: 172.18.0.1:31645/TCP, err: Expected only one service obtained=0
E1129 01:01:12.181112       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:22.278474       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:21.459378       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:17.442316       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 20:10:17.955872       1 proxier.go:1455] Failed to add IPVS service "dev-fffrf/mysql-testportal:mysql": file exists
E1129 20:10:17.955863       1 graceful_termination.go:89] Try delete rs "10.100.217.68:15004/TCP/192.100.143.225:15004" err: Failed to delete rs "10.100.217.68:15004/TCP/192.100.143.225:15004", can't find the real server
E1129 20:10:17.955892       1 proxier.go:1119] Failed to sync service: 172.17.0.1:31645/TCP, err: file exists
E1129 20:10:17.956412       1 proxier.go:1485] Failed to get IPVS service, error: invalid argument
E1129 20:10:17.956417       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: Failed to delete rs "10.96.0.1:443/TCP/172.31.10.38:6443", can't find the real server
E1129 20:10:17.956426       1 proxier.go:1116] Failed to sync endpoint for service: 172.31.200.67:31645/TCP, err: invalid argument
E1129 20:10:17.956448       1 graceful_termination.go:89] Try delete rs "10.101.1.98:5672/TCP/192.100.29.180:5672" err: invalid argument
E1129 20:10:17.956547       1 graceful_termination.go:183] Try flush graceful termination list err
E1129 01:01:17.484679       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 02:52:17.498135       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: invalid argument
E1129 02:52:17.498396       1 graceful_termination.go:183] Try flush graceful termination list err
E1129 06:28:17.607394       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: device or resource busy
E1129 06:28:17.607513       1 graceful_termination.go:183] Try flush graceful termination list err
E1129 10:02:17.718458       1 proxier.go:1544] Failed to add destination: 192.100.183.139:3306, error: file exists
E1129 10:02:17.718606       1 proxier.go:1544] Failed to add destination: 192.100.183.152:8080, error: invalid argument```

berlinsaint · Nov 30, 2018

BTW. from above my logs , kube-proxy seems like still have many problems with iptables-restore . all my node are UBUNTU 16.04 which have iptable's version 1.6.0.
@m1093782566 @Lion-Wei

rtreffer · Dec 5, 2018

We see this happening when upgrading from 1.12.1. it must thus be in the delta of 1.12.1 and 1.12.2.

m1093782566 · Dec 5, 2018

The issue is graceful termination related and I think we have fixed the bug in v1.13.1.

matthiasr · Dec 5, 2018

@m1093782566 will the fix be backported to the 1.12 line? right now this is blocking the 1.12.3 security fix for us.

adidenko · Dec 6, 2018

Got the same with upgrade from 1.11.2 to 1.11.5.

ChiefAlexander · Dec 6, 2018

@m1093782566 can you provide more information on the fix? I would be interested to look over the commit.

Quentin-M · Dec 7, 2018

This is pretty insane, we got bitten really hard earlier today.

Quentin-M · Dec 7, 2018

We see this happening when upgrading from 1.12.1. it must thus be in the delta of 1.12.1 and 1.12.2.

Got the same with upgrade from 1.11.2 to 1.11.5.

Was the bug chery-picked?

VincentFF · Dec 28, 2018

I have the same problem of Kube-proxy ipvs, version 1.12.3.
The real servers are wrong and Kube-proxy can't refresh the ipvs rules.

Here are last logs and it stuck here, didn't print new logs anymore.

27 23:31:24.113927       1 graceful_termination.go:93] lw: remote out of the list: 10.96.0.1:443/TCP/10.6.40.32:6443
E1227 23:31:24.113935       1 graceful_termination.go:183] Try flush graceful termination list err
I1227 23:32:24.114182       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.32:6443
E1227 23:32:24.114316       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/10.6.40.32:6443" err: Failed to delete rs "10.96.0.1:443/TCP/10.6.40.32:6443", can't find the real server
I1227 23:32:24.114332       1 graceful_termination.go:93] lw: remote out of the list: 10.96.0.1:443/TCP/10.6.40.32:6443
I1227 23:32:24.114348       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.33:6443
I1227 23:32:24.114397       1 graceful_termination.go:160] Trying to delete rs: 10.6.40.25:32444/TCP/10.244.20.41:32444
E1227 23:32:24.114446       1 graceful_termination.go:183] Try flush graceful termination list err
I1227 23:33:24.114853       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.32:6443
E1227 23:33:24.115043       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/10.6.40.32:6443" err: Failed to delete rs "10.96.0.1:443/TCP/10.6.40.32:6443", can't find the real server
I1227 23:33:24.115106       1 graceful_termination.go:93] lw: remote out of the list: 10.96.0.1:443/TCP/10.6.40.32:6443
I1227 23:33:24.115128       1 graceful_termination.go:160] Trying to delete rs: 10.96.0.1:443/TCP/10.6.40.33:6443

lbernail · Dec 28, 2018

@VincentFF : there are two issues:

failed deletions: fixed in 1.12.4
kube-proxy getting stuck due to a deadlock (which is the issue you are describing): hopefully fixed with the PR from yesterday. We are waiting for feedback from @emptywee and @bjornryden who are currently testing the patch. As soon as we have confirmation that the issue is not present anymore I'll create the cherrypicks for 1.13, 1.12 and 1.11 (if the patch works, it will probably be merged in all branches at the beginning of next week and will be part of the next patch releases)

matthiasr · Dec 28, 2018

I'm also re-running my experiment #71071 (comment) again, with the patch back-ported onto 1.12.4. Will report back!

emptywee · Dec 28, 2018

We haven't had any issues with accessing services while kube-proxy was(and is) running in the IPVS mode and we've had quite a few deployments and changes to the services. So far I'd say the patch is working well for us.

@lbernail Great job on fixing it! Much appreciated!

lbernail · Dec 28, 2018

Ok let's wait until monday so we also get feedback from @matthiasr and @bjornryden (I want to avoid rushing the cherry-picks and make sure that we have a clean fix this time)

VincentFF · Dec 28, 2018

@lbernail Good news, thanks to your reply.

matthiasr · Dec 28, 2018

No issues after 12+ hours on 1.12.4 with DataDog@9121f01 – without that I got a lockup in 1.5 hours, so this looks very good!

VincentFF · Dec 29, 2018

@lbernail I have another question, please. If the next patch is version 1.12.5?
If I just upgrade kube-proxy to 1.12.5, other units（apiserver, kubelet, controller-manager, scheduler ...) keep 1.12.3, is it ok? Or I have to upgrade all units?

lbernail · Dec 29, 2018

@VincentFF Hopefully it will make it to 1.12.5 yes. You can run kube-proxy 1.12.5 with other units running 1.12.3, no problem

matthiasr · Dec 29, 2018

Update: after 24 hours, still no issues, I'm stopping my experiment now & consider this a full success. Looking forward to 1.12.5!

lbernail · Dec 29, 2018

Ok this is very good news. I'm going to create the cherry-pick for all branches in order to get the fix in official releases as soon as possible.
Many thanks for your involvement in helping solve the issue and testing!

lbernail · Dec 29, 2018

Cherry-picks:
1.13: #72426
1.12: #72429
1.11: #72430

bjornryden · Dec 31, 2018

I can join the choir of happy testers. 4 days of no trouble happening.

Thanks a lot to all involved, especially @lbernail. Happy New Year!

VincentFF · Jan 1, 2019

I build it and deploy to test env, over 24 hours have no issues. Happy new year.

daigong · Jan 1, 2019

Hello, My env is
Ubuntu 16.04.4 LTS
Kubernetes version (use kubectl version): v1.13.0
Cloud provider or hardware configuration: bare metal, 64 bit linux
Kernel 4.4.0-75-generic

same bug:

I0101 22:20:09.742898 267576 graceful_termination.go:160] Trying to delete rs: 10.200.198.70:9090/TCP/172.200.101.32:9090
E0101 22:20:09.742975 267576 proxier.go:1519] Failed to get IPVS service, error: Expected only one service obtained=0
E0101 22:20:09.743036 267576 proxier.go:843] Failed to sync endpoint for service: 10.200.36.192:80/TCP, err: Expected only one service obtained=0

I need restart kube-proxy fix it

FrostyLeaf · Jan 2, 2019

@daigong v1.13.1 have fixed this. In my case, no issues after more than one week.

lbernail · Jan 2, 2019

@daigong : the full fix will actually be in 1.13.2 (it depends on how fast #72426 gets merged)

daigong · Jan 2, 2019

@lbernail @FrostyLeaf
thanks , I use #72426 to test this bug

Anything is ok now

lbernail · Jan 7, 2019

The PR has been merged in all release branches, so it will be present in:

1.11.7
1.12.5
1.13.2

grpubr · Jan 13, 2019

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊，，认识下。。少年

一样一样

Deshke · Jan 15, 2019

any chance for a new build in v1.11.x ?

verwilst · Jan 15, 2019

I've been refreshing the releases pages pretty much every hour for the last couple of days in the hope 1.12.5 will appear so i can patch this bug on multiple clusters. :-)

emptywee · Jan 15, 2019

@verwilst haha, same, can't wait for v1.11.7 to get released to upgrade current prod clusters. And v1.12.5 too to start upgrading in lower envs.

mrrandrade · Jan 17, 2019

I've been refreshing the releases pages pretty much every hour for the last couple of days in the hope 1.12.5 will appear so i can patch this bug on multiple clusters. :-)

Its here :-P

emptywee · Jan 17, 2019

I've been refreshing the releases pages pretty much every hour for the last couple of days in the hope 1.12.5 will appear so i can patch this bug on multiple clusters. :-)

Its here :-P

Why is it not there for 1.11? :(

k8s-ci-robot added kind/bug needs-sig labels Nov 15, 2018

k8s-ci-robot added sig/network and removed needs-sig labels Nov 15, 2018

k8s-ci-robot added the area/ipvs label Nov 23, 2018

Quentin-M referenced this issue Dec 7, 2018
Open
Trace works about enable the ipvs mode for the kube-proxy by default #71202
0 of 6 tasks complete

dims reopened this Dec 27, 2018

lbernail referenced this issue Dec 28, 2018
Open
Kube-proxy --> not setting up route in ipvs mode (race condition/cannot easily replicate) #72197

daigong pushed a commit to mxnavi/kubernetes that referenced this issue Jan 2, 2019

修改 kube-proxy ipvs 下，过一段时间ipvs rule不更新bug,kubernetes#71071

c41e082

rjosephwright added a commit to cloudboss/keights that referenced this issue Jan 6, 2019

Revert "Switch to iptables mode for kube-proxy"
This reverts commit 4ae9635, since v1.11.6 is now released with a fix for kubernetes/kubernetes#71071.

Loading status checks…

fef3c62

Lentil1016 referenced this issue Jan 11, 2019
Closed
kube-proxy在ipvs mode下工作几小时后停止更新ipvs规则，导致新Pod无法被访问 #44

kubernetes/kubernetes

Join GitHub today

kube-proxy ipvs mode hangs after a few hours, needs manual restart #71071

Comments

gjcarneiro commented Nov 15, 2018 • edited

k8s-ci-robot added kind/bug needs-sig labels Nov 15, 2018

This comment has been minimized.

gjcarneiro commented Nov 15, 2018

k8s-ci-robot added sig/network and removed needs-sig labels Nov 15, 2018

This comment has been minimized.

berlinsaint commented Nov 19, 2018 • edited

This comment has been minimized.

berlinsaint commented Nov 23, 2018

k8s-ci-robot added the area/ipvs label Nov 23, 2018

This comment has been minimized.

berlinsaint commented Nov 23, 2018

This comment has been minimized.

gjcarneiro commented Nov 23, 2018

This comment has been minimized.

berlinsaint commented Nov 26, 2018

This comment has been minimized.

k8s-ci-robot commented Nov 26, 2018

This comment has been minimized.

annProg commented Nov 26, 2018

This comment has been minimized.

berlinsaint commented Nov 27, 2018

This comment has been minimized.

berlinsaint commented Nov 27, 2018 • edited

This comment has been minimized.

annProg commented Nov 27, 2018

This comment has been minimized.

aleks-mariusz commented Nov 27, 2018 • edited

This comment has been minimized.

Lion-Wei commented Nov 29, 2018

This comment has been minimized.

gjcarneiro commented Nov 29, 2018

This comment has been minimized.

m1093782566 commented Nov 29, 2018

This comment has been minimized.

berlinsaint commented Nov 30, 2018 • edited

This comment has been minimized.

berlinsaint commented Nov 30, 2018 • edited

This comment has been minimized.

rtreffer commented Dec 5, 2018

This comment has been minimized.

m1093782566 commented Dec 5, 2018

This comment has been minimized.

matthiasr commented Dec 5, 2018

This comment has been minimized.

adidenko commented Dec 6, 2018

This comment has been minimized.

ChiefAlexander commented Dec 6, 2018

This comment has been minimized.

Quentin-M commented Dec 7, 2018 • edited

Quentin-M referenced this issue Dec 7, 2018

Trace works about enable the ipvs mode for the kube-proxy by default #71202

This comment has been minimized.

Quentin-M commented Dec 7, 2018

dims reopened this Dec 27, 2018

This comment has been minimized.

VincentFF commented Dec 28, 2018 • edited

This comment has been minimized.

lbernail commented Dec 28, 2018

lbernail referenced this issue Dec 28, 2018

Kube-proxy --> not setting up route in ipvs mode (race condition/cannot easily replicate) #72197

This comment has been minimized.

matthiasr commented Dec 28, 2018

This comment has been minimized.

emptywee commented Dec 28, 2018 • edited

This comment has been minimized.

lbernail commented Dec 28, 2018

This comment has been minimized.

VincentFF commented Dec 28, 2018

This comment has been minimized.

matthiasr commented Dec 28, 2018

This comment has been minimized.

VincentFF commented Dec 29, 2018

This comment has been minimized.

lbernail commented Dec 29, 2018

This comment has been minimized.

gjcarneiro commented Nov 15, 2018 •

edited

berlinsaint commented Nov 19, 2018 •

edited

berlinsaint commented Nov 27, 2018 •

edited

aleks-mariusz commented Nov 27, 2018 •

edited

berlinsaint commented Nov 30, 2018 •

edited

berlinsaint commented Nov 30, 2018 •

edited

Quentin-M commented Dec 7, 2018 •

edited

VincentFF commented Dec 28, 2018 •

edited

emptywee commented Dec 28, 2018 •

edited

mrrandrade commented Jan 17, 2019 •

edited

emptywee commented Jan 17, 2019 •

edited