kube-proxy ipvs mode hangs after a few hours, needs manual restart #71071

gjcarneiro · 2018-11-15T12:11:10Z

What happened: upgraded from v1.11.0 to v1.12.2, now kube-proxy hangs, usually in less than a day

What you expected to happen: it should continue working

How to reproduce it (as minimally and precisely as possible): just have a normal kube-proxy running in ipvs mode. It gets stuck usually in less than a day.

Anything else we need to know?:

I've sent SIGABRT to the kube-proxy process to get a stack trace, and the only thing interesting is that when the bug happens goroutine 1 is reading from netlink, whereas when I send SIGABRT and the process is not stuck it is not reading from netlink.

Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: goroutine 1 [syscall]:
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.Syscall6(0x2d, 0x3, 0xc42081c000, 0x1000, 0x0, 0xc42083deb0, 0xc42083dea4, 0x40fb86, 0x7f2ab9788aa8, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.recvfrom(0x3, 0xc42081c000, 0x1000, 0x1000, 0x0, 0xc42083deb0, 0xc42083dea4, 0x101ffffffffffff, 0x0, 0x1000)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1665 +0xa6
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: syscall.Recvfrom(0x3, 0xc42081c000, 0x1000, 0x1000, 0x0, 0x1000, 0x0, 0x0, 0x16ad4c0, 0x1f4d508)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /usr/local/go/src/syscall/syscall_unix.go:252 +0xaf
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xc4204d3440, 0x0, 0x0, 0x0, 0x16ad4c0, 0x1f4d508)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:613 +0x9b
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.execute(0xc4204d3440, 0xc42083e150, 0x0, 0x1, 0x2, 0xc420423740, 0x1, 0x2)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:219 +0xd3
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).doCmdwithResponse(0xc420573990, 0xc420990750, 0x0, 0xc42003d501, 0xa, 0x10, 0xc420990701, 0xc42083e230, 0x118dd6b)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:140 +0x20b
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).doCmd(0xc420573990, 0xc420990750, 0x0, 0x1, 0x60000020fc6e0, 0xc420990750)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:149 +0x48
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.(*Handle).NewService(0xc420573990, 0xc420990750, 0x0, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/ipvs.go:117 +0x43
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/ipvs.(*runner).AddVirtualServer(0xc42052a340, 0xc420b0e550, 0x1e, 0xc42083e3b8)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/ipvs/ipvs_linux.go:61 +0x60
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).syncService(0xc4201b3e00, 0xc420d53620, 0x21, 0xc420b0e550, 0xc420357901, 0x5, 0xc420d53620)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:1454 +0x635
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).syncProxyRules(0xc4201b3e00)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:803 +0x1081
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).(k8s.io/kubernetes/pkg/proxy/ipvs.syncProxyRules)-fm()
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:395 +0x2a
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/async.(*BoundedFrequencyRunner).tryRun(0xc42042e3f0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/async/bounded_frequency_runner.go:217 +0xb6
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/util/async.(*BoundedFrequencyRunner).Loop(0xc42042e3f0, 0xc4200b0120)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/async/bounded_frequency_runner.go:179 +0x211
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/pkg/proxy/ipvs.(*Proxier).SyncLoop(0xc4201b3e00)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:589 +0x4e
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.(*ProxyServer).Run(0xc4207f2000, 0xc4207f2000, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:568 +0x3dd
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.(*Options).Run(0xc4201082c0, 0xc42063e230, 0x0)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:238 +0x69
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/cmd/kube-proxy/app.NewProxyCommand.func1(0xc4200f2500, 0xc42063e230, 0x0, 0x7)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:360 +0x156
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200f2500, 0xc4200c4010, 0x7, 0x7, 0xc4200f2500, 0xc4200c4010)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200f2500, 0xc4205dc090, 0x16076a0, 0xc4207a7ee8)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200f2500, 0x1608b88, 0x20dcf40)
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]:         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b
Nov 15 11:42:05 hex-48b-pm kube-proxy[5389]: main.main()

The last lines that it logged were:

Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392942    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.241:4059/TCP/10.125.2.26:4059
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392952    5389 graceful_termination.go:160] Trying to delete rs: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: E1115 07:48:52.392978    5389 graceful_termination.go:89] Try delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432" err: Failed to delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432", can't 
find the real server
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: I1115 07:48:52.392985    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:48:52 hex-48b-pm kube-proxy[5389]: E1115 07:48:52.392992    5389 graceful_termination.go:183] Try flush graceful termination list err
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393147    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.156:8125/TCP/10.125.2.21:8125
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393260    5389 graceful_termination.go:89] Try delete rs "10.128.33.156:8125/TCP/10.125.2.21:8125" err: Failed to delete rs "10.128.33.156:8125/TCP/10.125.2.21:8125", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393272    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.156:8125/TCP/10.125.2.21:8125
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393281    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.142:6379/TCP/10.125.6.13:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393310    5389 graceful_termination.go:89] Try delete rs "10.128.33.142:6379/TCP/10.125.6.13:6379" err: Failed to delete rs "10.128.33.142:6379/TCP/10.125.6.13:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393317    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.142:6379/TCP/10.125.6.13:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393327    5389 graceful_termination.go:160] Trying to delete rs: 10.128.34.237:6379/TCP/10.125.6.11:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393355    5389 graceful_termination.go:89] Try delete rs "10.128.34.237:6379/TCP/10.125.6.11:6379" err: Failed to delete rs "10.128.34.237:6379/TCP/10.125.6.11:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393362    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.34.237:6379/TCP/10.125.6.11:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393371    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.200:5432/TCP/10.125.6.27:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393397    5389 graceful_termination.go:89] Try delete rs "10.128.33.200:5432/TCP/10.125.6.27:6432" err: Failed to delete rs "10.128.33.200:5432/TCP/10.125.6.27:6432", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393404    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.200:5432/TCP/10.125.6.27:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393412    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.228:6379/TCP/10.125.2.63:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393440    5389 graceful_termination.go:89] Try delete rs "10.128.33.228:6379/TCP/10.125.2.63:6379" err: Failed to delete rs "10.128.33.228:6379/TCP/10.125.2.63:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393447    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.228:6379/TCP/10.125.2.63:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393455    5389 graceful_termination.go:160] Trying to delete rs: 10.128.34.247:5672/TCP/10.125.2.6:5672
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393482    5389 graceful_termination.go:89] Try delete rs "10.128.34.247:5672/TCP/10.125.2.6:5672" err: Failed to delete rs "10.128.34.247:5672/TCP/10.125.2.6:5672", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393489    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.34.247:5672/TCP/10.125.2.6:5672
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393497    5389 graceful_termination.go:160] Trying to delete rs: 10.128.35.47:4007/TCP/10.125.2.15:4007
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393523    5389 graceful_termination.go:89] Try delete rs "10.128.35.47:4007/TCP/10.125.2.15:4007" err: Failed to delete rs "10.128.35.47:4007/TCP/10.125.2.15:4007", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393530    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.35.47:4007/TCP/10.125.2.15:4007
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393538    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.241:4059/TCP/10.125.2.26:4059
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393563    5389 graceful_termination.go:89] Try delete rs "10.128.33.241:4059/TCP/10.125.2.26:4059" err: Failed to delete rs "10.128.33.241:4059/TCP/10.125.2.26:4059", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393571    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.241:4059/TCP/10.125.2.26:4059
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393578    5389 graceful_termination.go:160] Trying to delete rs: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393605    5389 graceful_termination.go:89] Try delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432" err: Failed to delete rs "10.128.32.132:5432/TCP/10.125.2.13:6432", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393613    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.32.132:5432/TCP/10.125.2.13:6432
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393621    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.252:6379/TCP/10.125.6.26:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393648    5389 graceful_termination.go:89] Try delete rs "10.128.33.252:6379/TCP/10.125.6.26:6379" err: Failed to delete rs "10.128.33.252:6379/TCP/10.125.6.26:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393656    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.252:6379/TCP/10.125.6.26:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393664    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.237:6379/TCP/10.125.6.34:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393690    5389 graceful_termination.go:89] Try delete rs "10.128.33.237:6379/TCP/10.125.6.34:6379" err: Failed to delete rs "10.128.33.237:6379/TCP/10.125.6.34:6379", can't find the real server
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: I1115 07:49:52.393697    5389 graceful_termination.go:93] lw: remote out of the list: 10.128.33.237:6379/TCP/10.125.6.34:6379
Nov 15 07:49:52 hex-48b-pm kube-proxy[5389]: E1115 07:49:52.393704    5389 graceful_termination.go:183] Try flush graceful termination list err
Nov 15 07:50:52 hex-48b-pm kube-proxy[5389]: I1115 07:50:52.393838    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.156:8125/TCP/10.125.2.21:8125
Nov 15 07:50:52 hex-48b-pm kube-proxy[5389]: E1115 07:50:52.393940    5389 graceful_termination.go:89] Try delete rs "10.128.33.156:8125/TCP/10.125.2.21:8125" err: device or resource busy
Nov 15 07:50:52 hex-48b-pm kube-proxy[5389]: I1115 07:50:52.393957    5389 graceful_termination.go:160] Trying to delete rs: 10.128.33.142:6379/TCP/10.125.6.13:6379
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420734    5389 proxier.go:1496] Failed to list IPVS destinations, error: invalid argument
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420763    5389 proxier.go:809] Failed to sync endpoint for service: 10.128.33.156:8125/TCP, err: invalid argument
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420889    5389 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420903    5389 proxier.go:809] Failed to sync endpoint for service: 10.128.35.150:6379/TCP, err: Expected only one service obtained=0
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420950    5389 proxier.go:1455] Failed to add IPVS service "monitoring/extradata-inserter:": file exists
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.420961    5389 proxier.go:812] Failed to sync service: 10.128.34.54:80/TCP, err: file exists
Nov 15 07:51:22 hex-48b-pm kube-proxy[5389]: E1115 07:51:22.421886    5389 proxier.go:1544] Failed to add destination: 10.125.27.66:80, error: file exists

If I restart kube-proxy (manually, ugh), it all recovers and works fine. But it will get stuck again, eventually.

Environment:

Kubernetes version (use kubectl version): v1.12.2
Cloud provider or hardware configuration: bare metal, 64 bit linux
OS (e.g. from /etc/os-release): Ubuntu 16.04.5 LTS
Kernel (e.g. uname -a): 4.15.0-39-generic (but get it on the original 4.4 kernel as well)
Install tools: binary from https://storage.googleapis.com/kubernetes-release/release/v1.12.2/kubernetes-server-linux-amd64.tar.gz
Others:

/kind bug

The text was updated successfully, but these errors were encountered:

gjcarneiro · 2018-11-15T12:37:28Z

/sig network

berlinsaint · 2018-11-19T08:01:30Z

the same error. upgrade from 1.11.1. Everything become good except kube-proxy becomse not stable, some time pod can't access svc though their endpoints are ok .. And the same as the name resolution. so i think it may exist an important bug here.

berlinsaint · 2018-11-23T01:45:43Z

/area ipvs

berlinsaint · 2018-11-23T06:33:26Z

@gjcarneiro do you resolve this problem?

gjcarneiro · 2018-11-23T09:32:15Z

Well, I switched the cluster back to the default (iptables) mode, appears to be pretty stable since then...

berlinsaint · 2018-11-26T01:28:12Z

@kubernetes/sig-network-bugs

k8s-ci-robot · 2018-11-26T01:28:19Z

@berlinsaint: Reiterating the mentions to trigger a notification:
@kubernetes/sig-network-bugs

In response to this:

@kubernetes/sig-network-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

annProg · 2018-11-26T16:09:30Z

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

berlinsaint · 2018-11-27T06:07:16Z

@gjcarneiro do you mean that in kernel 4.15 , it has the same problem?

berlinsaint · 2018-11-27T06:07:51Z

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊，，认识下。。少年

annProg · 2018-11-27T06:44:42Z

@annProg

same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei

你邮箱多少啊啊，，认识下。。少年

点我头像就看到了

aleks-mariusz · 2018-11-27T17:58:26Z

I am also seeing this repeatedly in my centralized logging system:

  | Time | sys_name | log_lvl | tag | log_msg
  | November 27th 2018, 17:56:23.501 | k8lab2bs | 6 | kube.kube-system.kube-proxy | lw: remote out of the list: 10.12.12.1:443/TCP/10.12.100.133:6443
  | November 27th 2018, 17:56:23.500 | k8lab2bs | 6 | kube.kube-system.kube-proxy | Deleting rs: 10.12.12.1:443/TCP/10.12.100.133:6443
  | November 27th 2018, 17:56:23.497 | k8lab2bs | 6 | kube.kube-system.kube-proxy | Trying to delete rs: 10.12.12.1:443/TCP/10.12.100.133:6443

I'm running v1.12.3 (fresh install, not an upgrade).

Lion-Wei · 2018-11-29T01:25:58Z

Hi, all. Thanks for the report, I'll make a research and try to figure out what happened, any progress I'll report here.

gjcarneiro · 2018-11-29T11:17:54Z

Besides what happened, which is good to find out, I don't feel confident about IPVS until there is code in place that detects and resets a broken netlink socket. Because from what I can guess from the logs and traces it is that the netlink socket gets into a broken state and the task that reads/writes from/to it will hang forever. I think kube-proxy needs to have code that detects this problem, and if it happens create a new netlink socket from scratch, and maybe restart the task that handles it. But, no, I'm not volunteering, I don't even know Go, sorry.

m1093782566 · 2018-11-29T11:23:44Z

Because from what I can guess from the logs and traces it is that the netlink socket gets into a broken state and the task that reads/writes from/to it will hang forever.

We upgraded the go netlink version some days ago for supporting graceful termination. The netlink library was running pretty stable before. I am not sure if it's the cause.

berlinsaint · 2018-11-30T01:10:08Z

1.12.3 with the same error.
After filter the INFO log,
execute kubectl -n kube-system logs kube-proxy-29zh8|egrep ^E; kubectl -n kube-system logs kube-proxy-2pqdn|egrep ^E; kubectl -n kube-system logs kube-proxy-4xw8q|egrep ^E; kubectl -n kube-system logs kube-proxy-6j4bc|egrep ^E; kubectl -n kube-system logs kube-proxy-brbjb|egrep ^E; kubectl -n kube-system logs kube-proxy-r6cg2|egrep ^E; kubectl -n kube-system logs kube-proxy-rpl6t|egrep ^E;
and all the proxy node log is

E1129 01:01:10.862180       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 21:09:11.422553       1 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
E1129 21:09:11.422699       1 proxier.go:1116] Failed to sync endpoint for service: 172.17.0.1:31800/TCP, err: Expected only one service obtained=0
E1129 01:01:11.618125       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:11.905626       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:01:41.771596       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:02:11.923872       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:02:42.060658       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:03:12.326762       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:03:42.718417       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:04:12.859933       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:04:42.963527       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:05:13.078486       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:05:43.185141       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:06:13.303090       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:06:43.408582       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:07:13.550358       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:07:43.686235       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:08:13.786326       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:08:43.981764       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:09:14.188053       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:09:44.316316       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:10:14.440466       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:10:44.550092       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:11:14.658190       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:11:44.765396       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:12:14.975724       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:12:45.085632       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:13:15.317679       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:13:45.432469       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:14:15.689681       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:14:45.813565       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:15:33.434931       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:16:03.578017       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:16:33.709744       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:17:03.823325       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:17:33.935191       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:18:04.053183       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:18:34.248994       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:04.375946       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:34.552779       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:36.596182       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:47.433846       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:20:17.534206       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:20:47.682350       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:21:17.825175       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:21:47.942548       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:22:18.055820       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:22:48.149494       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:23:18.564985       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:23:48.678462       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:24:18.836217       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:24:48.991344       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:25:19.091032       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:25:49.187178       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:26:19.301770       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:26:49.425812       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:27:19.606711       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:27:49.721224       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:28:20.109117       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:28:50.211574       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:29:20.405007       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:29:50.507766       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:30:20.605471       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:30:50.702798       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:31:20.807400       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:31:50.903049       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:32:21.005618       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:32:51.106378       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:33:21.224447       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:33:51.354916       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:34:21.467898       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:34:51.583216       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:35:21.691099       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:35:51.797151       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:36:21.921057       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:36:52.083821       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:02.445737       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:07.427811       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:37.546362       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:38:07.651769       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:38:37.797858       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:39:08.064112       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:39:38.181089       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:40:08.282892       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:40:38.390167       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:41:08.508549       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:41:38.617921       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:42:08.728384       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:42:38.844074       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:43:08.973517       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:43:39.092095       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:44:09.202329       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:44:39.363721       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:45:09.476240       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:45:39.764944       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:46:09.894587       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:46:40.013388       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:47:10.127894       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:47:40.236119       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:48:10.397111       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:48:40.600084       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:49:10.877034       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:49:41.023593       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:50:11.149894       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:50:41.262320       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:51:11.410407       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:51:41.547057       1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:52:11.609974       1 proxier.go:1544] Failed to add destination: 172.31.200.9:443, error: file exists
E1129 01:52:11.610088       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: invalid argument
E1129 01:52:11.616690       1 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
E1129 01:52:11.616767       1 proxier.go:1116] Failed to sync endpoint for service: 172.18.0.1:31645/TCP, err: Expected only one service obtained=0
E1129 01:01:12.181112       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:22.278474       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:21.459378       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:17.442316       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 20:10:17.955872       1 proxier.go:1455] Failed to add IPVS service "dev-fffrf/mysql-testportal:mysql": file exists
E1129 20:10:17.955863       1 graceful_termination.go:89] Try delete rs "10.100.217.68:15004/TCP/192.100.143.225:15004" err: Failed to delete rs "10.100.217.68:15004/TCP/192.100.143.225:15004", can't find the real server
E1129 20:10:17.955892       1 proxier.go:1119] Failed to sync service: 172.17.0.1:31645/TCP, err: file exists
E1129 20:10:17.956412       1 proxier.go:1485] Failed to get IPVS service, error: invalid argument
E1129 20:10:17.956417       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: Failed to delete rs "10.96.0.1:443/TCP/172.31.10.38:6443", can't find the real server
E1129 20:10:17.956426       1 proxier.go:1116] Failed to sync endpoint for service: 172.31.200.67:31645/TCP, err: invalid argument
E1129 20:10:17.956448       1 graceful_termination.go:89] Try delete rs "10.101.1.98:5672/TCP/192.100.29.180:5672" err: invalid argument
E1129 20:10:17.956547       1 graceful_termination.go:183] Try flush graceful termination list err
E1129 01:01:17.484679       1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 02:52:17.498135       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: invalid argument
E1129 02:52:17.498396       1 graceful_termination.go:183] Try flush graceful termination list err
E1129 06:28:17.607394       1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: device or resource busy
E1129 06:28:17.607513       1 graceful_termination.go:183] Try flush graceful termination list err
E1129 10:02:17.718458       1 proxier.go:1544] Failed to add destination: 192.100.183.139:3306, error: file exists
E1129 10:02:17.718606       1 proxier.go:1544] Failed to add destination: 192.100.183.152:8080, error: invalid argument```

berlinsaint · 2018-11-30T02:06:24Z

BTW. from above my logs , kube-proxy seems like still have many problems with iptables-restore . all my node are UBUNTU 16.04 which have iptable's version 1.6.0.
@m1093782566 @Lion-Wei

rtreffer · 2018-12-05T11:29:46Z

We see this happening when upgrading from 1.12.1. it must thus be in the delta of 1.12.1 and 1.12.2.

m1093782566 · 2018-12-05T11:35:32Z

The issue is graceful termination related and I think we have fixed the bug in v1.13.1.

matthiasr · 2018-12-05T12:36:37Z

@m1093782566 will the fix be backported to the 1.12 line? right now this is blocking the 1.12.3 security fix for us.

adidenko · 2018-12-06T16:05:50Z

Got the same with upgrade from 1.11.2 to 1.11.5.

ChiefAlexander · 2018-12-06T17:31:36Z

@m1093782566 can you provide more information on the fix? I would be interested to look over the commit.

Quentin-M · 2018-12-07T05:41:26Z

This is pretty insane, we got bitten really hard earlier today.

Quentin-M · 2018-12-07T06:18:55Z

We see this happening when upgrading from 1.12.1. it must thus be in the delta of 1.12.1 and 1.12.2.

Got the same with upgrade from 1.11.2 to 1.11.5.

Was the bug chery-picked?

Deshke · 2019-01-15T16:25:42Z

any chance for a new build in v1.11.x ?

verwilst · 2019-01-15T16:34:21Z

I've been refreshing the releases pages pretty much every hour for the last couple of days in the hope 1.12.5 will appear so i can patch this bug on multiple clusters. :-)

emptywee · 2019-01-15T22:11:33Z

@verwilst haha, same, can't wait for v1.11.7 to get released to upgrade current prod clusters. And v1.12.5 too to start upgrading in lower envs.

mrrandrade · 2019-01-17T20:58:03Z

I've been refreshing the releases pages pretty much every hour for the last couple of days in the hope 1.12.5 will appear so i can patch this bug on multiple clusters. :-)

Its here :-P

emptywee · 2019-01-17T23:59:50Z

I've been refreshing the releases pages pretty much every hour for the last couple of days in the hope 1.12.5 will appear so i can patch this bug on multiple clusters. :-)

Its here :-P

Why is it not there for 1.11? :(

feie22 · 2019-01-25T08:04:22Z

Was this bug fixed in 1.11.7? I didn't see any release notes.

dawidmalina · 2019-01-25T08:44:50Z

Yes it is https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1117:

Fix race condition introduced by graceful termination which can lead to a deadlock in kube-proxy (#72361, @lbernail)

lbernail · 2019-01-25T09:04:41Z

Yes I forgot to add "IPVS" in the release-note in the PR, sorry about that
Let us know if it works ok now! (we'll leave the issue open for a weeks just in case)

matthiasr · 2019-01-29T10:56:34Z

We've had no issues since upgrading to 1.12.5 🎉

lbernail · 2019-01-29T10:58:47Z

Great news!

lbernail · 2019-03-09T10:46:06Z

@m1093782566 , @thockin I think we can close this issue (no new reports in about 2 months)

For reference, bug impacts initial releases with graceful termination:

1.11.5 - 1.11.6
1.12.2 - 1.12.4
1.13.0 - 1.13.1

Fix is present after:

1.11.7
1.12.5
1.13.2

m1093782566 · 2019-03-09T10:48:40Z

/close

k8s-ci-robot · 2019-03-09T10:48:41Z

@m1093782566: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fcvarela · 2019-06-22T09:07:49Z

Can we reopen? We're seeing this issue on 1.12.7.

sara4dev · 2019-07-09T16:08:43Z

yes, we are seeing the issues in 1.12.9 too

lbernail · 2019-07-09T17:32:07Z

Are you sure it is the same issue?
Can you post the logs of a kube-proxy instance?

sara4dev · 2019-07-11T02:31:05Z

sorry, it may have been a false alert... We see this is the logs and its not the same error

I0709 11:59:48.640888       1 graceful_termination.go:160] Trying to delete rs: 192.168.240.51:80/TCP/172.30.71.181:5050
I0709 11:59:48.640304       1 graceful_termination.go:93] lw: remote out of the list: 192.168.240.51:80/TCP/172.30.106.125:5050

for some reason, when the cluster is under load it couldn't route to some of the cluster service IPs. We are still not sure if it is an IPVS issue.

lbernail · 2019-07-11T12:04:08Z

Maybe the endpoint propagation is taking time if the apiserver is overloaded?

The messages are normal Info level messages:
https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/graceful_termination.go#L94

https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/graceful_termination.go#L161

cw-sakamoto · 2019-07-12T02:08:20Z

I can confirm the same log in 1.14.4. When does this log come out?

I0712 01:34:49.925012       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.22.42:443
I0712 01:34:49.925057       1 graceful_termination.go:175] Deleting rs: 11.3.0.1:443/TCP/10.0.22.42:443
I0712 01:35:11.907742       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.10:53/TCP/11.2.5.13:53
I0712 01:35:11.907874       1 graceful_termination.go:175] Deleting rs: 11.3.0.10:53/TCP/11.2.5.13:53
I0712 01:35:11.908006       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.10:53/UDP/11.2.5.13:53
I0712 01:35:11.908059       1 graceful_termination.go:175] Deleting rs: 11.3.0.10:53/UDP/11.2.5.13:53
I0712 01:35:21.992487       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.10:53/TCP/11.2.4.11:53
I0712 01:35:21.992594       1 graceful_termination.go:175] Deleting rs: 11.3.0.10:53/TCP/11.2.4.11:53
I0712 01:35:21.992784       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.10:53/UDP/11.2.4.11:53
I0712 01:35:21.992858       1 graceful_termination.go:175] Deleting rs: 11.3.0.10:53/UDP/11.2.4.11:53
I0712 01:36:29.411592       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.199:44134/TCP/11.2.0.5:44134
I0712 01:36:29.411660       1 graceful_termination.go:175] Deleting rs: 11.3.0.199:44134/TCP/11.2.0.5:44134
I0712 01:36:29.534158       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.17:8085/TCP/11.2.0.3:8085
I0712 01:36:29.534394       1 graceful_termination.go:175] Deleting rs: 11.3.0.17:8085/TCP/11.2.0.3:8085
I0712 01:36:50.745469       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:36:50.745508       1 graceful_termination.go:172] Not deleting, RS 11.3.0.1:443/TCP/10.0.22.162:443: 0 ActiveConn, 12 InactiveConn
I0712 01:37:37.475724       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:37:37.475841       1 graceful_termination.go:172] Not deleting, RS 11.3.0.1:443/TCP/10.0.22.162:443: 0 ActiveConn, 4 InactiveConn
I0712 01:38:37.475956       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:38:37.476201       1 graceful_termination.go:172] Not deleting, RS 11.3.0.1:443/TCP/10.0.22.162:443: 0 ActiveConn, 4 InactiveConn
I0712 01:39:37.476324       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:39:37.476585       1 graceful_termination.go:175] Deleting rs: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:39:37.476628       1 graceful_termination.go:94] lw: remote out of the list: 11.3.0.1:443/TCP/10.0.22.162:443
I0712 01:40:36.883345       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.245:5473/TCP/10.0.22.25:5473
I0712 01:40:36.883566       1 graceful_termination.go:175] Deleting rs: 11.3.0.245:5473/TCP/10.0.22.25:5473
I0712 01:40:36.886079       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.17:8085/TCP/11.2.1.3:8085
I0712 01:40:36.886310       1 graceful_termination.go:175] Deleting rs: 11.3.0.17:8085/TCP/11.2.1.3:8085
I0712 01:40:46.927450       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.245:5473/TCP/10.0.22.191:5473
I0712 01:40:46.927525       1 graceful_termination.go:175] Deleting rs: 11.3.0.245:5473/TCP/10.0.22.191:5473
I0712 01:40:56.968884       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.245:5473/TCP/10.0.22.75:5473
I0712 01:40:56.968936       1 graceful_termination.go:175] Deleting rs: 11.3.0.245:5473/TCP/10.0.22.75:5473
I0712 01:41:16.747362       1 graceful_termination.go:161] Trying to delete rs: 11.3.0.1:443/TCP/10.0.23.45:443

lbernail · 2019-07-12T08:13:54Z

These are just information logs:

Trying to delete rs : a backend pod from removed, trying to delete it. If it has existing connections, make it enter graceful termination, otherwise delete id
Deleting rs : the backend has no connection, removing it from IPVS

@andrewsykim Maybe we could now decrease the level of these logs so they only show above v=5 for instance ?

andrewsykim · 2019-07-12T14:32:21Z

In #78395 we bumped the log level from v=0 to v=2, I think v=5 makes sense. #80100 for update

k8s-ci-robot added kind/bug needs-sig labels Nov 15, 2018

k8s-ci-robot added sig/network and removed needs-sig labels Nov 15, 2018

k8s-ci-robot added the area/ipvs label Nov 23, 2018

Quentin-M mentioned this issue Dec 7, 2018

Trace works about enable the ipvs mode for the kube-proxy by default #71202

Closed

6 tasks

annProg mentioned this issue Feb 11, 2019

IPVS: rs delete failed after scaledown #70997

Closed

2PacIsAlive mentioned this issue Feb 27, 2019

1.13.1 kube-proxy ipvs don't work when pod ip is changed #72270

Closed

thockin added the triage/unresolved label Mar 8, 2019

k8s-ci-robot closed this Mar 9, 2019

lentil1016 mentioned this issue Mar 27, 2019

请问哪里有执行脚本之前要做事项的教程？ lentil1016/kubeadm-ha#57

Closed

foxyriver mentioned this issue Apr 23, 2019

ZooKeeperClientTimeoutException during KafkaServer startup strimzi/strimzi-kafka-operator#1392

Closed

lbernail mentioned this issue May 13, 2019

REQUEST: New membership for lbernail kubernetes/org#811

Closed

6 tasks

andrewsykim mentioned this issue Jul 12, 2019

proxy/ipvs: increase log level for graceful termination #80100

Merged

zh168654 mentioned this issue Nov 18, 2019

kube-proxy ipvs mode cannot access clusterip:port when node reboot #85422

Closed

kubernetes / kubernetes Public

kube-proxy ipvs mode hangs after a few hours, needs manual restart #71071

kube-proxy ipvs mode hangs after a few hours, needs manual restart #71071

Comments

gjcarneiro commented Nov 15, 2018 • edited

gjcarneiro commented Nov 15, 2018

berlinsaint commented Nov 19, 2018 • edited

berlinsaint commented Nov 23, 2018

berlinsaint commented Nov 23, 2018

gjcarneiro commented Nov 23, 2018

berlinsaint commented Nov 26, 2018

k8s-ci-robot commented Nov 26, 2018

annProg commented Nov 26, 2018

berlinsaint commented Nov 27, 2018

berlinsaint commented Nov 27, 2018 • edited

annProg commented Nov 27, 2018

aleks-mariusz commented Nov 27, 2018 • edited

Lion-Wei commented Nov 29, 2018

gjcarneiro commented Nov 29, 2018

m1093782566 commented Nov 29, 2018

berlinsaint commented Nov 30, 2018 • edited

berlinsaint commented Nov 30, 2018 • edited

rtreffer commented Dec 5, 2018

m1093782566 commented Dec 5, 2018

matthiasr commented Dec 5, 2018

adidenko commented Dec 6, 2018

ChiefAlexander commented Dec 6, 2018

Quentin-M commented Dec 7, 2018 • edited

Quentin-M commented Dec 7, 2018

Deshke commented Jan 15, 2019

verwilst commented Jan 15, 2019

emptywee commented Jan 15, 2019

mrrandrade commented Jan 17, 2019 • edited

emptywee commented Jan 17, 2019 • edited

feie22 commented Jan 25, 2019

dawidmalina commented Jan 25, 2019

lbernail commented Jan 25, 2019

matthiasr commented Jan 29, 2019

lbernail commented Jan 29, 2019

lbernail commented Mar 9, 2019

m1093782566 commented Mar 9, 2019

k8s-ci-robot commented Mar 9, 2019

fcvarela commented Jun 22, 2019

sara4dev commented Jul 9, 2019

lbernail commented Jul 9, 2019

sara4dev commented Jul 11, 2019 • edited

lbernail commented Jul 11, 2019

cw-sakamoto commented Jul 12, 2019 • edited

lbernail commented Jul 12, 2019 • edited

andrewsykim commented Jul 12, 2019

gjcarneiro commented Nov 15, 2018 •

edited

berlinsaint commented Nov 19, 2018 •

edited

berlinsaint commented Nov 27, 2018 •

edited

aleks-mariusz commented Nov 27, 2018 •

edited

berlinsaint commented Nov 30, 2018 •

edited

berlinsaint commented Nov 30, 2018 •

edited

Quentin-M commented Dec 7, 2018 •

edited

mrrandrade commented Jan 17, 2019 •

edited

emptywee commented Jan 17, 2019 •

edited

sara4dev commented Jul 11, 2019 •

edited

cw-sakamoto commented Jul 12, 2019 •

edited

lbernail commented Jul 12, 2019 •

edited