Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubernetes (v1.15.0~v1.15.2) IPVS Rule lost #81405

Open
zops opened this issue Aug 14, 2019 · 3 comments

Comments

@zops
Copy link

commented Aug 14, 2019

docker version:18.06.3-ce
kubeadm:v1.15.2
kubernetes:v1.15.2
CentOS Kernel:3.10.0-957.21.2.el7.x86_64
kubernetes-dashboard:v1.10.1

1、#### kubernetes-dashboard pod,status Running.
$ kubectl get pod -n kube-ops -l app=kubernetes-dashboard -o wide
NAME READY STATUS RESTARTS AGE IP
kubernetes-dashboard-79c94979cb-sqbql 1/1 Running 0 22h 172.26.145.195

2、#### kubernetes-dashboard svc,tatus Running.
$ kubectl get svc -n kube-ops
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard ClusterIP 10.111.167.126 443/TCP 21h

3、#### IPVS configuration
$ kubectl edit cm kube-proxy -n kube-system
ipvs:
excludeCIDRs: null
minSyncPeriod: 0s
scheduler: wlc
strictARP: false
syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: ipvs
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
resourceContainer: /kube-proxy
udpIdleTimeout: 250ms
winkernel:
enableDSR: false
networkName: ""
sourceVip: ""

4、#### Kubernetes-dashboard svc last log.
$ kubectl logs -n kube-ops svc/kubernetes-dashboard | tail -n 1
2019/08/14 05:09:56 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.

$ kubectl logs -n kube-ops svc/kubernetes-dashboard | grep 'Retrying in 30 seconds' | wc -l
2652

5、#### Ipvs rule is lost, real-server (Pod IP) is lost.
$ ipvsadm -ln | grep -C 2 '10.111.167.126'
TCP 10.106.80.22:44134 wlc
-> 172.26.145.193:44134 Masq 1 0 0
TCP 10.111.167.126:443 wlc # kubernetes-dashboard svc no destination available
TCP 10.108.75.241:8443 wlc
-> 172.26.145.194:8443 Masq 1 0 0

6、#############
Initially, Kubernetes-dashboard can be accessed.
Kubernetes-dashboard is inaccessible after 10 hours.

Finally found that lost real-server in IPVS.
Services lost by ipvs rules, including: nginx-ingress, kubernetes-dashboard(All Pod IP).

I don't know if it is a k8s bug or my own configuration error.

@zops zops added the kind/bug label Aug 14, 2019

@zouyee

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

/sig network

@k8s-ci-robot k8s-ci-robot added sig/network and removed needs-sig labels Aug 14, 2019

@athenabot

This comment has been minimized.

Copy link

commented Aug 15, 2019

/triage unresolved

Comment /remove-triage unresolved when the issue is assessed and confirmed.

🤖 I am a bot run by vllry. 👩‍🔬

@rikatz

This comment has been minimized.

Copy link
Contributor

commented Aug 18, 2019

@zops can you please post also a kubectl get endpoints of the objects before and after the error occurs?

Let's see if this is related.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.