Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

service ipvs mode. long request time. #89468

Open
itshikanov opened this issue Mar 25, 2020 · 5 comments
Open

service ipvs mode. long request time. #89468

itshikanov opened this issue Mar 25, 2020 · 5 comments

Comments

@itshikanov
Copy link

@itshikanov itshikanov commented Mar 25, 2020

What happened:
I have cluster with 3 master nodes. Master nodes tainted and can run pods.
service.conf.yml:

apiVersion: v1
kind: Service
metadata:
name: testproject-lb
spec:
type: ClusterIP
ports:

  • port: 9090
    protocol: TCP
    targetPort: 9090
    selector:
    app: testproject

project_deployment.yml:

apiVersion: apps/v1
kind: Deployment
metadata:
name: testproject
spec:
replicas: 3
minReadySeconds: 15
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
selector:
matchLabels:
app: testproject
template:
metadata:
labels:
app: testproject
spec:
containers:
- image: ANY_HTTP_APP_RETURN_IP
env:
- name: WORKERS
value: "16"
imagePullPolicy: Always
name: testproject
ports:
- containerPort: 9090
readinessProbe:
initialDelaySeconds: 10
httpGet:
path: /health
port: 9090

kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.233.0.1 443/TCP 6d
testproject-lb ClusterIP 10.233.34.81 9090/TCP 5d21h

ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.233.0.1:443 rr
-> 80.93.187.100:6443 Masq 1 1 0
-> 95.213.184.39:6443 Masq 1 1 0
-> 212.92.101.221:6443 Masq 1 1 0
TCP 10.233.0.3:53 rr
-> 10.233.64.3:53 Masq 1 0 0
-> 10.233.65.2:53 Masq 1 0 0
TCP 10.233.0.3:9153 rr
-> 10.233.64.3:9153 Masq 1 0 0
-> 10.233.65.2:9153 Masq 1 0 0
TCP 10.233.34.81:9090 rr
-> 10.233.64.4:9090 Masq 1 0 0
-> 10.233.65.4:9090 Masq 1 0 0
-> 10.233.65.5:9090 Masq 1 0 0
TCP 10.233.35.80:443 rr
-> 10.233.65.3:8443 Masq 1 0 0
UDP 10.233.0.3:53 rr
-> 10.233.64.3:53 Masq 1 0 0
-> 10.233.65.2:53 Masq 1 0 0

exec curl from console, on one server. 3 times to get all 3 pod.

time curl -s --output /dev/null 10.233.34.81:9090

real 0m3.067s
user 0m0.004s
sys 0m0.008s
time curl -s --output /dev/null 10.233.34.81:9090

real 0m0.060s
user 0m0.004s
sys 0m0.004s
time curl -s --output /dev/null 10.233.34.81:9090

real 0m3.065s
user 0m0.004s
sys 0m0.004s

3 seconds. When request go to another node.

What you expected to happen:
I am try to create my own ipvs rules with another port on some server.
Commands:
ipvsadm -A -t 10.233.62.199:9089 -s rr
ipvsadm -a -t 10.233.62.199:9089 -r 10.233.64.4:9090 -m
ipvsadm -a -t 10.233.62.199:9089 -r 10.233.65.5:9090 -m
ipvsadm -a -t 10.233.62.199:9089 -r 10.233.66.3:9090 -m

ipvsadm -ln

time curl -s --output /dev/null 10.233.62.199:9089
time curl -s --output /dev/null 10.233.62.199:9089
time curl -s --output /dev/null 10.233.62.199:9089

result

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.233.0.1:443 rr
-> 192.168.50.14:6443 Masq 1 0 0
-> 192.168.50.15:6443 Masq 1 1 0
-> 192.168.50.16:6443 Masq 1 1 0
TCP 10.233.0.3:53 rr
-> 10.233.64.3:53 Masq 1 0 0
-> 10.233.66.2:53 Masq 1 0 0
TCP 10.233.0.3:9153 rr
-> 10.233.64.3:9153 Masq 1 0 0
-> 10.233.66.2:9153 Masq 1 0 0
TCP 10.233.10.115:443 rr
-> 10.233.65.3:8443 Masq 1 0 0
TCP 10.233.62.199:9089 rr
-> 10.233.64.4:9090 Masq 1 0 1
-> 10.233.65.5:9090 Masq 1 0 1
-> 10.233.66.3:9090 Masq 1 0 1
TCP 10.233.62.199:9090 rr
-> 10.233.64.4:9090 Masq 1 0 0
-> 10.233.65.5:9090 Masq 1 0 0
-> 10.233.66.3:9090 Masq 1 0 0
UDP 10.233.0.3:53 rr
-> 10.233.64.3:53 Masq 1 0 0
-> 10.233.66.2:53 Masq 1 0 0

real 0m0.055s
user 0m0.004s
sys 0m0.000s

real 0m0.056s
user 0m0.004s
sys 0m0.000s

real 0m0.056s
user 0m0.000s
sys 0m0.000s

0 second to all 3 pods! With my own ipvs rules.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
    kubectl version
    Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration:
    Virtual box 3 servers and same on 3 Hardware servers.

  • OS (e.g: cat /etc/os-release):
    NAME="Ubuntu"
    VERSION="16.04.4 LTS (Xenial Xerus)"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 16.04.4 LTS"
    VERSION_ID="16.04"
    HOME_URL="http://www.ubuntu.com/"
    SUPPORT_URL="http://help.ubuntu.com/"
    BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
    VERSION_CODENAME=xenial
    UBUNTU_CODENAME=xenial

  • Kernel (e.g. uname -a):
    Linux 4.4.0-121-generic #145-Ubuntu SMP Fri Apr 13 13:47:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools:

  • Network plugin and version (if this is a network-related bug):

  • Others:

/sig Network

@itshikanov

This comment has been minimized.

Copy link
Author

@itshikanov itshikanov commented Mar 25, 2020

kube-proxy iptables mode the same problem. 3 second to pod on another node, if use service ip.

Try to install 1.16.3 - all Ok(iptables and ipvs mode) - 0m0.004s to all pods if use service ip.

@uablrek

This comment has been minimized.

Copy link
Contributor

@uablrek uablrek commented Mar 27, 2020

What CNI-plugin (version) are you using?

@uablrek

This comment has been minimized.

Copy link
Contributor

@uablrek uablrek commented Mar 27, 2020

I have seen that "vxlan" can be a problem. If possible, test without vxlan (if you are using it)

@uablrek

This comment has been minimized.

Copy link
Contributor

@uablrek uablrek commented Mar 27, 2020

See also #86507 (comment)

@itshikanov

This comment has been minimized.

Copy link
Author

@itshikanov itshikanov commented Mar 30, 2020

What CNI-plugin (version) are you using?

quay.io/coreos/flannel:v0.11.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.