New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPVS low throughput #70747

Closed
annProg opened this Issue Nov 7, 2018 · 18 comments

Comments

Projects
None yet
5 participants
@annProg

annProg commented Nov 7, 2018

What happened:
Service A with clusterIP 169.169.14.147 and a domain k8s.dev.example.com, the domain resolve to ingress-nginx. Anothor service B will request A, when use domain it works fine. I think using clusterIP directly will get better performance because it saves some time for dns resolve and ingress-nginx proxy, so I change domain to clusterIP. But the result was not expected, it got worse, very low throughput and high latency.

What you expected to happen:
On the above situation, clusterIP should have better performance than ingress-nginx.

How to reproduce it (as minimally and precisely as possible):
ipvs mode, do test with ab

# clusterIP
ab -c100 -t300 -n20000000 -r http://169.169.14.147/

# domain resolve to ingress-nginx
ab -c100 -t300 -n20000000 -r http://k8s.dev.example.com/

Anything else we need to know?:
I install an ipvs enviroment manually and do some test, and found it possiblly because ipvs will use iptables to realize SNAT

ipvs

Environment:

  • Kubernetes version (use kubectl version): 1.11.1
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release): centos 7.2
  • Kernel (e.g. uname -a): 3.10
  • Install tools: binary
  • Others:

/kind bug

@annProg

This comment has been minimized.

annProg commented Nov 7, 2018

@kubernetes/sig-network-bugs

@k8s-ci-robot k8s-ci-robot added sig/network and removed needs-sig labels Nov 7, 2018

@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Nov 7, 2018

@annProg: Reiterating the mentions to trigger a notification:
@kubernetes/sig-network-bugs

In response to this:

@kubernetes/sig-network-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@annProg

This comment has been minimized.

annProg commented Nov 10, 2018

I try to change some kernel param and found conn_reuse_mode is related to this issue
net.ipv4.vs.conn_reuse_mode , default 1. change to 0,then complete about 3000k request, pretty good performance.

So, should I change the param in production? Does it have any side effects?

@annProg

This comment has been minimized.

annProg commented Nov 10, 2018

/area ipvs

@annProg

This comment has been minimized.

annProg commented Nov 10, 2018

@m1093782566 Can you look at it?

@m1093782566

This comment has been minimized.

Member

m1093782566 commented Nov 10, 2018

I try to change some kernel param and found conn_reuse_mode is related to this issue
net.ipv4.vs.conn_reuse_mode , default 1. change to 0,then complte about 3000k request, pretty good performance.

Yes, seems you find the key to the problem - it's set to 0 in my environment and I have not found any issue so far(it's about IPVS module behavior and we will not find net.ipv4.vs.conn_reuse_mode parameter in old kernels ).

BTW, you can also try to set the ipvs conntrack(https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go#L163) to 0 - it will improve performance as well.

@annProg

This comment has been minimized.

annProg commented Nov 10, 2018

I try to change some kernel param and found conn_reuse_mode is related to this issue
net.ipv4.vs.conn_reuse_mode , default 1. change to 0,then complte about 3000k request, pretty good performance.

Yes, seems you find the key to the problem - it's set to 0 in my environment and I have not found any issue so far(it's about IPVS module behavior and we will not find net.ipv4.vs.conn_reuse_mode parameter in old kernels ).

BTW, you can also set the ipvs conntrack(https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go#L163) to 0 since we are no longer depend on conntrack any more - it will improve performance as well.

Thanks for reply.

I am using kubernetes 1.11.1 , and after set sysctl -w net.ipv4.vs.conntrack=0, clusterIP can not be reachable. Which version do I need ?

@m1093782566

This comment has been minimized.

Member

m1093782566 commented Nov 10, 2018

oops, have you ever configured Cluster-CIDR or iptables masq-all for kube-proxy?

@annProg

This comment has been minimized.

annProg commented Nov 10, 2018

oops, have you ever configure Cluster-CIDR or iptables masq-all for kube-proxy?

Yes, I configure Cluster-CIDR

# cat kube-proxy.config.yaml 
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 10.11.3.18
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 172.20.0.0/14
healthzBindAddress: 10.11.3.18:10256
hostnameOverride: 10.11.3.18
kind: KubeProxyConfiguration
metricsBindAddress: 10.11.3.18:10249
mode: "ipvs"
@m1093782566

This comment has been minimized.

Member

m1093782566 commented Nov 10, 2018

Sorry, Cluster IP still needs the IPVS conntrack and nodeport does not rely on it any more - we are trying to figure it out for Cluster IP.

@annProg

This comment has been minimized.

annProg commented Nov 10, 2018

Sorry, Cluster IP still need the IPVS conntrack and nodeport does not rely on it any more - we are trying to figure it out for Cluster IP.

Got it, thanks!

@m1093782566

This comment has been minimized.

Member

m1093782566 commented Nov 10, 2018

cc @Lion-Wei

Should we tune the kernel parameter in kube-proxy? Given that many users complain about the performance issue, probably the answer is YES.

@dawidmalina

This comment has been minimized.

dawidmalina commented Nov 10, 2018

There is a nice article describing even more kernel parameters to tune ipvs: https://jsravn.com/2017/12/24/ipvs-with-kubernetes-ingress/

@annProg

This comment has been minimized.

annProg commented Nov 10, 2018

Side effect like this comment: cloudnativelabs/kube-router#544 (comment)

hpa_prize

Note: time in picture below is UTC+8.
hpa_price

Abviously every time scaledown , response time will increase.
@m1093782566 Any idea for this?

@m1093782566

This comment has been minimized.

Member

m1093782566 commented Nov 11, 2018

@annProg

Thanks for reporting. Have you ever read the article posted by @dawidmalina? In this article, some ipvs parameters can be tuned, e.g.

net.ipv4.vs.sloppy_tcp=1 allows the passive node to handle preexisting connections (otherwise IPVS will ignore any TCP packets it doesn’t handle the initial SYN for).
net.ipv4.vs.expire_nodest_conn=1 to remove stale/dead connections so further packets will get resets, letting clients quickly retry.
net.ipv4.vs.conn_reuse_mode=2 so IPVS can properly detect terminated connections with direct return. Without this, it’s possible for source port reuse to lead to broken connections.
# be careful because if we set conn_reuse_mode to 2, ipvs will effectively set expire_nodes_conn to 0.
@annProg

This comment has been minimized.

annProg commented Nov 13, 2018

@m1093782566 yes, I had test these parameters

  • net.ipv4.vs.sloppy_tcp not found in centos 7.2(kernel 3.10)
  • net.ipv4.vs.expire_nodest_conn will not affect the result
  • net.ipv4.vs.conn_reuse_mode=2 is similar with 1, will be low throughput

According to this issue #57841 ,I guess it may be the root cause, so I upgrade my setup from 1.11.1 to 1.12.2.
But did not solve this problem, and found a new problem, rs delete failed after scaledown. I had open a new issue to this problem: #70997

@Lion-Wei

This comment has been minimized.

Contributor

Lion-Wei commented Nov 16, 2018

/assign

@Lion-Wei

This comment has been minimized.

Contributor

Lion-Wei commented Nov 16, 2018

Hi, @annProg @dawidmalina @m1093782566 , I raised a pr for this issue. Please check. : )
@jsravn Looking forward for your advise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment