New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPVS sync always sees all services as changed #53801

Closed
mcginne opened this Issue Oct 12, 2017 · 5 comments

Comments

Projects
None yet
4 participants
@mcginne

mcginne commented Oct 12, 2017

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
I am using IPVS with kube 1.8 and notice that every time the IPVS sync occurs it sees all Services as changed, even though they have not actually changed or moved.

I see the following messages for all Services, and then see the ipvs/iptables updates occurring:

Oct 12 12:43:23 worker-23.test.ibm.com kube-proxy.service[3408395]: I1012 12:43:23.627839 3408395 proxier.go:1288] IPVS service default/httpperf-np-service-njr:http was changed

It looks like the following check around line 1273 in https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go is always returning false:
if appliedVirtualServer == nil || !appliedVirtualServer.Equal(vs) {

I added some extra debug to print out the values for appliedVirtualServer & vs, and it showed the following:

Oct 12 12:43:23 worker-23.test.ibm.com kube-proxy.service[3408395]: I1012 12:43:23.627805 3408395 proxier.go:1273] appliedVirtualServer: 10.143.115.226,TCP,30089,rr,2,0
Oct 12 12:43:23 worker-23.test.ibm.com kube-proxy.service[3408395]: I1012 12:43:23.627830 3408395 proxier.go:1274] vs: 10.143.115.226,TCP,30089,rr,0,0
Oct 12 12:43:23 worker-23.test.ibm.com kube-proxy.service[3408395]: I1012 12:43:23.627839 3408395 proxier.go:1288] IPVS service default/httpperf-np-service-njr:http was changed

It is the utilipvs.VirtualServer.Flags field that is always different. appliedVirtualServer is always 2 whereas vs is always 0.

What you expected to happen:
ipvs & iptable rules only updated when Services have changed

How to reproduce it (as minimally and precisely as possible):
Start kube-proxy with the following parameters:

          --proxy-mode=ipvs \
          --ipvs-min-sync-period 60s \
          --ipvs-sync-period 300s \
          --feature-gates=SupportIPVSProxyMode=true \
          --v=4

Create a service (Mine is NodePort - not sure if this affects all services)
View the logs, and on every sync you will see it believes the Service has changed.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): Kubernetes v1.8.0-1+edd813e3ac8824
  • Cloud provider or hardware configuration**: VSIs running on IBM Softlayer
  • OS (e.g. from /etc/os-release):Ubuntu 16.04.2 LTS (Xenial)
  • Kernel (e.g. uname -a):4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Others:

/sig network
/area kube-proxy

@cmluciano

This comment has been minimized.

Show comment
Hide comment
@cmluciano
Member

cmluciano commented Oct 17, 2017

xref: #51602

cc @m1093782566

@cmluciano

This comment has been minimized.

Show comment
Hide comment
@cmluciano
Member

cmluciano commented Oct 17, 2017

/assign @m1093782566

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Oct 18, 2017

Member

Thanks for reporting it. @mcginne

It is the utilipvs.VirtualServer.Flags field that is always different. appliedVirtualServer is always 2 whereas vs is always 0.

That's true and it's fixed in #52330. Please check it again?

Member

m1093782566 commented Oct 18, 2017

Thanks for reporting it. @mcginne

It is the utilipvs.VirtualServer.Flags field that is always different. appliedVirtualServer is always 2 whereas vs is always 0.

That's true and it's fixed in #52330. Please check it again?

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Nov 20, 2017

Member

/area ipvs

Member

m1093782566 commented Nov 20, 2017

/area ipvs

@m1093782566

This comment has been minimized.

Show comment
Hide comment
@m1093782566

m1093782566 Dec 19, 2017

Member

I believe it's fixed in v1.9

I am going to close this issue now. Please re-open if it still persist.

/close

Member

m1093782566 commented Dec 19, 2017

I believe it's fixed in v1.9

I am going to close this issue now. Please re-open if it still persist.

/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment