support node local service without SNAT

[GuoyouZhang]

use case: in VNF, traffic come from external client, and go through router ECMP to k8s node. The pod needs to know the external client IP. To meet this case:

1. change iptables rule to avoid SNAT for traffic to the pod on local node
2. when annotation "kube-router.io/service.local" configured on service, only add local pod as VIP back-end.
   Finally, kube-router will advertise external IP for the node, which really has the local pod and can serve the external request without SNAT.

[murali-reddy]

Excellent @GuoyouZhang, changes look good to me. I will test out briefly and merge.

[andrewsykim]

Thanks @GuoyouZhang! Quick question:

Finally, kube-router will advertise external IP for the node, which really has the local pod and can serve the external request without SNAT.

I don't see the necessary changes to ensure only services running the service pods will advertise external IPs, am I missing something?

[murali-reddy]

@andrewsykim It does not address #262

@GuoyouZhang annotating service after service is deployed does not work. i will open seperate issue for that.

[GuoyouZhang]

@andrewsykim
if there is no pod on local Node, traffic will be dropped since we ask ipvs only add local pod as backend by annotation service.local.

[GuoyouZhang]

@murali-reddy
maybe no need to set this annotation dynamically. so it is a limitation but will not impact real use case.

[KJSui]

@GuoyouZhang how to enable the service? Would you like to provide cmd? Thanks.

[murali-reddy]

@KJSui please try adding kube-router.io/service.local=true annotaion in the service definition. it does not work dynamically (i.e annotating after service has been created) yet (it needs to be fixed). so add the annotation in the service defintion.

[KJSui]

@murali-reddy @GuoyouZhang After enabling this service as you suggest, if I did curling the service, the client ip wasn't preserved... If I curl the pod ip directly, I did see client ip appear, do you guys have any idea why the client ip doesn't appear if I curl the external ip of service?

[KJSui]

@murali-reddy the cmd should be kube-router.io/service.local="" not "true" after I tried many times.

[andrewsykim]

@GuoyouZhang @murali-reddy Another consideration I think we missed in this PR is that the iptable rules removing SNAT are being appended instead of replaced. So instead of seeing only

-A POSTROUTING ! -s 10.38.37.0/26 ! -d 10.38.37.0/26 -m ipvs --vdir ORIGINAL --vmethod MASQ -m comment --comment "" -j MASQUERADE

we saw

-A POSTROUTING ! -s <pod-cidr> -m ipvs --vdir ORIGINAL --vmethod MASQ -m comment --comment "" -j MASQUERADE
-A POSTROUTING ! -s <pod-cidr> ! -d <pod-cidr> -m ipvs --vdir ORIGINAL --vmethod MASQ -m comment --comment "" -j MASQUERADE

and I think since the previous rule is first in the chain it takes precedence?

Only a problem when updating kube-router on an existing cluster.

[murali-reddy]

@andrewsykim i think your observation is right. I will fix it.