externalTrafficPolicy:Local not working in Vnets outside the 10.0.0.0/8 range

[erjosito]

What happened:

1. Created an 11.5 AKS cluster with Azure CNI plugin in an existing vnet (vnet 10.13.0.0/16, subnet 10.13.76.0/24)
2. Deployed some pods and a LoadBalancer-Type service with the property externalTrafficPolicy: "Local"
3. Verified that the client IP address is preserved
4. Do the same in a different vnet (192.168.0.0/16, subnet 192.168.76.0/24)
5. The IP address is SNATted to the node's IP

What you expected to happen:

The behavior of the externalTrafficPolicy should be independent of the IP prefix of the subnet

How to reproduce it (as minimally and precisely as possible):

1. Create vnet/subnet in 10.0.0.0/8 space

az network vnet create -g akstest -n aksVnet10 --address-prefix 10.13.0.0/16 --subnet-name aks --subnet-prefix 10.13.76.0/24
subnetid=$(az network vnet subnet show -g akstest --vnet-name aksVnet10 -n aks --query id -o tsv)

2. Create AKS cluster

az aks create -g akstest -n myAks10 -c 1 --generate-ssh-keys -s Standard_B2ms -k 1.11.5 --network-plugin azure --vnet-subnet-id $subnetid --no-wait

3. Create file for test app (whereami.yaml)

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: whereami
spec:
  replicas: 2
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: whereami
    spec:
      containers:
      - name: whereami
        image: erjosito/whereami:1.3
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whereami
spec:
  type: LoadBalancer
  externalTrafficPolicy: "Local"
  ports:
  - port: 80
  selector:
    app: whereami

4. Deploy app

az aks get-credentials -g akstest -n myAks10
kubectl apply -f ./whereami.yaml

5. Browse to app and verify the client IP

$ k get svc
NAME           TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
kubernetes     ClusterIP      10.0.0.1       <none>          443/TCP        31m
whereami       LoadBalancer   10.0.170.186   13.81.9.209     80:30552/TCP   24m
$ curl 13.81.9.209

6. Deploy a second cluster in a 192.168.0.0 IP address space

az network vnet create -g akstest -n aksVnet192 --address-prefix 192.168.0.0/16 --subnet-name aks --subnet-prefix 192.168.76.0/24
subnetid=$(az network vnet subnet show -g akstest --vnet-name aksVnet192 -n aks --query id -o tsv)
az aks create -g akstest -n myAks192 -c 1 --generate-ssh-keys -s Standard_B2ms -k 1.11.5 --network-plugin azure --vnet-subnet-id $subnetid --no-wait
az aks get-credentials -g akstest -n myAks192
kubectl apply -f ./whereami.yaml

7. Verify that client IP address is not preserved there

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

$ k version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersi
on:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• Size of cluster (how many worker nodes are in the cluster?)
  1

• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.)
  None

• Others:

[brendandburns]

I suspect that this is not actually a kubernetes bug (though it's always possible)

Can you run tcpdump on one of the host machines and see what the source IP address is when it is received by the VM?

My suspicion is that there is SNAT happening somewhere prior to the machine.

Alternately, you can run iptables-save on the host machine and see.

For understanding, when you choose "local" this is what happens:

• LB health-check is programmed for only machines where the pod lives
• IP Tables rules are programmed to do destination IP rewrite only (instead of SNAT like they normally do)
• Traffic goes to local Pod IP address, and then straight back to client

Let us know how that goes.

[JunSun17]

Duplicate of #607

[JunSun17]

This is dupe of #607, it has been fixed now.