Weave-Net: client_address is 10.* despite externalTrafficPolicy: Local

[kachkaev]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

(or a missing note / caveat in https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip)

What happened:

I noticed that my ingress controller (traefik) was providing wrong values for the X-Forwarded-For HTTP header. It turned out that even when traefik was bypassed, I was not able to see a real remote client IP in the HTTP headers. Running gcr.io/google_containers/echoserver:1.8 as a NodePort or LoadBalancer service consistently reported client_address=10.32.0.1 in the requests (same as X-Forwarded-For assigned by traefik earlier).

I read Using Source IP, which suggested that such a problem could exist in multi-node clusters. Setting externalTrafficPolicy: Local for a service of type NodePort or LoadBalancer was supposed to be a cure, however, this did not help me unfortunately. Even a diligent repetition of all the steps mentioned in the topic did not allow me to see a real remote IP address.

My k8s cluster is supersimple – it has been created with kubeadm and sits in a single public-facing KVM. I've been using docker on the same machines previously and never had issues with figuring out a remote IP inside the containers. Maybe the issue is somehow related to the weave network that I'm using or there is something else with the setup that breaks the expected behavior?

What can you guys suggest?

What you expected to happen:

I expected remote_address to be equal ty my client's IP address, i.e. the same one I see when tcpdump -nnn -i any port $SERVICEPORT on the server.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"201
7-08-17T17:03:51Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"922a86cfcd65915a9b2f69f3f193b8907d741d9c", GitTreeState:"clean", BuildDate:"201
7-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration**:

A single-node cluster (Ubuntu KVM) at https://firstvds.ru/

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a):

4.4.0-83-generic

• Install tools:

kubeadm

• Others:

Using weave as a pod network. ps aux | grep -F 10.32 returns:

/home/weave/weaver
--port=6783
--datapath=datapath
--host-root=/host
--http-addr=127.0.0.1:6784
--status-addr=0.0.0.0:6782
--docker-api=
--no-dns
--db-prefix=/weavedb/weave-net
--ipalloc-range=10.32.0.0/12                       <----
--nickname=kachkaev
--ipalloc-init consensus=1
--conn-limit=30
--expect-npc 91.240.87.56

ps aux | grep api

kube-apiserver 
--service-node-port-range=80-10000 
--experimental-bootstrap-token-auth=true 
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname 
--requestheader-group-headers=X-Remote-Group 
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt 
--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key 
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key 
--insecure-port=0 
--requestheader-username-headers=X-Remote-User 
--requestheader-allowed-names=front-proxy-client 
--service-account-key-file=/etc/kubernetes/pki/sa.pub 
--client-ca-file=/etc/kubernetes/pki/ca.crt 
--allow-privileged=true 
--service-cluster-ip-range=10.96.0.0/12 
--secure-port=16443 
--requestheader-extra-headers-prefix=X-Remote-Extra- 
--tls-cert-file=/etc/kubernetes/pki/apiserver.crt 
--tls-private-key-file=/etc/kubernetes/pki/apiserver.key 
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt 
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt 
--admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota 
--authorization-mode=Node,RBAC 
--advertise-address=42.42.42.42 
--etcd-servers=http://127.0.0.1:2379

[kachkaev]

/sig network

[kachkaev]

/kind bug

[MrHohn]

I think Weave-Net support for externalTrafficPolicy=Local is not added yet.

• Weave-Net on Kubernetes should support externalTrafficPolicy: Local weaveworks/weave#2924
• Unexpected behaviour using "OnlyLocal" annotation on NodePort with 1.6.1 and Weave-Net 1.6 #44963

[kachkaev]

Thanks @MrHohn. Do you think this issue should be closed or kept open until the fix upstream?

[MrHohn]

@kachkaev Make sense to keep it open so folks won't open duplicate issue. Do you mind changing the title a bit (to indicate weave-net)?

[kachkaev]

@MrHohn done. I brought up my cluster using the kubeadm tutorial and other newbies like me are likely to take the same path. Just wondering if it's worth warning people in a weave net tab that they might get the same issue? What other networks might have malfunctioning externalTrafficPolicy?

Turns out that in order to get the remote IPs, I'll have to tear down my kubeadm instance and launch a new one on the same machine (because of --pod-network-cidr=10.244.0.0/16 for flannel). Not worth the hassle for me since I'm just experimenting with the cluster anyway, but others my discover this surprise too late. WDYT?

[MrHohn]

Just wondering if it's worth warning people in a weave net tab that they might get the same issue? What other networks might have malfunctioning externalTrafficPolicy?

Certainly make sense to me if we could find a proper way to warn people.

I'm a bit worried about putting warning directly on "weave net tab". From https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network I see 4 other CNI plugins. Putting warning only on weave tab may imply externalTrafficPolicy works on others, which we haven't verified. At least it used to be incompatible with flannel as well, but was recently fixed by flannel-io/flannel#745.

cc @kubernetes/sig-docs-maintainers for advice.

[MrHohn]

FWIW, sent kubernetes/website#5190 to include couple known issues in the externalTrafficPolicy documentation.

[cmluciano]

Anymore involvement needed on the k8s side here or should this report now be moved to a Weave issue?

[kachkaev]

Perhaps for people who'll take a similar path as me, it'd be great to mention the caveat at https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

Being a non-sysadmin person, I had no idea of what a pod network was at the time of reading, so for me it was just another step to get through ASAP. Because Weave network does not require --pod-network-cidr= etc., if feels the easiest option. For me it'd be great to know what the consequences would be at that point and many others might find themselves in the same boat.

[Scukerman]

Ran into the same issue

[page-fault-in-nonpaged-area]

I think I am having the same issue!! Will unplug weave for a replacement and see if it fixes it. Anyone know a network that will definitely do externalTrafficPolicy: Local 100% for sure?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[kachkaev]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[RRAlex]

/remove-lifecycle stale

[RRAlex]

According to the release note, this feature has been implemented in 2.4.0.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[cmluciano]

This issue should be closed since it is specific to a CNI plugin and further discussion should happen in a WeaveNet issue

/close

[k8s-ci-robot]

@cmluciano: Closing this issue.

In response to this:

This issue should be closed since it is specific to a CNI plugin and further discussion should happen in a WeaveNet issue

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.