NodePort only responding on node where pod is running

[sfitts]

What happened:
I deployed a cluster using kubeadm and Calico. The command line for the cluster creation was:

kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address <private IP addr of master> --apiserver-cert-extra-sans <public IP addr of master>

I then followed the Calico instructions and ran:

kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml 
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Lastly I joined 4 worker to the cluster using the join command generated by the first step.

I then installed the Nginx controller configuring it to use NodePort. The resulting service definition is:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-10-23T23:04:23Z
  labels:
    app: nginx-ingress
    chart: nginx-ingress-0.29.0
    component: controller
    heritage: Tiller
    release: alieast-lb
  name: alieast-lb-nginx-ingress-controller
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    nodePort: 30400
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 31874
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: nginx-ingress
    component: controller
    release: alieast-lb
  sessionAffinity: None
  type: NodePort

This service is only reachable on the node where the nginx controller pod is running.

What you expected to happen:

The service should be reachable via all nodes in the cluster.

How to reproduce it (as minimally and precisely as possible):

The above steps should do the trick (though any simple NodePort service + Pod should do it). You'll need at least 2 workers to confirm that only one of them provides access to the service.

Anything else we need to know?:

All other communications in the cluster appear to be working as expected. I have multiple pods deployed which communicate with each other via service names and they show no issues. The only problem appears to be the one with NodePort.

FWIW, I tried the iptables workaround described in #58908 to no avail.

Environment:

• Kubernetes version (use kubectl version):

kube@kube-master:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

Bare metal K8s running on AliCloud ECS instances.

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"

• Kernel (e.g. uname -a):

Linux kube-master 4.4.0-117-generic #141-Ubuntu SMP Tue Mar 13 11:58:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

kubeadm

kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:43:08Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

• Others:

Calico 3.1
Helm 2.9.1

/king bug

[sfitts]

/sig network

[MrHohn]

@sfitts Set externalTrafficPolicy: Cluster should achieve what you need?

Ref https://kubernetes.io/docs/tutorials/services/source-ip/.

[sfitts]

@MrHohn thanks -- that caused me to take a closer look at this -- https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#over-a-nodeport-service. If I want to use Local (to preserve the client IP) then I'll need to run the Nginx controller on all nodes (or at least all nodes in the balancing set). Or I can get the routing, but lose the IP preservation.

Thanks -- closing.

[SmartLyu]

Please , can you tell me how you solved this problem. I meet the same problem.

[MrHohn]

@SmartLyu Could you elaborate? Are you having issue with externalTrafficPolicy?

[sfitts]

In my case the use of externalTrafficPolicy: Local means that by design the NodePort routing doesn't occur. To work around that you have to run a copy of the Nginx controller on every node -- which you can do by making it a DaemonSet.

[mbrekhov]

In my case, I configured a Kubernetes cluster on Vultr cloud. The instances on Vultr have 2 NICs - private and public-facing.
The problem was with the Flannel Daemon set, which by default, used the public-facing NIC where UDP traffic was blocked on the firewall side.
So, I simply added '--iface=enp0s8' (with the name of my private-facing NIC) to flannel as described here: https://stackoverflow.com/a/48755233."