Can't Reach StatefulSet Pods Via Stable Network ID #2266
Description
Bug Report
What is the issue?
A meshed client isn't able to reach a meshed statefulset pod via the pod's stable network ID. This works if the client isn't meshed.
How can it be reproduced?
Deploy a meshed statefulset nginx
cat <<EOF | linkerd inject - | kubectl apply -f -
kind: Service
apiVersion: v1
metadata:
name: nginx
labels:
app: nginx
spec:
clusterIP: None
selector:
app: nginx
ports:
- name: http
port: 80
targetPort: http
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 3
serviceName: nginx
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: http
EOF
Use a curl pod to reach nginx:
$ kubectl run curl --image=appropriate/curl --restart=Never --command -- sleep 3600
# this works
$ kubectl exec curl -- curl --silent nginx-0.nginx.default.svc.cluster.local
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
Now meshed the curl pod and notice that it can't reach the nginx pod:
$ kubectl run curl --image=appropriate/curl --restart=Never --dry-run -o yaml --command -- sleep 3600 | linkerd inject - | kubectl apply -f -
# stuck
$ kubectl exec curl -c curl -- curl --silent nginx-0.nginx.default.svc.cluster.local
# this works
$ kubectl exec curl -c curl -- curl --silent nginx.default
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
...
Logs, error output, etc
Proxy logs from the curl client:
WARN admin={bg=resolver} linkerd2_proxy::control::destination::background::destination_set Destination.Get stream errored for NameAddr { name: "nginx-0.nginx.default.svc.cluster.local", port: 80 }: Grpc(Status { code: Unknown, message: "resolver [&{k8sDNSZoneLabels:[] controllerNamespace:linkerd endpointsWatcher:0xc4205e9f20 profileWatcher:0xc4203f2ae0}] found error resolving host [nginx-0.nginx.default.svc.cluster.local] port [80]: not a service: nginx-0.nginx.default.svc.cluster.local" })
WARN admin={bg=resolver} linkerd2_proxy::control::destination::background::destination_set Destination.Get stream errored for NameAddr { name: "nginx-0.nginx.default.svc.cluster.local", port: 80 }: Grpc(Status { code: Unknown, message: "resolver [&{k8sDNSZoneLabels:[] controllerNamespace:linkerd endpointsWatcher:0xc4205e9f20 profileWatcher:0xc4203f2ae0}] found error resolving host [nginx-0.nginx.default.svc.cluster.local] port [80]: not a service: nginx-0.nginx.default.svc.cluster.local" })
linkerd check output
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
linkerd-existence
-----------------
√ control plane namespace exists
√ controller pod is running
√ can initialize the client
√ can query the control plane API
linkerd-api
-----------
√ control plane pods are ready
√ can query the control plane API
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
linkerd-service-profile
-----------------------
√ no invalid service profiles
linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date
control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match
Environment
- Kubernetes Version: v1.13.2
- Cluster Environment: Minikube
- Host OS: Ubuntu 16.04
- Linkerd version: edge-19.2.2