Istio MultiPrimary Deployment cannot resolve headless service in remote cluster

[nirvanatao]

I configured 2 Kubernetes cluster as a singl service mesh using Istio MultiPrimary Deployment, however I found that the headless services could not be resolved from remote cluster, I'm using istio-1.8.1

[hzxuzhonghu]

From the implement details, istiod can watch service/endpoints from the other clusters, it is same for any kind of service.

[nirvanatao]

From the implement details, istiod can watch service/endpoints from the other clusters, it is same for any kind of service.

Hi Zhonghu, thanks for your reply, do you mean that I just need to create a stub service for headless service? do you have any sample?

Here is what I did it but not works:
created a test namespace with istio-injection=enabled in each clusters

kubectl create ns test  --context cluster-0
kubectl create ns test  --context cluster-1
kubectl label ns test istio-injection=enabled --context cluster-0
kubectl label ns test istio-injection=enabled --context cluster-1

and then I applied below yaml in cluster-0:

apiVersion: v1
kind: Service
metadata:
  name: nginx-headless
  namespace: test
spec:
  ports:
  - port: 80
    name: web
  selector:
    app: nginx
  clusterIP: None
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-1
  namespace: test
  labels:
    app: nginx
spec:
  hostname: nginx-1
  subdomain: nginx-headless
  containers:
  - image: k8s.gcr.io/nginx-slim:0.8
    name: nginx
    ports:
    - containerPort: 80
      name: web

and applied below yaml in cluster-1:

apiVersion: v1
kind: Service
metadata:
  name: nginx-headless
  namespace: test
spec:
  ports:
  - port: 80
    name: web
  selector:
    app: nginx
  clusterIP: None
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-2
  namespace: test
  labels:
    app: nginx
spec:
  hostname: nginx-2
  subdomain: nginx-headless
  containers:
  - image: k8s.gcr.io/nginx-slim:0.8
    name: nginx
    ports:
    - containerPort: 80
      name: web

what I expected is that I can curl http://nginx-2.nginx-headless.test.svc.cluster.local from both cluster-0 and cluster-1, however, when I tried to curl http://nginx-2.nginx-headless.test.svc.cluster.local from cluster-0 it always fails with error Name or service not known

ping from cluster-0:

root@dev:/# ping nginx-1.nginx-headless.test.svc.cluster.local
PING nginx-1.nginx-headless.test.svc.cluster.local (172.19.6.53) 56(84) bytes of data.
64 bytes from nginx-1.nginx-headless.test.svc.cluster.local (172.19.6.53): icmp_seq=1 ttl=64 time=6.96 ms
--- nginx-1.nginx-headless.test.svc.cluster.local ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 6.961/6.961/6.961/0.000 ms
root@dev:/# ping nginx-2.nginx-headless.test.svc.cluster.local
ping: nginx-2.nginx-headless.test: Name or service not known

ping from cluster-1:

root@dev:/#  ping nginx-2.nginx-headless.test.svc.cluster.local
PING nginx-2.nginx-headless.test.svc.cluster.local (172.19.10.150) 56(84) bytes of data.
64 bytes from nginx-2.nginx-headless.test.svc.cluster.local (172.19.10.150): icmp_seq=1 ttl=64 time=5.24 ms
64 bytes from nginx-2.nginx-headless.test.svc.cluster.local (172.19.10.150): icmp_seq=2 ttl=64 time=1.31 ms
64 bytes from nginx-2.nginx-headless.test.svc.cluster.local (172.19.10.150): icmp_seq=3 ttl=64 time=1.02 ms
--- nginx-2.nginx-headless.test.svc.cluster.local ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.021/2.528/5.247/1.926 ms
root@dev:/# ping nginx-1.nginx-headless.test.svc.cluster.local
ping: nginx-1.nginx-headless.test.svc.cluster.local: Name or service not known
root@dev:/#

[hzxuzhonghu]

IC, the dns of cluster1 can not resolve nginx-2.nginx-headless.test.svc.cluster.local in cluster2. This is expected

[nirvanatao]

IC, the dns of cluster1 can not resolve nginx-2.nginx-headless.test.svc.cluster.local in cluster2. This is expected

Ok .... I want to build a logical Zookeeper cluster across 3 Kubernetes clusters, so that means it's impossible to do that with Istio?

[stevenctl]

There was an open issue here that got auto-closed: #27342. @hzxuzhonghu, maybe it's possible to create a workaround using in-agent DNS, or does istio not want to be responsible for such a thing?

[ramaraochavali]

@stevenctl

maybe it's possible to create a workaround using in-agent DNS

What kind of workaround are you thinking we can do in agent DNS? I discusses offline with @hzxuzhonghu , I think the main problem is there is no way to identify which ipaddress in the endpoint response of the service belongs with individual stateful pod.

[irajdeep]

Hi @nirvanatao did you get this to working finally? I stumbled upon this: https://discuss.istio.io/t/istio-1-5-and-headless-services/5910 and created a ServiceEntry for the pods but still couldn't get it to working state.

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-01-18. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

[stevenctl]

not stale: #32515