Add ServiceEntry but Could not resolve host

[Marco-Zheng]

Bug Description

1、add ServiceEntry as below

kind: ServiceEntry
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: aeraki-org-apache-dubbo-samples-basic-api-demoservice
  namespace: dubbo
  selfLink: >-
    /apis/networking.istio.io/v1alpha3/namespaces/dubbo/serviceentries/aeraki-org-apache-dubbo-samples-basic-api-demoservice
  uid: 6e5c918f-164e-4c7e-a6d0-37088b82ac1f
  resourceVersion: '3011204340'
  generation: 2
  creationTimestamp: '2021-11-10T08:36:48Z'
  labels:
    manager: aeraki
    registry: dubbo2istio
  annotations:
    interface: org.apache.dubbo.samples.basic.api.DemoService
    workloadSelector: dubbo-sample-provider
spec:
  hosts:
    - org.apache.dubbo.samples.basic.api.demoservice
  addresses:
    - 240.240.0.30
  ports:
    - number: 20880
      protocol: tcp
      name: tcp-dubbo
      targetPort: 20880
  location: MESH_INTERNAL
  resolution: STATIC
  endpoints:
    - address: 172.31.0.253
      ports:
        tcp-dubbo: 20880
      labels:
        anyhost: 'true'
        application: dubbo-sample-provider
        bean.name: org.apache.dubbo.samples.basic.api.DemoService
        deprecated: 'false'
        dubbo: 2.0.2
        dynamic: 'true'
        generic: 'false'
        interface: org.apache.dubbo.samples.basic.api.DemoService
        methods: testVoid-sayHello
        pid: '6'
        register: 'true'
        registryName: default
        release: 1.0-SNAPSHOT
        revision: 1.0-SNAPSHOT
        service_group: batchjob
        side: provider
        timestamp: '1636531107615'
        version: v2
      locality: bj/800005
      serviceAccount: default
    - address: 172.31.0.254
      ports:
        tcp-dubbo: 20880
      labels:
        anyhost: 'true'
        application: dubbo-sample-provider
        bean.name: org.apache.dubbo.samples.basic.api.DemoService
        deprecated: 'false'
        dubbo: 2.0.2
        dynamic: 'true'
        generic: 'false'
        interface: org.apache.dubbo.samples.basic.api.DemoService
        methods: testVoid-sayHello
        pid: '6'
        register: 'true'
        registryName: default
        release: 1.0-SNAPSHOT
        revision: 1.0-SNAPSHOT
        service_group: user
        side: provider
        timestamp: '1636531107124'
        version: v1
      locality: bj/800002
      serviceAccount: default
status: {}

2、invoke nslookup, can't find the host

Version

istio version:1.10

Additional Information

bug-report.tar.gz

[kebe7jun]

do you enable DNS proxy? https://istio.io/latest/docs/ops/configuration/traffic-management/dns-proxy/

[Marco-Zheng]

@kebe7jun yes, i add below istiooperator

---
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default
  values:
    global:
      logging:
        level: default:debug
  meshConfig:
    accessLogFile: /dev/stdout
    defaultConfig:
      proxyMetadata:
        ISTIO_META_DNS_CAPTURE: "true"
      proxyStatsMatcher:
        inclusionPrefixes:
          - thrift
          - dubbo
          - kafka
          - meta_protocol
        inclusionRegexps:
          - .*dubbo.*
          - .*thrift.*
          - .*kafka.*
          - .*zookeeper.*
          - .*meta_protocol.*
  components:
    pilot:
      hub: istio
      tag: 1.10.0

[kebe7jun]

check your nds info by:

curl $ISTIOD_IP:15014/ndsz?proxyID=PODNAME.NAMESPACE

and also check your dns proxy iptables rules exists :

kubectl logs sleep-54584cfcb4-9grqh istio-init | grep 15053

[Marco-Zheng]

@kebe7jun i run the first command in istiod pod, but failed, if there any wrong with my action?

[Marco-Zheng]

@kebe7jun the second command result as below

[root@VM-11-7-centos dubbo-zk-test]# kubectl logs dubbo-sample-consumer-f766459fc-hmlpt istio-init -n dubbo | grep 15053
-A ISTIO_OUTPUT -p tcp --dport 53 -d 172.31.255.216/32 -j REDIRECT --to-ports 15053
-A OUTPUT -p udp --dport 53 -d 172.31.255.216/32 -j REDIRECT --to-port 15053
-A OUTPUT -d 172.31.255.216/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_OUTPUT -d 172.31.255.216/32 -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 15053

[Marco-Zheng]

@kebe7jun the correct command maybe curl $ISTIOD_IP:15014/ndsz?proxyID=PODNAME.NAMESPACE ? i used the command and got below result(actually i used my own sider car proxy not enovy)

[root@VM-11-7-centos dubbo-zk-test]# curl 172.31.254.202:15014/debug/ndsz?proxyID=dubbo-sample-consumer.dubbo
Proxy not connected to this Pilot instance. It may be connected to another instance

[kebe7jun]

@kebe7jun the correct command maybe curl $ISTIOD_IP:15014/ndsz?proxyID=PODNAME.NAMESPACE ? i used the command and got below result(actually i used my own sider car proxy not enovy)

[root@VM-11-7-centos dubbo-zk-test]# curl 172.31.254.202:15014/debug/ndsz?proxyID=dubbo-sample-consumer.dubbo

Proxy not connected to this Pilot instance. It may be connected to another instance

istioctl ps can fetch all proxies（with proxyID）。

[Marco-Zheng]

@kebe7jun Thanks, i got the result by curl $ISTIOD_IP:15014/debug/ndsz?proxyID=dubbo-sample-consumer-f766459fc-hmlpt.dubbo as below, please have a check

[{
  "@type": "type.googleapis.com/istio.networking.nds.v1.NameTable",
  "table": {
    "hpa-metrics-service.kube-system.svc.cluster.local": {
      "ips": [
        "172.31.250.131"
      ],
      "registry": "Kubernetes",
      "shortname": "hpa-metrics-service",
      "namespace": "kube-system"
    },
    "istio-ingressgateway.istio-system.svc.cluster.local": {
      "ips": [
        "172.31.249.196"
      ],
      "registry": "Kubernetes",
      "shortname": "istio-ingressgateway",
      "namespace": "istio-system"
    },
    "istiod.istio-system.svc.cluster.local": {
      "ips": [
        "172.31.254.202"
      ],
      "registry": "Kubernetes",
      "shortname": "istiod",
      "namespace": "istio-system"
    },
    "kube-dns.kube-system.svc.cluster.local": {
      "ips": [
        "172.31.255.216"
      ],
      "registry": "Kubernetes",
      "shortname": "kube-dns",
      "namespace": "kube-system"
    },
    "kube-user.default.svc.cluster.local": {
      "ips": [
        "172.31.248.30"
      ],
      "registry": "Kubernetes",
      "shortname": "kube-user",
      "namespace": "default"
    },
    "kubernetes.default.svc.cluster.local": {
      "ips": [
        "172.31.248.1"
      ],
      "registry": "Kubernetes",
      "shortname": "kubernetes",
      "namespace": "default"
    },
    "kuboard.kube-system.svc.cluster.local": {
      "ips": [
        "172.31.255.123"
      ],
      "registry": "Kubernetes",
      "shortname": "kuboard",
      "namespace": "kube-system"
    },
    "metrics-service.kube-system.svc.cluster.local": {
      "ips": [
        "172.31.251.77"
      ],
      "registry": "Kubernetes",
      "shortname": "metrics-service",
      "namespace": "kube-system"
    },
    "zookeeper.dubbo.svc.cluster.local": {
      "ips": [
        "172.31.251.82"
      ],
      "registry": "Kubernetes",
      "shortname": "zookeeper",
      "namespace": "dubbo"
    }
  }
}

[kebe7jun]

It seems that your proxies are not istio official proxies, the istiod did not send ServiceEntry's name table to proxy, which makes the dns proxy can not resolve ServiceEntry's hosts.

[Marco-Zheng]

It seems that your proxies are not istio official proxies, the istiod did not send ServiceEntry's name table to proxy, which makes the dns proxy can not resolve ServiceEntry's hosts.

@kebe7jun Thanks for your kindly reply, how can i get ServiceEntry's name table from istiod? if there any example

[kebe7jun]

Since you are not using the official istio version, we can't confirm the exact problem. You can try to debug the code in the name table generation part of istiod or try to upgrade the istio version to see if it can solve the problem.

istio/pkg/dns/server/name_table.go

Lines 40 to 51 in 8aea949

	func BuildNameTable(cfg Config) *dnsProto.NameTable {
	if cfg.Node.Type != model.SidecarProxy {
	// DNS resolution is only for sidecars
	return nil
	}
	out := &dnsProto.NameTable{
	Table: make(map[string]*dnsProto.NameTable_NameInfo),
	}
	for _, svc := range cfg.Push.Services(cfg.Node) {
	svcAddress := svc.GetAddressForProxy(cfg.Node)

[Marco-Zheng]

@kebe7jun Many thanks!

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-11-11. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.