Unable to resolve helloworld-go svc on minikube

[devguyio]

Expected Behavior

On minikube, after deploying helloworld-go , you should be able to lookup helloworld-go and helloword-go.default.svc.cluster.local from any pod inside the cluster. So this should succeed

nslookup helloworld-go

Actual Behavior

Deploying the following pod and exec into the pod

apiVersion: v1
kind: Pod
metadata:
  annotations:
    sidecar.istio.io/inject: "true"
  labels:
    run: dnsutils
    app: dnsutils
  name: dnsutils
  namespace: default
spec:
  containers:
  - image: tutum/dnsutils
    imagePullPolicy: IfNotPresent
    name: dnsutils
    resources:
      requests:
        cpu: 100m
    stdin: true
    stdinOnce: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
  dnsPolicy: ClusterFirst
  restartPolicy: Never

Using dig or nslookup for helloworld-go fails

root@dnsutils:/# nslookup helloworld-go
Server:		10.96.0.10
Address:	10.96.0.10#53

** server can't find helloworld-go: NXDOMAIN

Steps to Reproduce the Problem

1. Install knative serving on minikube following https://github.com/knative/docs/blob/master/install/Knative-with-Minikube.md
2. Running helloworld-go following https://github.com/knative/docs/tree/master/serving/samples/helloworld-go
3. Applying the yaml above of tutum/dnsutils inside the cluster to deploy
4. Execing into the dnsutil pod sh

➜  ~ kubectl exec -it dnsutils -- sh

5. Doing nslookup on helloworld-go or helloworld-go.default.svc.cluster.local fails.

Additional Info

The K8s service

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2019-02-01T17:59:50Z
  name: helloworld-go
  namespace: default
  ownerReferences:
  - apiVersion: serving.knative.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Route
    name: helloworld-go
    uid: f171b63b-264a-11e9-8b54-962d6019ad6b
  resourceVersion: "3109"
  selfLink: /api/v1/namespaces/default/services/helloworld-go
  uid: 2baba297-264b-11e9-8b54-962d6019ad6b
spec:
  externalName: istio-ingressgateway.istio-system.svc.cluster.local
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

The knative route

apiVersion: serving.knative.dev/v1alpha1
kind: Route
metadata:
  creationTimestamp: 2019-02-01T17:58:13Z
  generation: 1
  labels:
    serving.knative.dev/service: helloworld-go
  name: helloworld-go
  namespace: default
  ownerReferences:
  - apiVersion: serving.knative.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Service
    name: helloworld-go
    uid: f16b59a2-264a-11e9-8b54-962d6019ad6b
  resourceVersion: "3110"
  selfLink: /apis/serving.knative.dev/v1alpha1/namespaces/default/routes/helloworld-go
  uid: f171b63b-264a-11e9-8b54-962d6019ad6b
spec:
  generation: 1
  traffic:
  - configurationName: helloworld-go
    percent: 100
status:
  address:
    hostname: helloworld-go.default.svc.cluster.local
  conditions:
  - lastTransitionTime: 2019-02-01T17:59:50Z
    severity: Error
    status: "True"
    type: AllTrafficAssigned
  - lastTransitionTime: 2019-02-01T17:59:50Z
    severity: Error
    status: "True"
    type: IngressReady
  - lastTransitionTime: 2019-02-01T17:59:50Z
    severity: Error
    status: "True"
    type: Ready
  domain: helloworld-go.default.example.com
  domainInternal: helloworld-go.default.svc.cluster.local
  traffic:
  - percent: 100
    revisionName: helloworld-go-00001

[mattmoor]

I assume it isn't just a typo: nslookup hellowrold-go 😅

[devguyio]

@mattmoor omg NO! The typo was in my poor faking a copy/paste at 1 am skills 😄 I rechecked just to make sure and attached a screenshot!

[devguyio]

seems to be an issue with CoreDNS. minikube v0.33.1 iso on mac https://storage.googleapis.com/minikube/iso/minikube-v0.33.1.iso ships with CoreDNS 1.1.3 and after manually editing the deployment to latest CoreDNS 1.3.1 everything worked.

Workaround

1. Edit CoreDNS deployment to latest CoreDNS

kubectl -n kube-system edit deployment coredns

2. Change the coredns container image to image: k8s.gcr.io/coredns:1.3.1

[devguyio]

Root cause found coredns/coredns#2038 which seems to be fixed in CoreDNS v 1.2.0 and earliest healthy MiniKube is for Kubernetes 1.12. kubernetes/kubernetes@989f666

[mattmoor]

Awesome, thanks!