Pods and headless Services don't get DNS A records without at least one service port

[seh]

Observation

Per earlier discussion in #70, the documentation for kube-dns says that it will publish A records for pods so long as a headless Service named with the same subdomain exists in the same namespace. However, if the Service has no ports, kube-dns publishes no such records.

What follows is an example to demonstrate this discrepancy.

Example

We create the following objects in a given namespace:

• A headless Service named "sub"
  Initially, the Service exposes one port with value 80 and called "nonexistent," since the image used here doesn't have any servers listening, to point out that it doesn't matter whether the advertised port actually allows connecting to anything insider the container.
• Some number of pods.
  Though a single pod is sufficient, here we create three pods selected by the aforementioned Service, each with a single container running the busybox image, each also situated within the subdomain "sub," matching the name of the Service:
  • busybox-1
  • busybox-2
  • busybox-3

apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: Pod
  metadata: &metadata
    name: busybox-1
    labels:
      app: &app busybox
  spec: &spec
    hostname: host-1
    subdomain: &subdomain sub
    containers:
    - name: busybox
      image: busybox
      command:
      - sleep
      - "3600"
- apiVersion: v1
  kind: Pod
  metadata:
    <<: *metadata
    name: busybox-2
  spec:
    <<: *spec
    hostname: host-2
- apiVersion: v1
  kind: Pod
  metadata:
    <<: *metadata
    name: busybox-3
  spec:
    <<: *spec
    hostname: host-3
- apiVersion: v1
  kind: Service
  metadata:
    name: *subdomain
  spec:
    clusterIP: None
    selector:
      app: *app
    ports:
    # NB: Here the Service has at least one port.
    - name: nonexistent
      port: 80

Assuming that YAML document is available in a file called manifests.yaml, create these objects in some namespace:

kubectl apply -f manifests.yaml

Now run a container using an image with dig available in that same namespace, probing first for DNS A records for our subdomain "sub":

kubectl run dig --image=tutum/dnsutils \
  --restart=Never --rm=true --tty --stdin --command -- \
  dig sub a +search +noall +answer
; <<>> DiG 9.10.2 <<>> sub a +search +noall +answer
;; global options: +cmd
sub.my-ns.svc.cluster.local. 30	IN A	172.30.48.142
sub.my-ns.svc.cluster.local. 30	IN A	172.30.48.83
sub.my-ns.svc.cluster.local. 30	IN A	172.30.98.82

Next, confirm that records exist for all three of our pod host names:

for i in $(seq 3); do \
  kubectl run dig --image=tutum/dnsutils \
    --restart=Never --rm=true --tty --stdin --command -- \
    dig "host-${i}.sub" a +search +noall +answer \
done
; <<>> DiG 9.10.2 <<>> host-1.sub a +search +noall +answer
;; global options: +cmd
host-1.sub.my-ns.svc.cluster.local. 30 IN A 172.30.98.82
; <<>> DiG 9.10.2 <<>> host-2.sub a +search +noall +answer
;; global options: +cmd
host-2.sub.my-ns.svc.cluster.local. 30 IN A 172.30.48.83
; <<>> DiG 9.10.2 <<>> host-3.sub a +search +noall +answer
;; global options: +cmd
host-3.sub.my-ns.svc.cluster.local. 30 IN A 172.30.48.142

Next, amend the Service "sub" to remove all of its service ports:

apiVersion: v1
kind: Service
metadata:
  name: *subdomain
spec:
  clusterIP: None
  selector:
    app: *app
  ports:
  # NB: Here the Service has no ports.

With that change applied, we repeat our earlier invocations of dig:

kubectl run dig --image=tutum/dnsutils \
  --restart=Never --rm=true --tty --stdin --command -- \
  dig sub a +search +noall +answer
; <<>> DiG 9.10.2 <<>> sub a +search +noall +answer
;; global options: +cmd
for i in $(seq 3); do \
  kubectl run dig --image=tutum/dnsutils \
    --restart=Never --rm=true --tty --stdin --command -- \
    dig "host-${i}.sub" a +search +noall +answer \
done
; <<>> DiG 9.10.2 <<>> host-1.sub a +search +noall +answer
;; global options: +cmd
; <<>> DiG 9.10.2 <<>> host-2.sub a +search +noall +answer
;; global options: +cmd
; <<>> DiG 9.10.2 <<>> host-3.sub a +search +noall +answer
;; global options: +cmd

Note there how there are no DNS A records available for the Service or any of the pods it selects.

Cause

Why is this so? In method (*KubeDNS.generateRecordsForHeadlessService), it iterates over the Endpoints.Subsets sequence, which is only populated for ports that exist on the related Service object. If the Service defines no ports, the Endpoints object has no subsets for any of the selected pods.

kubectl get endpoints sub --output=jsonpath='{.subsets}'
[]

So long as kube-dns is implemented this way, we need a port to notice the pods backing the Service. Hence, we should either adjust the documentation to match the implementation's constraints, or reconsider the implementation (much harder, duplicating some of the monitoring and filtering work done by the endpoints controller).

Environment

kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T19:11:02Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.4-30+6c97db85c5ab05", GitCommit:"6c97db85c5ab0586c15be39b3e88c7a425b96947", GitTreeState:"clean", BuildDate:"2017-11-21T09:07:18Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[seh]

/remove-lifecycle stale

[thockin]

/lifecycle frozen
/remove-lifecycle stale

[seh]

Related issues:

• Endpoints controller demands ports for headless services kubernetes#32796
• DNS endpoint for headless services with no ports kubernetes#55158
• Local headless service with clusterIP field set as none still requires ports field to be specified  kubernetes#62440

Related fix:

• Populate endpoints for headless service with no ports kubernetes#47250
• Fix a bug that headless service without ports fails to have endpoint created. kubernetes#62497

I thought those would be sufficient to resolve this problem, but testing this again today with Kubernetes version 1.11.2 and CoreDNS still leaves us without A records for the pods, because the Endpoints object has no subsets for the pods.

@xiangpengzhao, do you think this case should work now?

[chrisohaver]

Whats the use case here?
A headless service that defines the open services is kind of the "Kubernetes" way to solve this...
In your case, is it difficult/complex to create and maintain a service definition?

[seh]

It's not that the Service is hard to write; it's that the documentation about how to do so is either wrong, or there's still a defect here. Please note that I was not complaining about why the Service is necessary; I was demonstrating that the despite the documentation, the Service requires at least one port to be defined, even if nothing will ever use that port, and even if no containers actually listen on that port.

[chrisohaver]

Ah OK - So updating the documentation is also an option...

[seh]

Right. I'm hoping someone can weigh in and say, "That was never supposed to work. Why did we claim it would?" Alternately, "Yes, we want this to work, but haven't been able to fix the problem yet."

Note, though, that @thockin did weigh in here:

I think it should work without ports.

[chrisohaver]

OK. According to the DNS Spec, this should work, as long as the endpoints are still deemed "ready" by the API if they have no open ports.

[seh]

Yes, and my reading of kubernetes/kubernetes#47250 says that it should have fixed this problem, but when I run kubectl get endpoints sub --output=yaml for the second phase of my test case above, the Endpoints object has no "subsets" field.

[thockin]

Understand the history and understand the state :) Services always required ports. Then we added headless Services. For the vast vast vast majority of cases, defining ports was logical and natural. Eventually someone said "hey, the port doesn't actually matter for headless services". It's not too surprising that this niche desire is incompletely supported. I'll brush up on context later and make a recommendation. In the mean time, is it egregious to say "define a port" ?

…

On Thu, Aug 16, 2018 at 7:52 AM Steven E. Harris ***@***.***> wrote: Yes, and my reading of kubernetes/kubernetes#47250 <kubernetes/kubernetes#47250> says that it should have fixed this problem, but when I run *kubectl get endpoints sub --output=yaml* for the second phase of my test case above, the *Endpoints* object has no "subsets" field. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#174 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVPcPxmQvRjMS2yAmAZ28spWL_PEEks5uRYdIgaJpZM4Qvtgo> .

[seh]

No, that's not bad, Tim. We can change the documentation to match the implementation.

What's confusing, though, is the recent efforts (the aforementioned merged PRs) to accommodate Services without ports, and why they don't cover this case. The documentation overpromises, but the implementation is trying to make it true. It seems that we're close.

[thockin]

Looking at this now. I feel like it SHOULD work as documented.

[thockin]

OK, I found the bug. It's not pretty to fix, but it's not terrible. Let me pull a PR together and see about tests.

[thockin]

Merged.

[seh]

Thank you, Tim. I'm happy to see one more rough edge—however obscure—sanded away.