dns-controller: :3999: bind: address already in use

[tvi]

Thanks for submitting an issue! Please fill in as much of the template below as
you can.

------------- BUG REPORT TEMPLATE --------------------

1. What kops version are you running? The command kops version, will display
   this information.
   Version dev (git-210a7bfe9)

2. What Kubernetes version are you running? kubectl version will print the
   version if a cluster is running or provide the Kubernetes version specified as
   a kops flag.
   Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?
   AWS

4. What commands did you run? What is the simplest way to reproduce this issue?
   kops create cluster \ --cloud=aws \ --zones us-west-1a \ --name=${KOPS_CLUSTER_NAME} \ --vpc ${KOPS_VPC_ID} \ --networking amazon-vpc-routed-eni \ --master-size c4.xlarge \ --kubernetes-version ${KOPS_KUBERNETES_VERSION:-1.9.3} \ --node-count ${KOPS_NODE_COUNT:-4} \ --node-size m4.2xlarge \ --dns private

5. What happened after the commands executed?
   The cluster bring up got stuck on:
   ks logs dns-controller-b44b56b6b-pjph2 dns-controller version 1.9.0-beta.1 2018/04/02 18:53:13 listen tcp :3999: bind: address already in use

6. What did you expect to happen?
   Cluster should work.

➜  k8s git:(ssh) ✗ ks describe deploy/dns-controller
Name:                   dns-controller
Namespace:              kube-system
CreationTimestamp:      Mon, 02 Apr 2018 11:47:06 -0700
Labels:                 k8s-addon=dns-controller.addons.k8s.io
                        k8s-app=dns-controller
                        version=v1.9.0-beta.1
Annotations:            deployment.kubernetes.io/revision=1
                        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"k8s-addon":"dns-controller.addons.k8s.io","k8s-app":"dns...
Selector:               k8s-app=dns-controller
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           k8s-addon=dns-controller.addons.k8s.io
                    k8s-app=dns-controller
                    version=v1.9.0-beta.1
  Annotations:      scheduler.alpha.kubernetes.io/critical-pod=
                    scheduler.alpha.kubernetes.io/tolerations=[{"key": "dedicated", "value": "master"}]
  Service Account:  dns-controller
  Containers:
   dns-controller:
    Image:  kope/dns-controller:1.9.0-beta.1
    Port:   <none>
    Command:
      /usr/bin/dns-controller
      --watch-ingress=false
      --dns=gossip
      --gossip-seed=127.0.0.1:3999
      --zone=*/*
      -v=2

I fixed it by adding --metrics-listen-addr=:4040
like this ks edit deploy/dns-controller

ks describe deploy/dns-controller
Name:                   dns-controller
Namespace:              kube-system
CreationTimestamp:      Mon, 02 Apr 2018 11:47:06 -0700
Labels:                 k8s-addon=dns-controller.addons.k8s.io
                        k8s-app=dns-controller
                        version=v1.9.0-beta.1
Annotations:            deployment.kubernetes.io/revision=2
                        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"k8s-addon":"dns-controller.addons.k8s.io","k8s-app":"dns...
Selector:               k8s-app=dns-controller
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           k8s-addon=dns-controller.addons.k8s.io
                    k8s-app=dns-controller
                    version=v1.9.0-beta.1
  Annotations:      scheduler.alpha.kubernetes.io/critical-pod=
                    scheduler.alpha.kubernetes.io/tolerations=[{"key": "dedicated", "value": "master"}]
  Service Account:  dns-controller
  Containers:
   dns-controller:
    Image:  kope/dns-controller:1.9.0-beta.1
    Port:   <none>
    Command:
      /usr/bin/dns-controller
      --watch-ingress=false
      --dns=gossip
      --gossip-seed=127.0.0.1:3999
      --metrics-listen-addr=:4040
      --zone=*/*
      -v=2
    Requests:
      cpu:        50m
      memory:     50Mi
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>

[tvi]

@justinsb This is still broken. Need to redeploy new version of image: kope/dns-controller:1.9.0-beta.1

[agrakhov]

Glad I found this. In my case, this was happening with dns-controller version 1.15.2 so I edited deployment kubectl -n kube-system edit deployments dns-controller, added to spec.containers.command: - --metrics-listen=127.0.0.1:4040 and it worked.
Hope this will help someone :)

[johngmyers]

The fix for this was removed in ed797d1 by @justinsb

/reopen
/assign @justinsb

[k8s-ci-robot]

@johngmyers: Reopened this issue.

In response to this:

The fix for this was removed in ed797d1 by @justinsb

/reopen
/assign @justinsb

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tvi]

#8771 Might be related

[olemarkus]

I just wonder if all of these gossip issues is actually related to #8771.
If so, just close this as duplicate.