Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coredns can not start with “Failed to list *v1.Endpoints: and “Failed to list *v1.Service” #85794

Open
vincent081 opened this issue Dec 2, 2019 · 6 comments

Comments

@vincent081
Copy link

@vincent081 vincent081 commented Dec 2, 2019

I have a problem when I install the kube-dns add on. My OS is CentOS Linux release 7.4

When I start the coredns thecoredns container log:

ace[768378026]: [30.002243414s] [30.002243414s] END
E1202 09:08:34.949036 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.9.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.9.0.1:443: i/o timeout
E1202 09:08:34.949036 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.9.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.9.0.1:443: i/o timeout
E1202 09:08:34.949036 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.9.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.9.0.1:443: i/o timeout
E1202 09:08:34.949013 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.9.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.9.0.1:443: i/o timeout
E1202 09:08:34.949036 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.9.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.9.0.1:443: i/o timeout
E1202 09:08:34.950370 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.9.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.9.0.1:443: i/o timeout
I1202 09:08:34.950302 1 trace.go:82] Trace[1212724865]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2019-12-02 09:08:04.949585685 +0000 UTC m=+31.020839521) (total time: 30.000703085s):
Trace[1212724865]: [30.000703085s] [30.000703085s] END

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Dec 2, 2019

@vincent081: There are no sig labels on this issue. Please add a sig label by either:

  1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
    e.g., @kubernetes/sig-contributor-experience-<group-suffix> to notify the contributor experience sig, OR

  2. specifying the label manually: /sig <group-name>
    e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. See the group list.
The <group-suffix> in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@liggitt

This comment has been minimized.

Copy link
Member

@liggitt liggitt commented Dec 2, 2019

what manifest did you use to deploy coredns? the default addon includes RBAC role/binding permissions (xref https://github.com/kubernetes/kubernetes/blob/release-1.16/cluster/addons/dns/coredns/coredns.yaml.in#L12-L53)

@chrisohaver

This comment has been minimized.

Copy link
Contributor

@chrisohaver chrisohaver commented Dec 2, 2019

The errors in the log shown in the description are timeouts, not auth failures... So may be more than one issue at hand.

@johscheuer

This comment has been minimized.

Copy link
Member

@johscheuer johscheuer commented Dec 2, 2019

Could you check if kube-proxy is running in your cluster ? And how did you set up the cluster (and which version).

@vincent081

This comment has been minimized.

Copy link
Author

@vincent081 vincent081 commented Dec 3, 2019

您用来部署coredns的清单是什么?默认插件包含RBAC角色/绑定权限(外部参照https://github.com/kubernetes/kubernetes/blob/release-1.16/cluster/addons/dns/coredns/coredns.yaml.in#L12-L53)

my coredns.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        beta.kubernetes.io/os: linux
      containers:
      - name: coredns
        image: coredns/coredns:1.6.2
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.9.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  - name: metrics
    port: 9153
    protocol: TCP
@vincent081

This comment has been minimized.

Copy link
Author

@vincent081 vincent081 commented Dec 3, 2019

Could you check if kube-proxy is running in your cluster ? And how did you set up the cluster (and which version).

kube-proxy is running ,cluster 3.0 tks!

@vincent081 vincent081 changed the title coredns can not start with “Failed to list *v1.Endpoints: Unauthorized” and “Failed to list *v1.Service: Unauthorized” coredns can not start with “Failed to list *v1.Endpoints: and “Failed to list *v1.Service” Dec 3, 2019
@liggitt liggitt added triage/support and removed kind/bug labels Dec 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.