Skip to content

Commit

Permalink
test/contrib: Bump CoreDNS version to 1.8.3
Browse files Browse the repository at this point in the history
As reported in [1], Go's HTTP2 client < 1.16 had some serious bugs which
could result in lost connections to kube-apiserver. Worse than this was
that the client couldn't recover.

In the case of CoreDNS the loose of connectivity to kube-apiserver was
even not logged. I have validated this by adding the following rule on
the node which was running the CoreDNS pod (6443 port as the socket-lb
was doing the service xlation):

    iptables -I FORWARD 1 -m tcp --proto tcp --src $CORE_DNS_POD_IP \
        --dport=6443 -j DROP

After upgrading CoreDNS to the one which was compiled with Go >= 1.16,
the pod was not only logging the errors, but also was able to recover
from them in a fast way. An example of such an error:

    W1126 12:45:08.403311       1 reflector.go:436]
    pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: watch
    of *v1.Endpoints ended with: an error on the server ("unable to
    decode an event from the watch stream: http2: client connection
    lost") has prevented the request from succeeding

To determine the min vsn bump, I was using the following:

    for i in 1.7.0 1.7.1 1.8.0 1.8.1 1.8.2 1.8.3 1.8.4; do
        docker run --rm -ti "k8s.gcr.io/coredns/coredns:v$i" \
            --version
    done

    CoreDNS-1.7.0
    linux/amd64, go1.14.4, f59c03d
    CoreDNS-1.7.1
    linux/amd64, go1.15.2, aa82ca6
    CoreDNS-1.8.0
    linux/amd64, go1.15.3, 054c9ae
    k8s.gcr.io/coredns/coredns:v1.8.1 not found: manifest unknown:
    k8s.gcr.io/coredns/coredns:v1.8.2 not found: manifest unknown:
    CoreDNS-1.8.3
    linux/amd64, go1.16, 4293992
    CoreDNS-1.8.4
    linux/amd64, go1.16.4, 053c4d5

Hopefully, the bumped version will fix the CI flakes in which a service
domain name is not available after 7min. In other words, CoreDNS is not
able to resolve the name which means that it hasn't received update from
the kube-apiserver for the service.

[1]: kubernetes/kubernetes#87615 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
  • Loading branch information
brb authored and qmonnet committed Nov 29, 2021
1 parent e03bfff commit 398d55c
Show file tree
Hide file tree
Showing 15 changed files with 64 additions and 15 deletions.
2 changes: 1 addition & 1 deletion test/provision/manifest/1.16/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ spec:
operator: "Exists"
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.7.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
2 changes: 1 addition & 1 deletion test/provision/manifest/1.16/eks/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ spec:
operator: "Exists"
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.7.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
2 changes: 1 addition & 1 deletion test/provision/manifest/1.17/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ spec:
operator: "Exists"
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.7.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
2 changes: 1 addition & 1 deletion test/provision/manifest/1.17/eks/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ spec:
operator: "Exists"
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.7.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
2 changes: 1 addition & 1 deletion test/provision/manifest/1.18/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ spec:
operator: "Exists"
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.7.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
2 changes: 1 addition & 1 deletion test/provision/manifest/1.18/eks/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ spec:
operator: "Exists"
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.7.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
2 changes: 1 addition & 1 deletion test/provision/manifest/1.19/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ spec:
kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.7.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
2 changes: 1 addition & 1 deletion test/provision/manifest/1.19/eks/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ spec:
kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.7.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
9 changes: 8 additions & 1 deletion test/provision/manifest/1.20/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ rules:
- nodes
verbs:
- get
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
Expand Down Expand Up @@ -138,7 +145,7 @@ spec:
kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.7.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
9 changes: 8 additions & 1 deletion test/provision/manifest/1.20/eks/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ rules:
- nodes
verbs:
- get
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
Expand Down Expand Up @@ -138,7 +145,7 @@ spec:
kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.7.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
9 changes: 8 additions & 1 deletion test/provision/manifest/1.21/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ rules:
- nodes
verbs:
- get
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
Expand Down Expand Up @@ -140,7 +147,7 @@ spec:
kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns/coredns:v1.8.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
9 changes: 8 additions & 1 deletion test/provision/manifest/1.21/eks/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ rules:
- nodes
verbs:
- get
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
Expand Down Expand Up @@ -140,7 +147,7 @@ spec:
kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns/coredns:v1.8.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
9 changes: 8 additions & 1 deletion test/provision/manifest/1.22/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ rules:
- nodes
verbs:
- get
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
Expand Down Expand Up @@ -140,7 +147,7 @@ spec:
kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns/coredns:v1.8.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
9 changes: 8 additions & 1 deletion test/provision/manifest/1.22/eks/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,13 @@ rules:
- nodes
verbs:
- get
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
Expand Down Expand Up @@ -140,7 +147,7 @@ spec:
kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns/coredns:v1.8.0
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
resources:
limits:
Expand Down
9 changes: 8 additions & 1 deletion test/provision/manifest/coredns_deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ spec:
- args:
- -conf
- /etc/coredns/Corefile
image: docker.io/coredns/coredns:1.0.6
image: k8s.gcr.io/coredns/coredns:v1.8.3
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 5
Expand Down Expand Up @@ -124,6 +124,13 @@ rules:
verbs:
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
Expand Down

0 comments on commit 398d55c

Please sign in to comment.