Update kubernetes to 1.19.3 #1030

surajssd · 2020-09-28T14:37:39Z

Remove the code added for HAProxy.
Modify the seccomp allowed profiles so that pods that have a default profile are allowed to run.

Fixes #866
Fixes #867

assets/charts/control-plane/kube-apiserver/values.yaml

assets/charts/control-plane/kube-apiserver/templates/kube-apiserver.yaml

knrt10 · 2020-09-29T14:32:51Z

Added #867 to this PR in description, as this fixes it too. but I don't know why am I am not able to find it in linked issue.

assets/charts/control-plane/kube-apiserver/templates/kube-apiserver.yaml

surajssd · 2020-09-30T10:45:46Z

Pods fail to create when the linkerd is installed:

cert-manager    0s   Warning   FailedCreate    replicaset/cert-manager-cainjector-fc6c787db    Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post "https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
cert-manager    0s   Warning   FailedCreate    replicaset/cert-manager-webhook-845d9df8bf      Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post "https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
projectcontour  0s   Warning   FailedCreate    daemonset/envoy                                 Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post "https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

And due to this the CI is timing out right now. This is a weird error.

invidian · 2020-09-30T13:08:58Z

And due to this the CI is timing out right now. This is a weird error.

This is perhaps because of:

Kubernetes is now built with golang 1.15.0-rc.1.
The deprecated, legacy behavior of treating the CommonName field on X.509 serving certificates as a host name when no Subject Alternative Names are present is now disabled by default. It can >be temporarily re-enabled by adding the value x509ignoreCN=0 to the GODEBUG environment variable. (kubernetes/kubernetes#93264, @justaugustus) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, >Instrumentation, Network, Node, Release, Scalability, Storage and Testing]

This seems to me like linkerd generates bad certificates then?

assets/charts/components/cert-manager/templates/cainjector-psp.yaml

assets/charts/control-plane/kube-apiserver/templates/kube-apiserver.yaml

ci/aws/aws-cluster.lokocfg.envsubst

assets/charts/components/prometheus-operator/templates/grafana/dashboards-1.14/kubelet.yaml

assets/charts/control-plane/kube-apiserver/templates/_helpers.tpl

invidian · 2020-10-12T19:31:31Z

assets/charts/control-plane/kube-apiserver/templates/kube-apiserver-ds.yaml

@@ -0,0 +1,29 @@
+{{- if gt (int .Values.apiserver.replicas) 1 }}
+apiVersion: apps/v1
+kind: DaemonSet


There is one disadvantage of running those components as DaemonSet instead of Deployment, which I recently hit myself.

If one runs 3 node controller pool, but each node is resource constrained (e.g. 2 vCPUs, 4GB of RAM), then one might want to run only 2 replicas of each component to save a bit of resources, as for example each kube-apiserver is very RAM-costly.

I think it should be our conscious decision, that we won't be able to support such scenario.

I would rather run less number of other components than APIServer. If I were to run my prod cluster then I should make sure that control plane is not fighting for resources. On the workers I will try to bin pack my work loads, not on the control plane.

Leaving this open so that we know this was decided consciously.

invidian · 2020-10-12T19:35:54Z

I forgot to summarize requested changes:

DaemonSet over Deployment has some disadvantage I recently hit, so we should agree we accept it.
Switch from Deployment to DaemonSet could be done in separate PR, to avoid this PR from overgrowing.
Some commit messages are out of date
Some commits add code which is removed in the next commits, but they do not follow logical flow, but rather fix failed attempt of resolving issue, so IMO they should be adjusted. Failed attempt can be described in commit message instead, for example.
Some of existing users might be affected by GODEBUG=x509ignoreCN=0, do we want to handle that?
We do some changes to upstream projects, which IMO should be reported upstream so we don't carry the maintenance cost.

rata

@surajssd can you update calico-hostendpoint-controller to use same kubernetes version? or not more than 2 versions older.

I mean here, just in case: https://github.com/kinvolk/calico-hostendpoint-controller/blob/fcf32d9e11c82a6522f49b860b6c4d3d411ca245/Dockerfile#L13. And update the calico component version here :)

While it probably will work with an older version, it is not guaranteed, so we should avoid it IMHO.

assets/charts/control-plane/kube-apiserver/Chart.yaml

surajssd · 2020-10-14T13:46:07Z

@rata done kinvolk/calico-hostendpoint-controller#8

invidian

Largely LGTM, I still found some nits which could be addressed.

assets/charts/control-plane/kubelet/Chart.yaml

assets/charts/components/cert-manager/templates/cainjector-psp.yaml

README.md

assets/charts/components/prometheus-operator/templates/grafana/dashboards-1.14/kubelet.yaml

test/components/coredns/coredns_test.go

assets/terraform-modules/bootkube/resources/bootstrap-manifests/bootstrap-apiserver.yaml

pkg/platform/packet/template.go

The pods that don't specify seccomp profile in their `securityContext` explicitly get `runtime/default` injected by default. So this profile has to be allowed by PSPs or else all pods' config should be updated to have seccomp profile mentioned explicitly. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

Release Notes: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#downloads-for-v1192 Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

invidian

LGTM. Great work @surajssd 🎉

Pull the image from quay.io instead of docker hub. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

- Sets the flag `--permit-port-sharing` on apiserver, bootstrap and self-hosted both, so that now multiple apiserver pods (which use host network) can bind on port 6443 simultaneously. - Makes the self-hosted apiserver "Deployment" based instead of Daemonset. - Remove HAProxy sidecar and the init container needed to create config. - Remove all appearance of `exposeOnAllInterfaces` parameter. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

For the control plane components if the control plane nodes are more than 1 then use DaemonSets but if it is just one node then use Deployment. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

To be able to run on same port as the bootstrap kube-apiserver this should run as root. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

This commit renames the metrics: - `kubelet_running_pod_count` to `kubelet_running_pods`. - `kubelet_running_container_count` to `kubelet_running_containers`. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

These tests were earlier run on deployment but now they should be run on daemonset, since these components are deployed as ones. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

- With go 1.15 there are strict checks for common name so this commit adds a new variable `ignore_x509_cn_check` to Packet and AWS so that user can disable such strict checks. - This commit disables the checks on the CI AWS and Packet, since we test linkerd on these two platforms. Once linkerd provides builds with go 1.15 then we can revert changes from this commit. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

Release Notes: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#v1193 Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

surajssd requested a review from invidian September 28, 2020 14:37

invidian suggested changes Sep 28, 2020

View reviewed changes

surajssd force-pushed the surajssd/update-k8s-1.19.2 branch from 680465e to 89cfac4 Compare September 29, 2020 13:18

surajssd force-pushed the surajssd/update-k8s-1.19.2 branch from 89cfac4 to 910b09a Compare September 30, 2020 08:42

surajssd requested a review from invidian September 30, 2020 08:43

invidian suggested changes Sep 30, 2020

View reviewed changes

invidian reviewed Sep 30, 2020

View reviewed changes

assets/charts/control-plane/kube-apiserver/templates/kube-apiserver.yaml Show resolved Hide resolved

surajssd force-pushed the surajssd/update-k8s-1.19.2 branch 9 times, most recently from bc9d34b to eab5513 Compare October 9, 2020 11:37

surajssd force-pushed the surajssd/update-k8s-1.19.2 branch 2 times, most recently from c280165 to 4f6cebf Compare October 12, 2020 10:27

invidian suggested changes Oct 12, 2020

View reviewed changes

surajssd force-pushed the surajssd/update-k8s-1.19.2 branch 3 times, most recently from e59aa43 to 8d1dd96 Compare October 14, 2020 08:22

rata reviewed Oct 14, 2020

View reviewed changes

assets/charts/control-plane/kube-apiserver/Chart.yaml Show resolved Hide resolved

invidian suggested changes Oct 14, 2020

View reviewed changes

surajssd added 2 commits October 15, 2020 14:01

Update Kubernetes to v1.19.2

45d2fd0

Release Notes: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#downloads-for-v1192 Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

surajssd force-pushed the surajssd/update-k8s-1.19.2 branch from 8d1dd96 to bad8ede Compare October 15, 2020 09:13

invidian previously approved these changes Oct 15, 2020

View reviewed changes

surajssd dismissed invidian’s stale review via 000713c October 15, 2020 09:46

surajssd force-pushed the surajssd/update-k8s-1.19.2 branch from bad8ede to 000713c Compare October 15, 2020 09:46

surajssd requested a review from invidian October 15, 2020 10:12

invidian changed the title ~~Update kubernetes to 1.19.2~~ Update kubernetes to 1.19.3 Oct 15, 2020

invidian previously approved these changes Oct 15, 2020

View reviewed changes

surajssd added 10 commits October 15, 2020 16:16

calico-host-endpoint controller: Bump to v0.0.4

931ce75

Pull the image from quay.io instead of docker hub. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

control plane charts: Update the versions

4f23380

Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

control plane charts: Add daemonsets

3c1fb4a

For the control plane components if the control plane nodes are more than 1 then use DaemonSets but if it is just one node then use Deployment. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

self hosted kube-apiserver: Run as root

e151c6f

To be able to run on same port as the bootstrap kube-apiserver this should run as root. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

monitoring: Rename kubelet metrics

26753d7

This commit renames the metrics: - `kubelet_running_pod_count` to `kubelet_running_pods`. - `kubelet_running_container_count` to `kubelet_running_containers`. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

e2e test: Fix controller manager and coredns test

2f20a5b

These tests were earlier run on deployment but now they should be run on daemonset, since these components are deployed as ones. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

Update Kubernetes to v1.19.3

98bb2db

Release Notes: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#v1193 Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

Update generated_assets

bb4d412

Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

surajssd dismissed invidian’s stale review via bb4d412 October 15, 2020 10:47

surajssd force-pushed the surajssd/update-k8s-1.19.2 branch from 000713c to bb4d412 Compare October 15, 2020 10:47

surajssd requested a review from invidian October 15, 2020 11:29

invidian approved these changes Oct 15, 2020

View reviewed changes

surajssd merged commit 49cbc6e into master Oct 15, 2020

surajssd deleted the surajssd/update-k8s-1.19.2 branch October 15, 2020 11:38

johananl mentioned this pull request Oct 19, 2020

kube-controller-manager doesn't respawn with 1 controller #1097

Closed

invidian mentioned this pull request Nov 2, 2020

Ensure that controlplane pods are spread equally across controller nodes #90

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update kubernetes to 1.19.3 #1030

Update kubernetes to 1.19.3 #1030

surajssd commented Sep 28, 2020 •

edited by invidian

Loading

knrt10 commented Sep 29, 2020 •

edited

Loading

surajssd commented Sep 30, 2020 •

edited

Loading

invidian commented Sep 30, 2020

invidian Oct 12, 2020

surajssd Oct 13, 2020

surajssd Oct 15, 2020

invidian commented Oct 12, 2020

rata left a comment •

edited

Loading

surajssd commented Oct 14, 2020

invidian left a comment

invidian left a comment

Update kubernetes to 1.19.3 #1030

Update kubernetes to 1.19.3 #1030

Conversation

surajssd commented Sep 28, 2020 • edited by invidian Loading

knrt10 commented Sep 29, 2020 • edited Loading

surajssd commented Sep 30, 2020 • edited Loading

invidian commented Sep 30, 2020

invidian Oct 12, 2020

Choose a reason for hiding this comment

surajssd Oct 13, 2020

Choose a reason for hiding this comment

surajssd Oct 15, 2020

Choose a reason for hiding this comment

invidian commented Oct 12, 2020

rata left a comment • edited Loading

Choose a reason for hiding this comment

surajssd commented Oct 14, 2020

invidian left a comment

Choose a reason for hiding this comment

invidian left a comment

Choose a reason for hiding this comment

surajssd commented Sep 28, 2020 •

edited by invidian

Loading

knrt10 commented Sep 29, 2020 •

edited

Loading

surajssd commented Sep 30, 2020 •

edited

Loading

rata left a comment •

edited

Loading