kubeadm 710 Switch to a dedicated CA for kubeadm etcd identities #60385

stealthybox · 2018-02-25T05:57:07Z

What this PR does / why we need it:
On kubeadm init/kubeadm upgrade, this PR generates an etcd specific CA for signing the following certs:

etcd serving cert
etcd peer cert
apiserver etcd client cert

These certs were previously signed by the kubernetes CA.
The etcd static pod in local.go has also been updated to only mount the /etcd subdir of cfg.CertificatesDir.

New phase command:

kubeadm alpha phase certs etcd-ca

See the linked issue for details on why this change is an important security feature.

Which issue(s) this PR fixes
Fixes kubernetes/kubeadm#710

Special notes for your reviewer:

on the master

this should still fail:

curl localhost:2379/v2/keys  # no output
curl --cacert /etc/kubernetes/pki/etcd/ca.crt https://localhost:2379/v2/keys  # handshake error

this should now fail: (previously would succeed)

cd /etc/kubernetes/pki
curl --cacert etcd/ca.crt --cert apiserver-kubelet-client.crt --key apiserver-kubelet-client.key https://localhost:2379/v2/keys
  # curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

this should still succeed:

cd /etc/kubernetes/pki
curl --cacert etcd/ca.crt --cert apiserver-etcd-client.crt --key apiserver-etcd-client.key https://localhost:2379/v2/keys

Release note:

On cluster provision or upgrade, kubeadm generates an etcd specific CA for all etcd related certificates.

timothysc · 2018-02-27T16:44:54Z

/cc @detiber @ericchiang @kubernetes/sig-cluster-lifecycle-pr-reviews

timothysc · 2018-02-27T16:46:12Z

This was an implementation request/bug from the original PR so it meets the criteria for post freeze inclusion.

detiber · 2018-02-27T16:49:28Z

/lgtm

xiangpengzhao

just a nit. otherwise LGTM.

xiangpengzhao · 2018-02-27T17:03:54Z

cmd/kubeadm/app/phases/certs/certs.go

@@ -302,6 +322,17 @@ func NewAPIServerKubeletClientCertAndKey(caCert *x509.Certificate, caKey *rsa.Pr
 	return apiClientCert, apiClientKey, nil
 }

+// NewEtcdCACertAndKey generate a self signed front proxy CA.


Why is it a front proxy CA?

Fixed -- thanks peter!

stealthybox · 2018-02-28T01:04:59Z

oops... did not notice it was already in the queue 🙃
could somebody review the spelling updates please? thanks!

++ @xiangpengzhao

dixudx · 2018-02-28T03:28:24Z

/lgtm

dixudx · 2018-02-28T03:28:36Z

/area kubeadm

dixudx · 2018-02-28T03:29:12Z

@stealthybox Could you please squash your commits?

stealthybox · 2018-02-28T05:14:23Z

@dixudx thanks for reviewing!

I separated the typos so that the CA related commit is easier to understand.
I think it's best to leave it like this. Do you agree?

dixudx · 2018-02-28T06:37:33Z

@stealthybox IMO The second commit is more about renaming. But it's okay to keep it as it is.

detiber · 2018-02-28T16:14:28Z

/lgtm

0xmichalis · 2018-02-28T21:04:23Z

cmd/kubeadm/app/phases/upgrade/staticpods.go

@@ -136,17 +136,22 @@ func upgradeComponent(component string, waiter apiclient.Waiter, pathMgr StaticP
 	}

 	// ensure etcd certs are generated for etcd and kube-apiserver
+	if component == constants.Etcd || component == constants.KubeAPIServer {


Why CreateEtcdCACertAndKeyFiles needs to run when upgrading the api server?

IIUC, CreateEtcdServerCertAndKeyFiles in L145 and CreateAPIServerEtcdClientCertAndKeyFiles in L153 depends on etcd CA to generate their cert and key files, CreateEtcdCACertAndKeyFiles ensures etcd CA files are existing or generated.

Makes sense

/hold cancel

Peter explained correctly what I was thinking 👍

The happy path would work without it, but if you ever wanted the apiserver to start independently of etcd, it would fail.

timothysc

/approve

k8s-ci-robot · 2018-03-01T02:22:43Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: detiber, dixudx, stealthybox, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [timothysc]
docs/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

xiangpengzhao · 2018-03-01T02:58:12Z

/hold
until @Kargakis question #60385 (comment) is finally explained. Hopefully my #60385 (comment) is correct :)

timothysc · 2018-03-01T03:03:02Z

xref #60608

k8s-github-robot · 2018-03-01T09:21:14Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-03-01T10:03:57Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

andrewrynhard · 2018-03-17T21:30:01Z

This seems to break self-hosting, self-hosted kube-apiserver:

err (open /etc/kubernetes/pki/apiserver-etcd-client.crt: no such file or directory)

@timothysc I'm not sure what the status is for storing certs as secrets in the cluster. I know it is behind a feature gate, but is that something we wan't to continue to do? IIRC there were security concerns with that. I can add apiserver-etcd-client* in a PR if that is the route we are sticking with.

stealthybox · 2018-03-18T00:41:18Z

@andrewrynhard do we rely on external etcd for self-hosted?
Is there a case where the self-hosted apiserver contacts an etcd static pod?
I was under the impression that we deprecated etcd-operator.

I know there's upgrade logic that modifies the static pod manifests for self-hosted -- we could ensure the cert is generated and mounted or just remove this option unless a user specifies an external etcd cert.

andrewrynhard · 2018-03-18T00:47:12Z

@stealthybox sorry, I should have been more specific... this breaks self-hosting if StoreCertsInSecrets is set to true. When StoreCertsInSecrets is set to true, the certs in /etc/kubernetes/pki get stored as secrets in the cluster: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/selfhosting/selfhosting.go#L63-L73.

stealthybox · 2018-03-18T00:52:09Z

I think I found the bug.

stealthybox · 2018-03-18T00:58:25Z

At a minimum, we should add the apiserver client cert to getTLSKeyPairs().
uploadTLSSecrets() uses it to get the collection when features.StoreCertsInSecrets is enabled:

https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/cmd/kubeadm/app/phases/selfhosting/selfhosting.go#L67:13

https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/cmd/kubeadm/app/phases/selfhosting/selfhosting_volumes.go#L184:29

https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/cmd/kubeadm/app/phases/selfhosting/selfhosting_volumes.go#L265:6

stealthybox · 2018-03-18T01:06:41Z

@andrewrynhard
would you be willing to open an issue for this including the command and config that reproduces this bug?

I believe the patch should include something like this:

diff --git a/cmd/kubeadm/app/phases/selfhosting/selfhosting_volumes.go b/cmd/kubeadm/app/phases/selfhosting/selfhosting_volumes.go
index 627fb01f04..2446843c34 100644
--- a/cmd/kubeadm/app/phases/selfhosting/selfhosting_volumes.go
+++ b/cmd/kubeadm/app/phases/selfhosting/selfhosting_volumes.go
@@ -279,6 +279,11 @@ func getTLSKeyPairs() []*tlsKeyPair {
                        cert: kubeadmconstants.APIServerKubeletClientCertName,
                        key:  kubeadmconstants.APIServerKubeletClientKeyName,
                },
+               {
+                       name: kubeadmconstants.APIServerEtcdClientCertAndKeyBaseName,
+                       cert: kubeadmconstants.APIServerEtcdClientCertName,
+                       key:  kubeadmconstants.APIServerEtcdClientKeyName,
+               },
                {
                        name: kubeadmconstants.ServiceAccountKeyBaseName,
                        cert: kubeadmconstants.ServiceAccountPublicKeyName,

Probably needs a test as well -- I can help review.

stealthybox · 2018-03-18T01:09:39Z

We also need to add a matching &v1.ProjectedVolumeSource to apiServerCertificatesVolumeSource()

andrewrynhard · 2018-03-18T02:19:40Z

@stealthybox I can create an issue once I’m back to the computer. First glance I think that patch is all we need.

andrewrynhard · 2018-03-18T02:20:13Z

That patch and the projected volume

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 25, 2018

k8s-ci-robot requested review from dmmcquay and 0xmichalis February 25, 2018 05:57

stealthybox force-pushed the feature/kubeadm_710-etcd-ca branch from dddf030 to 07802e2 Compare February 25, 2018 06:48

k8s-github-robot added the kind/old-docs label Feb 25, 2018

stealthybox force-pushed the feature/kubeadm_710-etcd-ca branch 3 times, most recently from 9eb98eb to be66dcf Compare February 25, 2018 20:41

timothysc requested a review from detiber February 27, 2018 16:44

k8s-ci-robot added the sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. label Feb 27, 2018

timothysc added kind/bug Categorizes issue or PR as related to a bug. status/approved-for-milestone labels Feb 27, 2018

timothysc added this to the v1.10 milestone Feb 27, 2018

k8s-ci-robot assigned detiber Feb 27, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 27, 2018

xiangpengzhao reviewed Feb 27, 2018

View reviewed changes

stealthybox added 2 commits February 27, 2018 17:42

Switch to a dedicated CA for kubeadm etcd identities

2d9b2d9

Fix typos

41974cb

stealthybox force-pushed the feature/kubeadm_710-etcd-ca branch from be66dcf to 41974cb Compare February 28, 2018 00:57

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 28, 2018

k8s-ci-robot assigned dixudx Feb 28, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 28, 2018

k8s-ci-robot added the area/kubeadm label Feb 28, 2018

0xmichalis reviewed Feb 28, 2018

View reviewed changes

timothysc approved these changes Mar 1, 2018

View reviewed changes

timothysc added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 1, 2018

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 1, 2018

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 1, 2018

k8s-github-robot merged commit 5cff6c9 into kubernetes:master Mar 1, 2018

stealthybox mentioned this pull request Mar 3, 2018

etcd fails kubelet's health checks kubernetes/kubeadm#720

Closed

andrewrynhard mentioned this pull request Mar 18, 2018

kubeadm: etcd certs missing in self-hosted deploy #61322

Closed

wojtek-t mentioned this pull request Apr 5, 2018

'PATCH node-status' latency slo violations #62064

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm 710 Switch to a dedicated CA for kubeadm etcd identities #60385

kubeadm 710 Switch to a dedicated CA for kubeadm etcd identities #60385

stealthybox commented Feb 25, 2018 •

edited

timothysc commented Feb 27, 2018

timothysc commented Feb 27, 2018

detiber commented Feb 27, 2018

xiangpengzhao left a comment

xiangpengzhao Feb 27, 2018

stealthybox Feb 28, 2018

stealthybox commented Feb 28, 2018 •

edited

dixudx commented Feb 28, 2018

dixudx commented Feb 28, 2018

dixudx commented Feb 28, 2018

stealthybox commented Feb 28, 2018

dixudx commented Feb 28, 2018

detiber commented Feb 28, 2018

0xmichalis Feb 28, 2018

xiangpengzhao Mar 1, 2018

0xmichalis Mar 1, 2018

stealthybox Mar 2, 2018

timothysc left a comment

k8s-ci-robot commented Mar 1, 2018

xiangpengzhao commented Mar 1, 2018

timothysc commented Mar 1, 2018

k8s-github-robot commented Mar 1, 2018

k8s-github-robot commented Mar 1, 2018

andrewrynhard commented Mar 17, 2018 •

edited

stealthybox commented Mar 18, 2018

andrewrynhard commented Mar 18, 2018

stealthybox commented Mar 18, 2018

stealthybox commented Mar 18, 2018

stealthybox commented Mar 18, 2018

stealthybox commented Mar 18, 2018

andrewrynhard commented Mar 18, 2018

andrewrynhard commented Mar 18, 2018

kubeadm 710 Switch to a dedicated CA for kubeadm etcd identities #60385

kubeadm 710 Switch to a dedicated CA for kubeadm etcd identities #60385

Conversation

stealthybox commented Feb 25, 2018 • edited

on the master

timothysc commented Feb 27, 2018

timothysc commented Feb 27, 2018

detiber commented Feb 27, 2018

xiangpengzhao left a comment

Choose a reason for hiding this comment

xiangpengzhao Feb 27, 2018

Choose a reason for hiding this comment

stealthybox Feb 28, 2018

Choose a reason for hiding this comment

stealthybox commented Feb 28, 2018 • edited

dixudx commented Feb 28, 2018

dixudx commented Feb 28, 2018

dixudx commented Feb 28, 2018

stealthybox commented Feb 28, 2018

dixudx commented Feb 28, 2018

detiber commented Feb 28, 2018

0xmichalis Feb 28, 2018

Choose a reason for hiding this comment

xiangpengzhao Mar 1, 2018

Choose a reason for hiding this comment

0xmichalis Mar 1, 2018

Choose a reason for hiding this comment

stealthybox Mar 2, 2018

Choose a reason for hiding this comment

timothysc left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 1, 2018

xiangpengzhao commented Mar 1, 2018

timothysc commented Mar 1, 2018

k8s-github-robot commented Mar 1, 2018

k8s-github-robot commented Mar 1, 2018

andrewrynhard commented Mar 17, 2018 • edited

stealthybox commented Mar 18, 2018

andrewrynhard commented Mar 18, 2018

stealthybox commented Mar 18, 2018

stealthybox commented Mar 18, 2018

stealthybox commented Mar 18, 2018

stealthybox commented Mar 18, 2018

andrewrynhard commented Mar 18, 2018

andrewrynhard commented Mar 18, 2018

stealthybox commented Feb 25, 2018 •

edited

stealthybox commented Feb 28, 2018 •

edited

andrewrynhard commented Mar 17, 2018 •

edited