tls: add scripts and instructions for rotating certificates #155

ericchiang · 2018-03-29T18:38:19Z

This is a revamp of our TLS rotation docs. I've been testing them on more recent clusters (1.8.x) on AWS.

Etcd rotation instructions will be added in a bit, but I'd like early feedback.

@kbrwn for testing
@robszumski for general review
@zbwright for docs

ericchiang · 2018-03-29T18:47:51Z

Some open questions:

Where should these scripts live?
Does this work on other clouds?

ericavonb · 2018-03-29T19:06:48Z

Documentation/tls/rotate-tls.md

-Start by reviewing the general [TLS documentation][tls-certs] and the [TLS topology][tls-topology] for Tectonic to identify the various certificates in the cluster.
-
-We will be using the [CFSSL][cfssl-util] utility to view and manage the certificates, which may be downloaded from [https://pkg.cfssl.org/][cfssl-package].
+__WARNING:__ Rotating certificates by hand can break component connectivity and leave the cluster in an unrecoverable state. Before performing any of these instructions on a live cluster backup your cluster state and migrate critical workloads to another cluster.


Could we link to backup docs?

We don't have them and I imagine they're custom to different cluster setups. E.g. if you've got stateless apps it's just the manifests. If you've got persistent volumes you'll need to backup the data.

zbwright · 2018-03-29T20:03:25Z

Documentation/tls/rotate-tls.md


-Copy the archive with the new certificates into place and change to the
-directory.
+__WARNING:__ you MUST use `kubectl apply` in the following command and NOT other `kubectl` creation sub-commands.


WARNING: You MUST use

zbwright · 2018-03-29T20:03:48Z

Documentation/tls/rotate-tls.md

 ```

-Remove the old certificates and unzip the archive with the new certificates.
+To force the various deployments to restart and pick-up the new TLS assets, force the rotation of the various components. Note that the API server may become temporarily unavailable after this action.


... pick up ...

zbwright · 2018-03-29T20:04:17Z

Documentation/tls/rotate-tls.md

-sudo chown etcd: peer.* server.*
-ls -lAh
-```
+Unlike other cluster components, kubelets are configured through host files and require SSH access to the modify. Because Tectonic often deploys worker nodes behind firewalls, this document assumes using one of the control plane nodes as a [bastion host][bastion-host] for access to the cluster.


... require SSH access to modify. ...

zbwright · 2018-03-29T20:05:23Z

Documentation/tls/rotate/gencerts.sh

+                    base domain is "example.com"
+
+    CLUSTER_NAME    Name of the cluster. If your API server is running on the
+                    domain "my-cluster-k8s.example.com" the name of the cluster


... "my-cluster-k8s.example.com", the name ...

zbwright · 2018-03-29T20:06:02Z

Documentation/tls/rotate/gencerts.sh

+rm $CERT_DIR/serial*
+rm $CERT_DIR/*.csr
+
+# Use openssl for base64'ing instead of base64 which has different wrap behavior


I think we actually mean openssl here (i.e. the CLI binary).

ericchiang · 2018-03-29T22:49:42Z

It was pointed out that kube-proxy reuses the kubelet's certs https://github.com/coreos/tectonic-installer/blob/1.8.9-tectonic.1/modules/bootkube/resources/manifests/kube-proxy.yaml#L31

Need to roll that daemonset as well.

ericchiang · 2018-04-04T23:40:07Z

This document is almost ready to merge, but has a bug in it that keeps bricking my clusters...

I cannot stress this warning at the beginning more

__WARNING:__ Rotating certificates by hand can break component connectivity and leave the cluster in an unrecoverable state. Before performing any of these instructions on a live cluster backup your cluster state and migrate critical workloads to another cluster.

ericchiang · 2018-04-06T22:11:16Z

This is done. @zbwright would you take a look one last time?

justaugustus · 2018-04-06T22:20:55Z

@ericchiang Do you have any context on why the clusters are getting bricked?
A warning like that seems super troubling.

ericchiang · 2018-04-06T22:23:22Z

If you mess up the etcd CA rotation, then it's really hard to change anything on the self hosted control plane. I can expand on that in the doc.

…

On Fri, Apr 6, 2018, 3:21 PM Stephen Augustus ***@***.***> wrote: @ericchiang <https://github.com/ericchiang> Do you have any context on why the clusters are getting bricked? A warning like that seems super troubling. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#155 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACO_XWQxDZ4jWE5cRs58CXBnFf0g3m_sks5tl-pJgaJpZM4TAyzg> .

ericchiang · 2018-04-06T22:26:50Z

To be clear that bug I mentioned earlier was resolved, but still I'd tread very carefully here.

justaugustus · 2018-04-06T22:29:18Z

@ericchiang cool, cool. Thanks for the clarification!

ericchiang · 2018-04-11T20:21:40Z

bumping this thread. there was some interest in more testing beside's me just doing it. did that ever get planned/done?

justaugustus · 2018-04-11T20:27:44Z

@ericchiang I was hope to get some field validation on this, but as it's only Dan (and me, in a diminished capacity), I don't think we can commit to any testing in the near-term, so don't block this on my account.

@kbrwn mentioned he was working with someone, but would need to check-in again to try out the etcd rotation.

zbwright · 2018-04-11T23:30:28Z

Documentation/tls/rotate/gencerts.sh

+    openssl x509 -in $CERT -noout -text > "${CERT%.crt}.txt"
+done
+
+# Use openssl for base64'ing instead of base64 which has different wrap behavior


"openssl" is the tool, right?

zbwright · 2018-04-11T23:34:01Z

Documentation/tls/rotate-tls.md

-```
-
-This will generate several files in the current directory.
+The scripts creates a directory of generated TLS assets. If you provided the etcd CA, this will include etcd certificates and manifest patches.


either 'script creates' or 'scripts create'. I believe it's the former.

zbwright · 2018-04-11T23:34:46Z

Documentation/tls/rotate-tls.md


-## etcd
+## Rotating certificates for Tectonic and Kubernetes components.


no periods in headers

zbwright · 2018-04-11T23:35:30Z

Documentation/tls/rotate-tls.md

-### Verify cluster health
-
-First, verify the current health of the etcd cluster. Connect to one of the etcd members of the cluster using SSH.
+__WARNING:__ The following commands MUST use `kubectl patch` and NOT other `kubectl` creation sub-commands.


subcommands

zbwright · 2018-04-11T23:36:52Z

Documentation/tls/rotate-tls.md

 ```

-etcd clusters should be configured to require client authentication. Therefore, we will need the existing CA certificate, and the client certificate and key for the cluster. These artifacts should be located in the `/etc/ssl/etcd` directory if the Tectonic cluster was set up using self-signed certificates.
+To force the various deployments to restart and pick up the new TLS assets, force the rotation of the various components. Note that the API server may become temporarily unavailable after this action.


To force the deployments to restart and pick up the new TLS assets, force the rotation of the deployments' components.

zbwright · 2018-04-11T23:39:27Z

Documentation/tls/rotate-tls.md

 ```

-Generate the new certificate and private key using the CFSSL utility.
+The addresses of a cluster's etcd instances be found by inspecting the API server's `--etcd-servers` flag.


Inspect the API server's --etcd-servers flag to find the address of a cluster's etcd instances.

zbwright · 2018-04-11T23:40:07Z

Documentation/tls/rotate-tls.md

 ```

-### Client
+Finally, for each etcd instance, rotate the peer and serving certs.


ericchiang · 2018-04-13T21:57:08Z

Docs updated.

ericchiang · 2018-04-19T21:41:33Z

Okay it's been a bit. I'm merging this tomorrow afternoon unless someone says otherwise.

ericavonb reviewed Mar 29, 2018

View reviewed changes

zbwright reviewed Mar 29, 2018

View reviewed changes

zbwright reviewed Apr 11, 2018

View reviewed changes

tls: add scripts and instructions for rotating certificates

54ea284

ericchiang merged commit 56f7173 into coreos:master Apr 20, 2018

ericchiang deleted the rotate-tls branch April 20, 2018 23:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tls: add scripts and instructions for rotating certificates #155

tls: add scripts and instructions for rotating certificates #155

ericchiang commented Mar 29, 2018 •

edited

Loading

ericchiang commented Mar 29, 2018 •

edited

Loading

ericavonb Mar 29, 2018

ericchiang Mar 29, 2018

zbwright Mar 29, 2018

zbwright Mar 29, 2018

zbwright Mar 29, 2018

zbwright Mar 29, 2018

zbwright Mar 29, 2018

lucab Mar 31, 2018

ericchiang commented Mar 29, 2018

ericchiang commented Apr 4, 2018

ericchiang commented Apr 6, 2018

justaugustus commented Apr 6, 2018

ericchiang commented Apr 6, 2018 via email

ericchiang commented Apr 6, 2018

justaugustus commented Apr 6, 2018

ericchiang commented Apr 11, 2018

justaugustus commented Apr 11, 2018 •

edited

Loading

zbwright Apr 11, 2018

ericchiang Apr 13, 2018

zbwright Apr 11, 2018 •

edited

Loading

zbwright Apr 11, 2018

zbwright Apr 11, 2018

zbwright Apr 11, 2018

zbwright Apr 11, 2018

zbwright Apr 11, 2018

ericchiang commented Apr 13, 2018

ericchiang commented Apr 19, 2018


		## etcd
		## Rotating certificates for Tectonic and Kubernetes components.

tls: add scripts and instructions for rotating certificates #155

tls: add scripts and instructions for rotating certificates #155

Conversation

ericchiang commented Mar 29, 2018 • edited Loading

ericchiang commented Mar 29, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericchiang commented Mar 29, 2018

ericchiang commented Apr 4, 2018

ericchiang commented Apr 6, 2018

justaugustus commented Apr 6, 2018

ericchiang commented Apr 6, 2018 via email

ericchiang commented Apr 6, 2018

justaugustus commented Apr 6, 2018

ericchiang commented Apr 11, 2018

justaugustus commented Apr 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zbwright Apr 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericchiang commented Apr 13, 2018

ericchiang commented Apr 19, 2018

ericchiang commented Mar 29, 2018 •

edited

Loading

ericchiang commented Mar 29, 2018 •

edited

Loading

justaugustus commented Apr 11, 2018 •

edited

Loading

zbwright Apr 11, 2018 •

edited

Loading