Skip to content

Update secure Kubernetes instructions with RBAC support #2281

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 13, 2017
Merged

Update secure Kubernetes instructions with RBAC support #2281

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 16 additions & 11 deletions _includes/orchestration/start-cluster.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ $ kubectl create -f https://raw.githubusercontent.com/cockroachdb/cockroach/mast
~~~

~~~
serviceaccount "cockroachdb" created
role "cockroachdb" created
clusterrole "cockroachdb" created
rolebinding "cockroachdb" created
clusterrolebinding "cockroachdb" created
service "cockroachdb-public" created
service "cockroachdb" created
poddisruptionbudget "cockroachdb-budget" created
Expand All @@ -18,17 +23,17 @@ statefulset "cockroachdb" created

1. From your local workstation, use our [`cockroachdb-statefulset.yaml`](https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/cockroachdb-statefulset.yaml) file to create the StatefulSet that automatically creates 3 pods, each with a CockroachDB node running inside it:

{% include copy-clipboard.html %}
~~~ shell
$ kubectl create -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cockroachdb-statefulset.yaml
~~~

~~~
service "cockroachdb-public" created
service "cockroachdb" created
poddisruptionbudget "cockroachdb-budget" created
statefulset "cockroachdb" created
~~~
{% include copy-clipboard.html %}
~~~ shell
$ kubectl create -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cockroachdb-statefulset.yaml
~~~

~~~
service "cockroachdb-public" created
service "cockroachdb" created
poddisruptionbudget "cockroachdb-budget" created
statefulset "cockroachdb" created
~~~

2. Confirm that three pods are `Running` successfully:

Expand Down
25 changes: 25 additions & 0 deletions _includes/orchestration/start-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,31 @@ Choose whether you want to orchestrate CockroachDB with Kubernetes using the hos

The process can take a few minutes, so don't move on to the next step until you see a `Creating cluster cockroachdb...done` message and details about your cluster.

{% if page.secure == true %}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@a-robinson, I added this filter to make sure these RBAC steps don't get pulled into the insecure tutorials.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also made sense to move this content into this include file. But it's only showing up for Hosted GKE. Is RBAC relevant for the other methods as well?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RBAC in general is relevant for other methods. These two steps below are a workaround for GKE limitations and aren't relevant for any other environments.


3. Get the email address associated with your Google Cloud account:

~~~ shell
$ gcloud info | grep Account
~~~

~~~
Account: [your.google.cloud.email@example.org]
~~~

4. [Create the RBAC roles](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control#prerequisites_for_using_role-based_access_control) CockroachDB needs for running on GKE, using the address from the previous step:

~~~ shell
$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=<your.google.cloud.email@example.org>
~~~

~~~
clusterrolebinding "cluster-admin-binding" created
~~~


{% endif %}

</div>

<div class="filter-content" markdown="1" data-scope="gce-manual">
Expand Down
54 changes: 17 additions & 37 deletions v1.1/orchestrate-cockroachdb-with-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ This page shows you how to orchestrate the deployment and management of a secure

If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.

{{site.data.alerts.callout_info}}Currently, it's possible to orchestrate a secure CockroachDB cluster using the <a href="https://cloud.google.com/kubernetes-engine/docs/">hosted Google Kubernetes Engine service</a>. Additional methods will be supported soon.{{site.data.alerts.end}}

<div id="toc"></div>

## Before You Begin
Expand All @@ -31,40 +29,17 @@ instance | A physical or virtual machine. In this tutorial, you'll create GCE or
[StatefulSet](http://kubernetes.io/docs/concepts/abstractions/controllers/statefulsets/) | A StatefulSet is a group of pods treated as stateful units, where each pod has distinguishable network identity and always binds back to the same persistent storage on restart. StatefulSets are a beta feature as of Kubernetes version 1.5.
[persistent volume](http://kubernetes.io/docs/user-guide/persistent-volumes/) | A persistent volume is a piece of networked storage (Persistent Disk on GCE, Elastic Block Store on AWS) mounted into a pod. The lifetime of a persistent volume is decoupled from the lifetime of the pod that's using it, ensuring that each CockroachDB node binds back to the same storage on restart.<br><br>This tutorial assumes that dynamic volume provisioning is available. When that is not the case, [persistent volume claims](http://kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims) need to be created manually.
[CSR](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/) | A CSR, or Certificate Signing Request, is a request to have a TLS certificate signed by the Kubernetes cluster's built-in CA. As each pod is created, it issues a CSR for the CockroachDB node running in the pod, which must be manually checked and approved. The same is true for clients as they connect to the cluster.
[RBAC](https://kubernetes.io/docs/admin/authorization/rbac/) | RBAC, or Role-Based Access Control, is the system Kubernetes uses to manage permissions within the cluster. In order to take an action (e.g., `get` or `create`) on an API resource (e.g., a `pod` or `CSR`), the client must have a `Role` that allows it to do so. This tutorial creates the RBAC resources necessary for CockroachDB to create and access certificates.

{% include orchestration/kubernetes-limitations.md %}

<!-- ## Step 1. Choose your deployment environment

Choose whether you want to orchestrate CockroachDB with Kubernetes using the hosted Google Kubernetes Engine (GKE) service or manually on Google Compute Engine (GCE) or AWS. The instructions below will change slightly depending on your choice.

<div class="filters filters-big clearfix">
<button class="filter-button" data-scope="gke-hosted">Hosted GKE</button>
<button class="filter-button" data-scope="gce-manual">Manual GCE</button>
<button class="filter-button" data-scope="aws-manual">Manual AWS</button>
</div>
-->
## Step 1. Start Kubernetes

1. Complete the **Before You Begin** steps described in the [Google Kubernetes Engine Quickstart](https://cloud.google.com/kubernetes-engine/docs/quickstart) documentation.

This includes installing `gcloud`, which is used to create and delete Kubernetes Engine clusters, and `kubectl`, which is the command-line tool used to manage Kubernetes from your workstation.

2. From your local workstation, start the Kubernetes cluster:

~~~ shell
$ gcloud container clusters create cockroachdb
~~~

This creates GKE instances and joins them into a single Kubernetes cluster named `cockroachdb`.

The process can take a few minutes, so don't move on to the next step until you see a `Creating cluster cockroachdb...done` message and details about your cluster.
{% include orchestration/start-kubernetes.md %}

## Step 2. Start CockroachDB nodes
## Step 3. Start CockroachDB nodes

{% include orchestration/start-cluster.md %}

## Step 3. Approve node certificates
## Step 4. Approve node certificates

As each pod is created, it issues a Certificate Signing Request, or CSR, to have the node's certificate signed by the Kubernetes CA. You must manually check and approve each node's certificates, at which point the CockroachDB node is started in the pod.

Expand Down Expand Up @@ -125,7 +100,7 @@ As each pod is created, it issues a Certificate Signing Request, or CSR, to have

4. Repeat steps 1-3 for the other 2 pods.

## Step 4. Initialize the cluster
## Step 5. Initialize the cluster

1. Confirm that three pods are `Running` successfully:

Expand Down Expand Up @@ -191,7 +166,7 @@ As each pod is created, it issues a Certificate Signing Request, or CSR, to have

{{site.data.alerts.callout_success}}The StatefulSet configuration sets all CockroachDB nodes to write to <code>stderr</code>, so if you ever need access to a pod/node's logs to troubleshoot, use <code>kubectl logs &lt;podname&gt;</code> rather than checking the log on the persistent volume.{{site.data.alerts.end}}

## Step 5. Test the cluster
## Step 6. Test the cluster

To use the built-in SQL client, you need to launch a pod that runs indefinitely with the `cockroach` binary inside it, check and approve the CSR for the pod, get a shell into the pod, and then start the built-in SQL client.

Expand Down Expand Up @@ -268,15 +243,15 @@ To use the built-in SQL client, you need to launch a pod that runs indefinitely

{{site.data.alerts.callout_success}}This pod will continue running indefinitely, so any time you need to reopen the built-in SQL client or run any other <a href="cockroach-commands.html"><code>cockroach</code> client commands</a>, such as <code>cockroach node</code> or <code>cockroach zone</code>, repeat step 2 using the appropriate <code>cockroach</code> command.<br></br>If you'd prefer to delete the pod and recreate it when needed, run <code>kubectl delete pod cockroachdb-client-secure</code>{{site.data.alerts.end}}

## Step 6. Monitor the cluster
## Step 7. Monitor the cluster

{% include orchestration/monitor-cluster.md %}

## Step 7. Simulate node failure
## Step 8. Simulate node failure

{% include orchestration/kubernetes-simulate-failure.md %}

## Step 8. Scale the cluster
## Step 9. Scale the cluster

{% include orchestration/kubernetes-scale-cluster.md %}

Expand Down Expand Up @@ -357,16 +332,15 @@ To use the built-in SQL client, you need to launch a pod that runs indefinitely

8. Back in the Admin UI, click **View nodes list** on the right to ensure that the fourth node successfully joined the cluster.

## Step 9. Stop the cluster
## Step 10. Stop the cluster

To shut down the CockroachDB cluster:

1. Delete all of the resources associated with the `cockroachdb` label, including the logs and remote persistent volumes:

{% include copy-clipboard.html %}
~~~ shell
$ kubectl delete pods,statefulsets,services,persistentvolumeclaims,persistentvolumes,poddisruptionbudget,jobs \
-l app=cockroachdb
$ kubectl delete pods,statefulsets,services,persistentvolumeclaims,persistentvolumes,poddisruptionbudget,jobs,rolebinding,clusterrolebinding,role,clusterrole,serviceaccount -l app=cockroachdb
~~~

~~~
Expand All @@ -382,6 +356,12 @@ To shut down the CockroachDB cluster:
persistentvolumeclaim "datadir-cockroachdb-2" deleted
persistentvolumeclaim "datadir-cockroachdb-3" deleted
poddisruptionbudget "cockroachdb-budget" deleted
job "cluster-init-secure" deleted
rolebinding "cockroachdb" deleted
clusterrolebinding "cockroachdb" deleted
role "cockroachdb" deleted
clusterrole "cockroachdb" deleted
serviceaccount "cockroachdb" deleted
~~~

2. Delete the pod created for `cockroach` client commands, if you didn't do so earlier:
Expand Down
54 changes: 17 additions & 37 deletions v1.2/orchestrate-cockroachdb-with-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ This page shows you how to orchestrate the deployment and management of a secure

If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.

{{site.data.alerts.callout_info}}Currently, it's possible to orchestrate a secure CockroachDB cluster using the <a href="https://cloud.google.com/kubernetes-engine/docs/">hosted Google Kubernetes Engine service</a>. Additional methods will be supported soon.{{site.data.alerts.end}}

<div id="toc"></div>

## Before You Begin
Expand All @@ -31,40 +29,17 @@ instance | A physical or virtual machine. In this tutorial, you'll create GCE or
[StatefulSet](http://kubernetes.io/docs/concepts/abstractions/controllers/statefulsets/) | A StatefulSet is a group of pods treated as stateful units, where each pod has distinguishable network identity and always binds back to the same persistent storage on restart. StatefulSets are a beta feature as of Kubernetes version 1.5.
[persistent volume](http://kubernetes.io/docs/user-guide/persistent-volumes/) | A persistent volume is a piece of networked storage (Persistent Disk on GCE, Elastic Block Store on AWS) mounted into a pod. The lifetime of a persistent volume is decoupled from the lifetime of the pod that's using it, ensuring that each CockroachDB node binds back to the same storage on restart.<br><br>This tutorial assumes that dynamic volume provisioning is available. When that is not the case, [persistent volume claims](http://kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims) need to be created manually.
[CSR](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/) | A CSR, or Certificate Signing Request, is a request to have a TLS certificate signed by the Kubernetes cluster's built-in CA. As each pod is created, it issues a CSR for the CockroachDB node running in the pod, which must be manually checked and approved. The same is true for clients as they connect to the cluster.
[RBAC](https://kubernetes.io/docs/admin/authorization/rbac/) | RBAC, or Role-Based Access Control, is the system Kubernetes uses to manage permissions within the cluster. In order to take an action (e.g., `get` or `create`) on an API resource (e.g., a `pod` or `CSR`), the client must have a `Role` that allows it to do so. This tutorial creates the RBAC resources necessary for CockroachDB to create and access certificates.

{% include orchestration/kubernetes-limitations.md %}

<!-- ## Step 1. Choose your deployment environment

Choose whether you want to orchestrate CockroachDB with Kubernetes using the hosted Google Kubernetes Engine (GKE) service or manually on Google Compute Engine (GCE) or AWS. The instructions below will change slightly depending on your choice.

<div class="filters filters-big clearfix">
<button class="filter-button" data-scope="gke-hosted">Hosted GKE</button>
<button class="filter-button" data-scope="gce-manual">Manual GCE</button>
<button class="filter-button" data-scope="aws-manual">Manual AWS</button>
</div>
-->
## Step 1. Start Kubernetes

1. Complete the **Before You Begin** steps described in the [Google Kubernetes Engine Quickstart](https://cloud.google.com/kubernetes-engine/docs/quickstart) documentation.

This includes installing `gcloud`, which is used to create and delete Kubernetes Engine clusters, and `kubectl`, which is the command-line tool used to manage Kubernetes from your workstation.

2. From your local workstation, start the Kubernetes cluster:

~~~ shell
$ gcloud container clusters create cockroachdb
~~~

This creates GKE instances and joins them into a single Kubernetes cluster named `cockroachdb`.

The process can take a few minutes, so don't move on to the next step until you see a `Creating cluster cockroachdb...done` message and details about your cluster.
{% include orchestration/start-kubernetes.md %}

## Step 2. Start CockroachDB nodes
## Step 3. Start CockroachDB nodes

{% include orchestration/start-cluster.md %}

## Step 3. Approve node certificates
## Step 4. Approve node certificates

As each pod is created, it issues a Certificate Signing Request, or CSR, to have the node's certificate signed by the Kubernetes CA. You must manually check and approve each node's certificates, at which point the CockroachDB node is started in the pod.

Expand Down Expand Up @@ -125,7 +100,7 @@ As each pod is created, it issues a Certificate Signing Request, or CSR, to have

4. Repeat steps 1-3 for the other 2 pods.

## Step 4. Initialize the cluster
## Step 5. Initialize the cluster

1. Confirm that three pods are `Running` successfully:

Expand Down Expand Up @@ -191,7 +166,7 @@ As each pod is created, it issues a Certificate Signing Request, or CSR, to have

{{site.data.alerts.callout_success}}The StatefulSet configuration sets all CockroachDB nodes to write to <code>stderr</code>, so if you ever need access to a pod/node's logs to troubleshoot, use <code>kubectl logs &lt;podname&gt;</code> rather than checking the log on the persistent volume.{{site.data.alerts.end}}

## Step 5. Test the cluster
## Step 6. Test the cluster

To use the built-in SQL client, you need to launch a pod that runs indefinitely with the `cockroach` binary inside it, check and approve the CSR for the pod, get a shell into the pod, and then start the built-in SQL client.

Expand Down Expand Up @@ -268,15 +243,15 @@ To use the built-in SQL client, you need to launch a pod that runs indefinitely

{{site.data.alerts.callout_success}}This pod will continue running indefinitely, so any time you need to reopen the built-in SQL client or run any other <a href="cockroach-commands.html"><code>cockroach</code> client commands</a>, such as <code>cockroach node</code> or <code>cockroach zone</code>, repeat step 2 using the appropriate <code>cockroach</code> command.<br></br>If you'd prefer to delete the pod and recreate it when needed, run <code>kubectl delete pod cockroachdb-client-secure</code>{{site.data.alerts.end}}

## Step 6. Monitor the cluster
## Step 7. Monitor the cluster

{% include orchestration/monitor-cluster.md %}

## Step 7. Simulate node failure
## Step 8. Simulate node failure

{% include orchestration/kubernetes-simulate-failure.md %}

## Step 8. Scale the cluster
## Step 9. Scale the cluster

{% include orchestration/kubernetes-scale-cluster.md %}

Expand Down Expand Up @@ -357,16 +332,15 @@ To use the built-in SQL client, you need to launch a pod that runs indefinitely

8. Back in the Admin UI, click **View nodes list** on the right to ensure that the fourth node successfully joined the cluster.

## Step 9. Stop the cluster
## Step 10. Stop the cluster

To shut down the CockroachDB cluster:

1. Delete all of the resources associated with the `cockroachdb` label, including the logs and remote persistent volumes:

{% include copy-clipboard.html %}
~~~ shell
$ kubectl delete pods,statefulsets,services,persistentvolumeclaims,persistentvolumes,poddisruptionbudget,jobs \
-l app=cockroachdb
$ kubectl delete pods,statefulsets,services,persistentvolumeclaims,persistentvolumes,poddisruptionbudget,jobs,rolebinding,clusterrolebinding,role,clusterrole,serviceaccount -l app=cockroachdb
~~~

~~~
Expand All @@ -382,6 +356,12 @@ To shut down the CockroachDB cluster:
persistentvolumeclaim "datadir-cockroachdb-2" deleted
persistentvolumeclaim "datadir-cockroachdb-3" deleted
poddisruptionbudget "cockroachdb-budget" deleted
job "cluster-init-secure" deleted
rolebinding "cockroachdb" deleted
clusterrolebinding "cockroachdb" deleted
role "cockroachdb" deleted
clusterrole "cockroachdb" deleted
serviceaccount "cockroachdb" deleted
~~~

2. Delete the pod created for `cockroach` client commands, if you didn't do so earlier:
Expand Down