docs: add operator manual for GCP #62

Ajarmar · 2021-02-17T14:34:12Z

Added operator manual for setting up Compliant Kubernetes on GCP using compliantkubernetes-kubespray and compliantkubernetes-apps, using the GCP Persistent Disk Driver for block storage.

Fixes elastisys/compliantkubernetes-kubespray#14 and #58

cristiklein

Nice! Please fix:

the minor issues I suggested
double-check the formatting with mkdocs serve
fix the pipeline errors

cristiklein · 2021-02-17T14:42:15Z

docs/operator-manual/gcp.md

+4. Modify `kubespray/contrib/terraform/gcp/tfvars.json` in the following way:
+    - Set `gcp_project_id` to the ID of your GCP project.
+    - Set `keyfile_location` to the location of your JSON keyfile.
+    - Set `ssh_pub_key` to the path of your public ssh key.


Please capitalize SSH throughout.

Also, somewhat orthogonal to the present PR, please note that for Exoscale we opted to include the SSH key inline to facilitate operators sharing operation of a cluster. See full discussion here.

~~Since this is a variable name I think we should stick to the naming convention of terraform~~
EDIT: nvm. I didn't see the ssh word in the end of the sentence

cristiklein · 2021-02-17T14:44:27Z

docs/operator-manual/gcp.md

+2. In `compliantkubernetes-apps`, run:
+    ```bash
+    export CK8S_ENVIRONMENT_NAME=<environment-name>
+    export CK8S_CLOUD_PROVIDER=baremetal


A few questions for myself:

Do we use GCP load-balancers?

Do we have an issue to iron this out?

cristiklein · 2021-02-17T14:46:31Z

docs/operator-manual/gcp.md

+
+Note that in release v0.9.0 of compliantkubernetes-apps, fluentd will not work in the service cluster.
+
+1. Set up [ck8s-dns](https://github.com/elastisys/ck8s-dns) on a provider of your choice, using the `ingress_controller_lb_ip_address` from `terraform apply` as your loadbalancer IPs.


This repo is internal. Can you write what DNS entries to create in the following format:

echo " *.$BASE_DOMAIN 60s A 203.0.113.123 *.ops.$BASE_DOMAIN 60s A 203.0.113.123 "

llarsson

Very nice! Some comments/questions, and then I am really looking forward to learning what @ph4n666's run through of this stuff shows. :)

docs/operator-manual/gcp.md

llarsson · 2021-02-17T14:48:30Z

docs/operator-manual/gcp.md

+
+The following instructions were made for release v0.9.0 of compliantkubernetes-apps. There may be discrepancies with newer versions.
+
+Note that in release v0.9.0 of compliantkubernetes-apps, fluentd will not work in the service cluster.


Please add a line about why it will not work. What alternative should they go with instead?

Don't we need a for loop for every bash script mentioned in this commit?

docs/operator-manual/gcp.md

Xartos

Great tutorial! just some minor comments

docs/operator-manual/gcp.md

Xartos · 2021-02-17T15:43:33Z

docs/operator-manual/gcp.md

+    - Set `keyfile_location` to the location of your JSON keyfile.
+    - Set `ssh_pub_key` to the path of your public ssh key.
+    - In `ssh_whitelist`, `api_server_whitelist` and `nodeport_whitelist`, add IP address(es) that you want to be able to access the cluster.
+5. Set up the nodes by performing the following steps, replacing `<prefix>` with `sc`/`wc`:


I would go with something more generic, like

Suggested change

5. Set up the nodes by performing the following steps, replacing `<prefix>` with `sc`/`wc`:

5. Set up the nodes by performing the following steps, replacing `<prefix>` with `my-sc-cluster`/`my-wc-cluster`:

To make it feel like you want to change this and not that it's a requirement to use sc/wc

The compliantkubernetes-kubespray readme still says:

For now you need to set this to wc or sc if you want to install compliantkubernetes apps on top afterwards, this restriction will be removed later.

Is this no longer true? I'm also a bit unsure in general how much this documentation should accommodate for multitenancy setups since there still are some issues on the apps side, like the DNS problem

I think that is no longer true. Otherwise, how did I set up two WCs on AWS? 😄

Can you clarify "DNS problem"? I am aware of a few sharp corners, but not blockers.

I'm not sure about the specifics but @lentzi90 and @pettersv ran into some DNS issues when setting up a MT cluster. Not really "blockers" because it was still possible to set it up, but there were some limitations as a consequence. You'd have to ask them for the details

For compliantkubernetes-kubespray it will work fine with different names now. In apps it is a bit rough around the edges since you will need to use each prefix as a separate CK8S_CONFIG_PATH and the kubeconfigs for the workload clusters must all be named kube_config_wc.yaml. See elastisys/compliantkubernetes-apps#85.

The DNS issue is that the service cluster cannot measure uptime or alert correctly for WC API servers. It will think that they (actually "it", as it only knows about one) are down and send out alerts accordingly. This is because all clusters in one environment must have the same ops and base domains pointing to the SC for this to work. This means that the health check for the WC API server ends up targeting SC instead of one of the WCs and of course fails. So expect constant alerts. 😉 🚨

@lentzi90 Do we have an issue for the uptime alert?

Regarding the former, this does the trick for me:

for CLUSTER in $WORKLOAD_CLUSTERS; do ln -sf $CK8S_CONFIG_PATH/.state/kube_config_${CLUSTER}.yaml $CK8S_CONFIG_PATH/.state/kube_config_wc.yaml ./bin/ck8s apply wc # Respond "n" if you get a WARN done

Created an issue now for the DNS/alerting: elastisys/compliantkubernetes-apps#253

I bet your snippet works great Cristian. Could you add it somewhere so that it is impossible to miss it when setting up a cluster?

The program PC Pitstop included a clause in their end-user license agreement stating that anybody who read the clause and contacted the company would receive a monetary reward, but it took four months and over 3,000 software downloads before anybody collected it.

Should I include something similar around my snippet? 😂

One more:

Please don't contact me before you've read all the relevant parts of this page. I will know if you haven't and I'll ignore your message.
[2 screen scrolls later]
Write me a letter that indicates that you've read this page by including the phrase “parens rock”,

Xartos · 2021-02-17T15:45:56Z

docs/operator-manual/gcp.md

+    1. Set up the nodes with terraform. If desired, first modify `"machines"` in `kubespray/contrib/terraform/gcp/tfvars.json` to add/remove nodes, change node sizes, etc. (For setting up compliantkubernetes-apps in the service cluster, one `n1-standard-8` worker and one `n1-standard-4` worker is enough.)
+    ```bash
+    cd kubespray/contrib/terraform/gcp
+    export CLUSTER=<prefix>


Would we want to have the for loop as we have in the AWS and Exoscale tutorials here? To make them more inline with each other?

Yes, please we support multiple workload clusters. 😄

Xartos · 2021-02-17T15:47:31Z

docs/operator-manual/gcp.md

+5. Set up the nodes by performing the following steps, replacing `<prefix>` with `sc`/`wc`:
+    1. Set up the nodes with terraform. If desired, first modify `"machines"` in `kubespray/contrib/terraform/gcp/tfvars.json` to add/remove nodes, change node sizes, etc. (For setting up compliantkubernetes-apps in the service cluster, one `n1-standard-8` worker and one `n1-standard-4` worker is enough.)
+    ```bash
+    cd kubespray/contrib/terraform/gcp


Since you use this for other snippets as well. Should you use pushd/popd so that you don't need to run cd ../../../../ between each snippet?

Yeah, that's a bit nicer if you want the commands to just be copy-pastable - my idea with writing it the way I did was just to make it clear which folder the commands should be executed in, not necessarily that the user should cd ../../../../ after each snippet

Yea, I totally understand. That's why I like the pushd/popd because then it's clear where the commands are running AND you can still copy/paste it and it will automagically work. Best of both worlds 😉

Xartos · 2021-02-17T15:48:45Z

docs/operator-manual/gcp.md

+    ```  
+    * `path to ssh key` should point to your private ssh key. It will be copied into your config path and encrypted with SOPS, the original file left as it were.
+    * `SOPS fingerprint` is the gpg fingerprint that will be used for SOPS encryption. You need to set this or the environment variable `CK8S_PGP_FP` the first time SOPS is used in your specified config path.
+    4. Edit the IP addresses and nodes in your `inventory.ini` (found in your config path) to match the VMs that should be part of the cluster. The contents of the `$CLUSTER-inventory.ini` file that you generated in the previous section can be copy-pasted into the appropriate `inventory.ini` file.


You should be able to just mv $CLUSTER-inventory.ini inventory.ini right?
Or is there some things missing?

Yeah, that should work fine.

docs/operator-manual/gcp.md

Xartos · 2021-02-17T15:55:02Z

docs/operator-manual/gcp.md

+    bin/ck8s ops kubectl sc "patch storageclass csi-gce-pd -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"true\"}}}'"
+    bin/ck8s ops kubectl wc "patch storageclass csi-gce-pd -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"true\"}}}'"


I'm guessing this isn't possible to do in kubespray?

I'm not sure about this, haven't really looked into it.

Pavan-Gunda · 2021-02-17T15:20:44Z

docs/operator-manual/gcp.md

+
+The following instructions were made for release v0.9.0 of compliantkubernetes-apps. There may be discrepancies with newer versions.
+
+Note that in release v0.9.0 of compliantkubernetes-apps, fluentd will not work in the service cluster.


Don't we need a for loop for every bash script mentioned in this commit?

cristiklein

LGTM.

docs/operator-manual/gcp.md

Ajarmar requested review from cristiklein, Pavan-Gunda and Xartos February 17, 2021 14:34

Ajarmar force-pushed the axel/gcp-operator-docs branch from 97b5378 to 352df35 Compare February 17, 2021 14:40

cristiklein approved these changes Feb 17, 2021

View reviewed changes

llarsson requested changes Feb 17, 2021

View reviewed changes

Xartos reviewed Feb 17, 2021

View reviewed changes

Pavan-Gunda requested changes Feb 18, 2021

View reviewed changes

Ajarmar force-pushed the axel/gcp-operator-docs branch from 352df35 to 3043db4 Compare February 22, 2021 12:24

Ajarmar requested review from llarsson, cristiklein, Xartos and Pavan-Gunda February 22, 2021 12:32

cristiklein approved these changes Feb 22, 2021

View reviewed changes

docs/operator-manual/gcp.md Outdated Show resolved Hide resolved

Ajarmar force-pushed the axel/gcp-operator-docs branch from 3043db4 to 1414b61 Compare February 22, 2021 14:16

llarsson approved these changes Mar 1, 2021

View reviewed changes

docs: add operator manual for GCP

fd76b3e

Ajarmar force-pushed the axel/gcp-operator-docs branch from 1414b61 to fd76b3e Compare March 1, 2021 09:07

Pavan-Gunda approved these changes Mar 1, 2021

View reviewed changes

Ajarmar merged commit c0b0ad3 into main Mar 1, 2021

cristiklein deleted the axel/gcp-operator-docs branch November 19, 2021 08:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add operator manual for GCP #62

docs: add operator manual for GCP #62

Ajarmar commented Feb 17, 2021

cristiklein left a comment

cristiklein Feb 17, 2021

cristiklein Feb 17, 2021

Xartos Feb 17, 2021 •

edited

Loading

cristiklein Feb 17, 2021

cristiklein Feb 17, 2021

llarsson left a comment

llarsson Feb 17, 2021

Pavan-Gunda Feb 17, 2021

Xartos left a comment

Xartos Feb 17, 2021

Ajarmar Feb 18, 2021

cristiklein Feb 18, 2021

Ajarmar Feb 18, 2021

lentzi90 Feb 18, 2021

cristiklein Feb 18, 2021

lentzi90 Feb 18, 2021

cristiklein Feb 18, 2021

cristiklein Feb 18, 2021

Xartos Feb 17, 2021

cristiklein Feb 18, 2021

Xartos Feb 17, 2021

Ajarmar Feb 18, 2021

Xartos Feb 18, 2021

Xartos Feb 17, 2021

Ajarmar Feb 18, 2021

Xartos Feb 17, 2021

Ajarmar Feb 18, 2021

Pavan-Gunda Feb 17, 2021

cristiklein left a comment


		Note that in release v0.9.0 of compliantkubernetes-apps, fluentd will not work in the service cluster.

		1. Set up [ck8s-dns](https://github.com/elastisys/ck8s-dns) on a provider of your choice, using the `ingress_controller_lb_ip_address` from `terraform apply` as your loadbalancer IPs.


		The following instructions were made for release v0.9.0 of compliantkubernetes-apps. There may be discrepancies with newer versions.

		Note that in release v0.9.0 of compliantkubernetes-apps, fluentd will not work in the service cluster.

	5. Set up the nodes by performing the following steps, replacing `<prefix>` with `sc`/`wc`:
	5. Set up the nodes by performing the following steps, replacing `<prefix>` with `my-sc-cluster`/`my-wc-cluster`:

		bin/ck8s ops kubectl sc "patch storageclass csi-gce-pd -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"true\"}}}'"
		bin/ck8s ops kubectl wc "patch storageclass csi-gce-pd -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"true\"}}}'"

docs: add operator manual for GCP #62

docs: add operator manual for GCP #62

Conversation

Ajarmar commented Feb 17, 2021

cristiklein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xartos Feb 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

llarsson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xartos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cristiklein left a comment

Choose a reason for hiding this comment

Xartos Feb 17, 2021 •

edited

Loading