Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
README.md
client-secure.yaml
cluster-init-secure.yaml
cockroachdb-statefulset-secure.yaml
dns-lb.yaml
example-app-secure.yaml
external-name-svc.yaml
setup.py
teardown.py

README.md

Running CockroachDB across multiple Kubernetes clusters

The script and configuration files in this directory enable deploying CockroachDB across multiple Kubernetes clusters that are spread across different geographic regions. It deploys a CockroachDB StatefulSet into each separate cluster, and links them together using DNS.

To use the configuration provided here, check out this repository (or otherwise download a copy of this directory), fill in the constants at the top of setup.py with the relevant information about your Kubernetes clusters, optionally make any desired modifications to cockroachdb-statefulset-secure.yaml as explained in our Kubernetes performance tuning guide, then finally run setup.py.

You should see a lot of output as it does its thing, hopefully ending after printing out job "cluster-init-secure" created. This implies that everything was created successfully, and you should soon see the CockroachDB cluster initialized with 3 pods in the "READY" state in each Kubernetes cluster. At this point you can manage the StatefulSet in each cluster independently if you so desire, scaling up the number of replicas, changing their resource requests, or making other modifications as you please.

If anything goes wrong along the way, please let us know via any of the normal troubleshooting channels. While we believe this creates a highly available, maintainable multi-region deployment, it is still pushing the boundaries of how Kubernetes is typically used, so feedback and issue reports are very appreciated.

Limitations

Pod-to-pod connectivity

The deployment outlined in this directory relies on pod IP addresses being routable even across Kubernetes clusters and regions. This achieves optimal performance, particularly when compared to alternative solutions that route all packets between clusters through load balancers, but means that it won't work in certain environments.

This requirement is satisfied by clusters deployed in cloud environments such as Google Kubernetes Engine, and can also be satisfied by on-prem environments depending on the Kubernetes networking setup used. If you want to test whether your cluster will work, you can run this basic network test:

$ kubectl run network-test --image=alpine --restart=Never -- sleep 999999
pod "network-test" created
$ kubectl describe pod network-test | grep IP
IP:           THAT-PODS-IP-ADDRESS
$ kubectl config use-context YOUR-OTHER-CLUSTERS-CONTEXT-HERE
$ kubectl run -it network-test --image=alpine --restart=Never -- ping THAT-PODS-IP-ADDRESS
If you don't see a command prompt, try pressing enter.
64 bytes from 10.12.14.10: seq=1 ttl=62 time=0.570 ms
64 bytes from 10.12.14.10: seq=2 ttl=62 time=0.449 ms
64 bytes from 10.12.14.10: seq=3 ttl=62 time=0.635 ms
64 bytes from 10.12.14.10: seq=4 ttl=62 time=0.722 ms
64 bytes from 10.12.14.10: seq=5 ttl=62 time=0.504 ms
...

If the pods can directly connect, you should see successful ping output like the above. If they can't, you won't see any successful ping responses. Make sure to delete the network-test pod in each cluster when you're done!

Exposing DNS servers to the Internet

As currently configured, the way that the DNS servers from each Kubernetes cluster are hooked together is by exposing them via a load balanced IP address that's visible to the public Internet. This is because Google Cloud Platform's Internal Load Balancers do not currently support clients in one region using a load balancer in another region.

None of the services in your Kubernetes cluster will be made accessible, but their names could leak out to a motivated attacker. If this is unacceptable, please let us know and we can demonstrate other options. Your voice could also help convince Google to allow clients from one region to use an Internal Load Balancer in another, eliminating the problem.

Cleaning up

To remove all the resources created in your clusters by setup.py, copy the parameters you provided at the top of setup.py to the top of teardown.py and run teardown.py.

More information

For more information on running CockroachDB in Kubernetes, please see the README in the parent directory.