Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
README.md
gke-cluster.json
gke-deployment.json
pke-cluster.json
pke-deployment.json

README.md

Multi cloud, multi cluster deployment

In this scenario we deploy CockroachDB to a multiple Kubernetes clusters in two separate clouds:

  • PKE (Pipeline Kubernetes Engine) on AWS
  • GKE

Overview

This scenario contains a more complex example of what can be done with Pipeline and CockroachDB.

CockroachDB is installed using the official Helm chart.

Prerequisites

Make sure you followed the Pipeline, Amazon and Google setup instructions in the repo readme.

Setup your environment (skip if you followed the instructions in the repo readme in this session):

export PIPELINE_TOKEN=$(cat ~/.banzai/config.yaml | grep "token:" | sed 's/.*token: //')
export AWS_REGION=us-west-1
export AWS_BUCKET=cockroachtest
export AWS_ZONE=us-west-1a
export AWS_SECRET_ID=$(banzai secret list --type amazon --output json | jq -r '.[] | first(select(.name=="aws")).id')
export GCE_PROJECT=my_project
export GCE_REGION=us-west1
export GCE_BUCKET=cockroachtest
export GCE_ZONE=us-west1-a
export GCE_SECRET_ID=$(banzai secret list --type google --output json | jq -r '.[] | first(select(.name=="gce")).id')

Steps

  1. Create a new PKE cluster:
eval "echo \"$(cat pke-cluster.json | sed 's/"/\\"/g')\"" | banzai cluster create
  1. Create a new GKE cluster:
eval "echo \"$(cat gke-cluster.json | sed 's/"/\\"/g')\"" | banzai cluster create
  1. Wait for the clusters to become ready:
banzai cluster get pke-multi-test
banzai cluster get gke-multi-test
  1. Log in to the PKE cluster
banzai cluster shell --cluster-name pke-multi-test
  1. Install CockroachDB on the PKE cluster
HELM_PACKAGE=$(cat ../../charts/cockroachdb-wrapper-0.1.0.tgz | base64) eval "echo \"$(cat pke-deployment.json | sed 's/"/\\"/g')\"" | curl -X POST https://beta.banzaicloud.io/pipeline/api/v1/orgs/$BANZAI_CURRENT_ORG_ID/clusters/$BANZAI_CURRENT_CLUSTER_ID/deployments -H "Content-Type: application/json" -H "Authorization: Bearer $PIPELINE_TOKEN" -d @-

Note: DNS entries may take a few minutes to set up.

  1. Exit the cluster shell

  2. Log in to the GKE cluster

banzai cluster shell --cluster-name gke-multi-test
  1. Export the remote node addresses as envirnment variables
export CDB_HOST_1=cockroachdb-cockroachdb-0-node-pke-multi-test.spotguidetest.beta.banzaicloud.io
export CDB_HOST_2=cockroachdb-cockroachdb-1-node-pke-multi-test.spotguidetest.beta.banzaicloud.io
export CDB_HOST_3=cockroachdb-cockroachdb-2-node-pke-multi-test.spotguidetest.beta.banzaicloud.io
  1. Install CockroachDB
HELM_PACKAGE=$(cat ../../charts/cockroachdb-wrapper-0.1.0.tgz | base64) eval "echo \"$(cat gke-deployment.json | sed 's/"/\\"/g')\"" | curl -X POST https://beta.banzaicloud.io/pipeline/api/v1/orgs/$BANZAI_CURRENT_ORG_ID/clusters/$BANZAI_CURRENT_CLUSTER_ID/deployments -H "Content-Type: application/json" -H "Authorization: Bearer $PIPELINE_TOKEN" -d @-

Note: DNS entries may take a few minutes to set up.

Usage

Get the public hostname of the service and connect to CockroachDB:

export CDB_HOST=$(kubectl get svc cockroachdb-cockroachdb-public -o  jsonpath='{.status.loadBalancer.ingress[0].ip}')
cockroachdb sql --insecure --url "postgresql://root@$CDB_HOST:26257?sslmode=disable"

Open CockroachDB dashboard:

open "http://cockroach-${BANZAI_CURRENT_CLUSTER_NAME}.${BANZAI_CURRENT_ORG_NAME}.beta.banzaicloud.io"

Get Grafana secrets and login to the Grafana dashboard with admin user:

banzai secret get --name cluster-${BANZAI_CURRENT_CLUSTER_ID}-grafana --output json | jq -r '.[] | first(.values.password)'
open "https://${BANZAI_CURRENT_CLUSTER_NAME}.${BANZAI_CURRENT_ORG_NAME}.beta.banzaicloud.io/grafana"

Check there are log archives in the bucket:

aws s3 ls s3://$AWS_BUCKET/logs/fluent.info/$(date +%Y)/$(date +%m)/$(date +%d)
gsutil ls -p $GCE_PROJECT gs://$GCE_BUCKET/logs/fluent.info/$(date +%Y)/$(date +%m)/$(date +%d)

Exit the cluster shell when you are done.

Delete the clusters

You can remove the clusters with the following commands (if you don't need them anymore):

banzai cluster delete --no-interactive pke-multi-test
banzai cluster delete --no-interactive gke-multi-test
You can’t perform that action at this time.