Docker Enterprise 3-node cluster on Digital Ocean
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cafe
roles
.gitignore
README.md
ansible-hosts.template
ansible.cfg
babyswarm.auto.tfvars.template
babyswarm.tf
digitalocean-secret.yml.template
install-dockeree.yml
install-swarm.yml
install-ucp.yml
setdomain.sh
ssh.cfg

README.md

Docker Enterprise on Digital Ocean

This repository serves files used along the articles:

You'll find here terraform scripts and ansible playbooks that provision a whole Docker Enterprise cluster that:

  • Uses native DigitalOcean block storage for kubernetes persistent volumes
  • Uses native DigitalOcean load balancers for kubernetes ingress controllers
  • Configures HTTPS termination for load balancers or ingress endpoints

Quick steps

Steps described in the articles resumed here:

  1. Own a domain or subdomain managed by/delegated to DigitalOcean (like "devops.mycompany.com")

  2. Clone this repo

  3. Run:

./setdomain.sh <YOUR-DOMAIN>
  1. Edit "babyswarm.auto.tfvars" and provide token & key fingerprints

  2. Terraform your cluster

terraform init
terraform apply
  1. Test SSH connectivity to nodes
ansible -m ping all
  1. Install Docker Enterprise Engine
ansible-playbook install-dockeree.yml
  1. Test docker engines with ansible:
ansible -a "docker version" all
  1. Create and backup certificates for UCP node
export DOCKER_HOST=ssh://root@do-manager.devops.mycompany.com
docker volume create ucp-controller-server-certs
# creates certs with certbot
docker run --rm -ti \
  -p 80:80 -p 443:443 \
  -v ucp-controller-server-certs:/etc/letsencrypt \
  certbot/certbot certonly --standalone \
  --email admin@example.com \
  -n --agree-tos \
  -d ucp.devops.mycompany.com
# copies certs to UCP "hotspot"
docker run --rm -v ucp-controller-server-certs:/dummy \
  -w /dummy/live/ucp.devops.mycompany.com \
  alpine sh -c "cp privkey.pem /dummy/key.pem && \
     cp fullchain.pem /dummy/ca.pem && \
     cp fullchain.pem /dummy/cert.pem"

...OR, if you already have the certificates:

export DOCKER_HOST=ssh://root@do-manager.devops.mycompany.com
docker volume create ucp-controller-server-certs
docker run --rm -d --name dummy \
  -v ucp-controller-server-certs:/etc/letsencrypt \
  alpine tail -f /dev/null
docker cp ./letsencrypt dummy:/etc/
docker stop dummy
  1. Install UCP
ansible-playbook install-ucp.yml

UCP will be available at https://ucp.devops.mycompany.com

  1. Install worker nodes
ansible-playbook install-swarm.yml
  1. Change orchestrator for worker nodes

  2. Create and download client bundle, run "env.sh" script, list nodes

. ./env.sh
docker node ls
  1. Create and edit "digitalocean-secret.yml" from template, deploy secret
cp digitalocean-secret.yml.template digitalocean-secret.yml
<edit file>
kubectl apply -f digitalocean-secret.yml
  1. Install Storage CSI, check config, create dummy PVC
kubectl apply -f https://raw.githubusercontent.com/digitalocean/csi-digitalocean/master/deploy/kubernetes/releases/csi-digitalocean-v0.2.0.yaml
kubectl get sc
kubectl apply -f https://raw.githubusercontent.com/digitalocean/csi-digitalocean/master/examples/kubernetes/deployment-single-volume/pvc.yaml
  1. Install CCM
kubectl apply -f https://raw.githubusercontent.com/digitalocean/digitalocean-cloud-controller-manager/master/releases/v0.1.8.yml
  1. Install Helm
helm init
kubectl create rolebinding default-view \
  --clusterrole=view \
  --serviceaccount=kube-system:default \
  --namespace=kube-system
kubectl create clusterrolebinding add-on-cluster-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:default
  1. Create wildcard certificate for apps domain, upload it to DO
docker run --rm -ti \
  -v $(pwd)/letsencrypt:/etc/letsencrypt \
  certbot/certbot certonly --agree-tos \
  -d "*.apps.devops.mycompany.com" \
  --preferred-challenges=dns --manual \
  --email=admin@example.com
<create "_acme-challenge.apps" entry in DNS as requested>
<wait until certificate is created>
doctl compute certificate create \
  --private-key-path letsencrypt/live/apps.devops.mycompany.com/privkey.pem \
  --certificate-chain-path ./letsencrypt/live/apps.devops.mycompany.com/fullchain.pem \
  --leaf-certificate-path ./letsencrypt/live/apps.devops.mycompany.com/fullchain.pem \
  --name apps-devops
  1. Get certificate ID, install Ingress Controller (HTTPS in DO LB only)
doctl compute certificate list [-t "YOUR-DO-TOKEN-HERE"]
<FIND YOUR-CERT-ID>
helm install stable/nginx-ingress \
  --name my-nginx \
  --set rbac.create=true \
  --namespace nginx-ingress \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/do-loadbalancer-protocol"="http" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/do-loadbalancer-algorithm"="round_robin" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/do-loadbalancer-tls-ports"="443" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/do-loadbalancer-certificate-id"="YOUR-CERT-ID-HERE" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/do-loadbalancer-healthcheck-path"="/healthz" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/do-loadbalancer-redirect-http-to-https"="true" \
  --set controller.service.targetPorts.https="http"

This creates a new load balancer in DO that points to Ingress Controller

  1. Create "*.apps.devops.mycompany.com" wildcard DNS entry, use the load balancer IP, test health endpoint
curl xxx.apps.devops.mycompany.com/healthz