Merge b952e48 into 5b2b6c0

contrib/digitalocean/README.md

@@ -1,9 +1,106 @@
-Deis on DigitalOcean
-====================
+# Provision a Deis Cluster on Digital Ocean
 
-Unfortunately, DigitalOcean does not yet provide CoreOS images. This
-prevents Deis from being deployed on DigitalOcean.
+Note that Digital Ocean does not support CoreOS images natively. To work around this, the provision
+scripts for Digital Ocean first create a CoreOS image which will be used for provisioning the cluster.
 
-If you use DigitalOcean, please
-[show your support](http://digitalocean.uservoice.com/forums/136585-digital-ocean/suggestions/4250154-suport-coreos-as-a-deployment-platform)
-for CoreOS and help us to support Deis on DO.
+Digital Ocean support was contributed by [sttts](https://github.com/sttts). The CoreOS bootstrapping
+is heavily based on [Levi Aul's code](https://gist.github.com/tsutsu/490f35f48897df0f5173).
+
+To deploy Deis to Digital Ocean:
+
+## Customize cloud-config.yml
+Edit [user-data](../coreos/user-data) and add a discovery URL. This URL will be used by all nodes in this Deis cluster. You can get a new discovery URL by sending a request to http://discovery.etcd.io/new.
+
+## Install tugboat and authorize:
+The tugboat gem consumes the Digital Ocean API.
+```console
+$ gem install tugboat
+$ tugboat authorize
+```
+You can leave all but the client and API keys as the defaults.
+
+## Upload keys
+Choose an SSH keypair to use for Deis and import it to Digital Ocean:
+```console
+$ tugboat add-key deis
+```
+
+Then, get the ID of the key:
+```console
+$ tugboat keys
+```
+
+## Create a Deis image:
+```console
+$ ./provision-digitalocean-deis-image.sh <SSH KEY ID>
+```
+
+## Choose number of instances
+By default, the script will provision 3 servers. You can override this by setting `DEIS_NUM_INSTANCES`:
+```console
+$ export DEIS_NUM_INSTANCES=5
+```
+
+Note that for scheduling to work properly, clusters must consist of at least 3 nodes and always have an odd number of members.
+For more information, see [optimal etcd cluster size](https://github.com/coreos/etcd/blob/master/Documentation/optimal-cluster-size.md).
+
+Deis clusters of less than 3 nodes are unsupported.
+
+## Deploy cluster
+Run the provision script:
+```console
+$ ./provision-do-cluster.sh <REGION_ID> <IMAGE_ID> <SSH_ID> <SIZE>
+```
+
+Not all regions allow private networks. Choose one which does, e.g. NY 2, Amsterdam 2 or
+Singapore 1 at the time of this writing (check the web UI for the current private network
+support). You can enumerate all the regions with:
+
+```console
+$ tugboat regions
+```
+
+The provisioning script uses a 512 MB droplet by default because for image creation
+more memory is not needed. Deis controller nodes will need at least 2 GB to even start all
+the services. Add the memory requirements of deployed applications and choose an adequate
+droplet size. The default is 8 GB (ID "65"). You can enumerate all sizes with:
+
+```console
+$ tugboat sizes
+```
+
+## Choose number of routers
+By default, the Makefile will provision 1 router. You can override this by setting `DEIS_NUM_ROUTERS`:
+```console
+$ export DEIS_NUM_ROUTERS=2
+```
+
+## Initialize the cluster
+Once the cluster is up, get the IPs of any of the machines using `tugboat droplets`, set
+FLEETCTL_TUNNEL to one of these IPs:
+```console
+$ export FLEETCTL_TUNNEL=23.253.219.94
+$ cd ../.. && make run
+```
+The script will deploy Deis and make sure the services start properly.
+
+### Configure DNS
+You'll need to configure DNS records so you can access applications hosted on Deis. See [Configuring DNS](http://docs.deis.io/en/latest/operations/configure-dns/) for details.
+
+### Use Deis!
+After that, register with Deis!
+```console
+$ deis register http://deis.example.org
+username: deis
+password:
+password (confirm):
+email: info@opdemand.com
+```
+
+## Hack on Deis
+If you'd like to use this deployment to build Deis, you'll need to set `DEIS_HOSTS` to an array of your cluster hosts:
+```console
+$ DEIS_HOSTS="1.2.3.4 2.3.4.5 3.4.5.6" make build
+```
+
+This variable is used in the `make build` command.

contrib/digitalocean/cloud-config.yml

@@ -0,0 +1,76 @@
+#cloud-config
+
+hostname: HOSTNAME
+
+users:
+  - name: core
+    groups:
+      - sudo
+      - docker
+    ssh-authorized-keys:
+      - SSH_KEY
+
+coreos:
+  units:
+    - name: public.network
+      content: |
+        [Match]
+        Name=ens3
+
+        [Network]
+        Address=PUBLIC_IP
+        Gateway=GATEWAY
+        DNS=8.8.8.8
+        DNS=8.8.4.4
+    - name: private.network
+      content: |
+        [Match]
+        Name=ens4v1
+
+        [Network]
+        Address=PRIVATE_IP
+    - name: media-doroot.mount
+      command: start
+      content: |
+        [Mount]
+        What=/dev/vda
+        Where=/media/doroot
+        Type=ext4
+    - name: format-docker-store.service
+      command: start
+      content: |
+        [Unit]
+        Requires=media-doroot.mount
+        [Service]
+        Type=oneshot
+        RemainAfterExit=yes
+        ExecStart=/usr/share/oem/bin/create-coreos-docker-store
+    - name: var-lib-docker.mount
+      command: start
+      content: |
+        [Unit]
+        Requires=format-docker-store.service
+        Before=docker.service
+        [Mount]
+        What=/dev/disk/by-label/docker
+        Where=/var/lib/docker
+        Type=btrfs
+    - name: coreos-setup-environment.service
+      command: restart
+      runtime: yes
+      content: |
+        [Unit]
+        Before=docker.service
+        [Service]
+        Type=oneshot
+        ExecStart=/usr/share/oem/bin/coreos-setup-environment /etc/environment
+    - name: coreos-apply-user-data.service
+      command: restart
+      runtime: yes
+      content: |
+        [Unit]
+        After=coreos-setup-environment.service
+        [Service]
+        EnvironmentFile=/etc/environment
+        Type=oneshot
+        ExecStart=/usr/share/oem/bin/coreos-apply-user-data

contrib/digitalocean/coreos-apply-user-data

@@ -0,0 +1,2 @@
+#!/bin/bash
+coreos-cloudinit --from-file=/usr/share/oem/user-data.yml

contrib/digitalocean/coreos-setup-environment

@@ -0,0 +1,21 @@
+#!/bin/bash -e
+
+ENV=$1
+
+if [ -z "$ENV" ]; then
+        echo usage: $0 /etc/environment
+        exit 1
+fi
+
+touch $ENV
+if [ $? -ne 0 ]; then
+        echo exiting, unable to modify: $ENV
+        exit 1
+fi
+
+sed -i -e '/^COREOS_PUBLIC_IPV4=/d;/^COREOS_PRIVATE_IPV4=/d' $ENV
+
+COREOS_PUBLIC_IPV4=$(ip -4 -o addr show dev ens3 | awk '{ print $4; }' | cut -d / -f1)
+COREOS_PRIVATE_IPV4=$(ip -4 -o addr show dev ens4v1 | awk '{ print $4; }' | cut -d / -f1)
+echo COREOS_PUBLIC_IPV4=$COREOS_PUBLIC_IPV4 >> $ENV
+echo COREOS_PRIVATE_IPV4=$COREOS_PRIVATE_IPV4 >> $ENV

contrib/digitalocean/create-coreos-docker-store

@@ -0,0 +1,13 @@
+#!/bin/sh
+mkdir -p /media/doroot/var/lib/coreos
+
+if [ ! -f "/media/doroot/var/lib/coreos/docker.img" ]; then
+  DOFORMAT=1
+fi
+
+truncate -s 10G /media/doroot/var/lib/coreos/docker.img
+LODEV=`losetup -f --show /media/doroot/var/lib/coreos/docker.img`
+
+if [ -n "$DOFORMAT" ]; then
+  mkfs.btrfs -L docker "$LODEV"
+fi

contrib/digitalocean/provision-digitalocean-deis-image.sh

@@ -0,0 +1,104 @@
+#!/bin/bash
+
+set -e
+
+THIS_DIR=$(cd $(dirname $0); pwd) # absolute path
+CONTRIB_DIR=$(dirname $THIS_DIR)
+
+source $CONTRIB_DIR/utils.sh
+
+if ! which tugboat &>/dev/null; then
+	echo_red 'Digital Ocean command line client tugboat not found.'
+	exit 1
+fi
+
+function wait_ssh () {
+	echo -n "Trying to ssh into $1@$2..."
+	while ! $SSH -o ConnectTimeout=10 $1@$2 hostname &>/dev/null; do
+		echo -n "."
+		sleep 1
+	done
+	echo_green "done"
+}
+
+# parse parameters
+if [ -z "$1" ]; then
+	echo_red 'Usage: $0 SSH_ID [REGION_ID]'
+	echo
+	echo 'Use `tugboat keys` to list available SSH_IDs.'
+	exit 1
+fi
+
+SSH_ID="$1"
+REGION="${2:-5}" # Amsterdam 1 by default
+SIZE="66" # 512 MB
+NAME="deis-controller-image-$(date +%Y%m%d%H%M%S)"
+
+BASE_IMAGE='Ubuntu 14.04 x64'
+BASE_IMAGE_ID=$(tugboat images -g | grep "$BASE_IMAGE" | sed 's/.*id: \([0-9]*\).*/\1/')
+SSH_OPTIONS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=1 -o ConnectionAttempts=10"
+SSH="ssh $SSH_OPTIONS"
+SCP="scp $SSH_OPTIONS -q"
+
+# create droplet
+tugboat create "$NAME" -i $BASE_IMAGE_ID -p true -k $SSH_ID -s $SIZE -r $REGION
+
+# destroy droplets on error
+function cleanup () {
+	set +e
+	tugboat destroy -c -n "$NAME"
+	exit 1
+}
+trap cleanup ERR
+trap cleanup SIGINT
+
+# wait for droplet to come up with ssh login
+tugboat wait -n "$NAME" --state active
+IP=$(tugboat info -n "$NAME" | egrep '^IP' | awk '{ print $2; }')
+wait_ssh root $IP
+
+# bootstrap
+echo "Deploying CoreOS on top of $BASE_IMAGE..."
+(
+	set -ex
+	$SCP update-coreos root@$IP:/usr/sbin
+	$SSH root@$IP mkdir -p /usr/share/oem/bin
+	$SCP cloud-config.yml root@$IP:/usr/share/oem/cloud-config.yml.template
+	$SCP ../coreos/user-data root@$IP:/usr/share/oem/user-data.yml
+	$SCP create-coreos-docker-store coreos-setup-environment coreos-apply-user-data root@$IP:/usr/share/oem/bin
+	$SCP rc.local root@$IP:/etc
+	$SSH root@$IP <<EOF
+set -xe
+
+DEBIAN_FRONTEND=noninteractive
+apt-get install debconf-utils -y
+echo "kexec-tools kexec-tools/load_kexec boolean false" | debconf-set-selections
+apt-get install squashfs-tools kexec-tools -y
+shutdown -h now
+EOF
+) 2>&1 | sed 's/^/    /'; test ${PIPESTATUS[0]} -eq 0
+
+# switch off and make snapshot
+tugboat wait -n "$NAME" --state off
+tugboat snapshot $NAME -n $NAME
+
+# wait for snapshot to finish
+echo -n "Waiting for snapsnot to appear..."
+while ! tugboat images | grep $NAME &>/dev/null; do
+	sleep 5
+	echo -n "."
+done
+IMAGE_ID=$(tugboat images | grep $NAME | awk '{print $3;}' | sed 's/,//')
+echo_green "done"
+
+echo -n "Trying to destroy droplet..."
+while ! tugboat destroy -c -n $NAME &>/dev/null; do
+	sleep 5
+	echo -n "."
+done
+echo_green "done"
+
+echo
+echo_green "Congratulations: The Deis image $NAME was created with ID $IMAGE_ID."
+echo
+echo_yellow "Launch the cluster with: ./provision-do-cluster.sh $REGION $IMAGE_ID $SSH_ID 65"

contrib/digitalocean/provision-do-cluster.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+#
+# Usage: ./provision-do-cluster.sh <REGION_ID> <IMAGE_ID> <SSH_ID> <SIZE>
+#
+
+set -e
+
+THIS_DIR=$(cd $(dirname $0); pwd) # absolute path
+CONTRIB_DIR=$(dirname $THIS_DIR)
+
+source $CONTRIB_DIR/utils.sh
+
+# check for DO tools in $PATH
+if ! which tugboat > /dev/null; then
+  echo_red 'Please install the tugboat gem and ensure it is in your $PATH.'
+  exit 1
+fi
+
+if [ -z "$DEIS_NUM_INSTANCES" ]; then
+    DEIS_NUM_INSTANCES=3
+fi
+
+# check that the CoreOS user-data file is valid
+$CONTRIB_DIR/util/check-user-data.sh
+
+# launch the Deis cluster on Digital Ocean
+i=1 ; while [[ $i -le $DEIS_NUM_INSTANCES ]] ; do \
+    NAME=deis-$i
+    echo_yellow "Provisioning ${NAME}..."
+    tugboat create $NAME -r $1 -i $2 -p true -k $3 -s $4
+    ((i = i + 1)) ; \
+done
+
+echo_green "Your Deis cluster has successfully deployed to Digital Ocean."
+echo_green "Please continue to follow the instructions in the README."