Skip to content

Commit

Permalink
Add an example for running rethinkdb cluster on k8s
Browse files Browse the repository at this point in the history
  • Loading branch information
antmanler committed Feb 4, 2015
1 parent ffe6149 commit de25c0a
Show file tree
Hide file tree
Showing 9 changed files with 288 additions and 0 deletions.
Binary file added examples/rethinkdb/.DS_Store
Binary file not shown.
106 changes: 106 additions & 0 deletions examples/rethinkdb/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
RethinkDB Cluster on Kubernetes
==============================

Setting up a [rethinkdb](http://rethinkdb.com/) cluster on [kubernetes](http://kubernetes.io)

**Features**

* Auto configuration cluster by querying info from k8s
* Simple

Quick start
-----------

**Step 1**

antmanler/rethinkdb will discover peer using endpoints porvided by kubernetes_ro service,
so firstly create a service so the following pod can query its endpoint

```shell
kubectl create -f driver-service.yaml
```

check out:

```shell
$kubectl get se
NAME LABELS SELECTOR IP PORT
rethinkdb-driver db=influxdb db=rethinkdb 10.241.105.47 28015
```

**Step 2**

start fist server in cluster

```shell
kubectl create -f rc.yaml
```

Actually, you can start servers as many as you want at one time, just modify the `replicas` in `rc.ymal`

check out again:

```shell
$kubectl get po
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS
99f6d361-abd6-11e4-a1ea-001c426dbc28 10.240.2.68 rethinkdb rethinkdb:1.16.0 10.245.2.2/ db=rethinkdb,role=replicas Running
```

**Done!**


---

Scale
-----

You can scale up you cluster using `kubectl resize`, and new pod will join to exsits cluster automatically, for example


```shell
$kubectl resize rc rethinkdb-rc-1.16.0 --replicas=3
resized
$kubectl get po
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS
99f6d361-abd6-11e4-a1ea-001c426dbc28 10.240.2.68 rethinkdb rethinkdb:1.16.0 10.245.2.2/ db=rethinkdb,role=replicas Running
d10182b5-abd6-11e4-a1ea-001c426dbc28 10.240.26.14 rethinkdb rethinkdb:1.16.0 10.245.2.4/ db=rethinkdb,role=replicas Running
d101c1a4-abd6-11e4-a1ea-001c426dbc28 10.240.11.14 rethinkdb rethinkdb:1.16.0 10.245.2.3/ db=rethinkdb,role=replicas Running
```

Admin
-----

You need a separate pod (which labled as role:admin) to access Web Admin UI

```shell
kubectl create -f admin-pod.yaml
kubectl create -f admin-service.yaml
```

find the service

```shell
$kubectl get se
NAME LABELS SELECTOR IP PORT
rethinkdb-admin db=influxdb db=rethinkdb,role=admin 10.241.220.209 8080
rethinkdb-driver db=influxdb db=rethinkdb 10.241.105.47 28015
```

open a web browser and access to *http://10.241.220.209:8080* to manage you cluster

**Why not just using pods in replicas?**

This is because kube-proxy will act as a load balancer and send your traffic to different server,
since the ui is not stateless when playing with Web Admin UI will cause `Connection not open on server` error.


- - -

**BTW**

* All services and pods are placed under namespace `rethinkdb`.

* `gen_pod.sh` is using to generate pod templates for my local cluster,
the generated pods which is using `nodeSelector` to force k8s to schedule containers to my designate nodes, for I need to access persistent data on my host dirs.

* see [antmanler/rethinkdb-k8s](https://github.com/antmanler/rethinkdb-k8s) for detail
30 changes: 30 additions & 0 deletions examples/rethinkdb/admin-pod.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
apiVersion: v1beta1
namespace: rethinkdb
kind: Pod
id: rethinkdb-admin-1.16.0
labels:
db: rethinkdb
role: admin
desiredState:
manifest:
version: v1beta1
id: rethinkdb
containers:
- name: rethinkdb
image: antmanler/rethinkdb:1.16.0
ports:
- name: admin-port
containerPort: 8080
- name: driver-port
containerPort: 28015
- name: cluster-port
containerPort: 29015
volumeMounts:
- name: rethinkdb-storage
mountPath: /data/rethinkdb_data
volumes:
- name: rethinkdb-storage
source:
emptyDir: {}
restartPolicy:
always: {}
11 changes: 11 additions & 0 deletions examples/rethinkdb/admin-service.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
apiVersion: v1beta1
namespace: rethinkdb
kind: Service
id: rethinkdb-admin
containerPort: admin-port
port: 8080
labels:
db: influxdb
selector:
db: rethinkdb
role: admin
10 changes: 10 additions & 0 deletions examples/rethinkdb/driver-service.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
apiVersion: v1beta1
namespace: rethinkdb
kind: Service
id: rethinkdb-driver
containerPort: driver-port
port: 28015
labels:
db: influxdb
selector:
db: rethinkdb
58 changes: 58 additions & 0 deletions examples/rethinkdb/gen-pod.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
#!/bin/bash

set -o errexit
set -o nounset
set -o pipefail

: ${VERSION:=1.16.0}

readonly NAME=${1-}
if [[ -z "${NAME}" ]]; then
echo -e "\033[1;31mName must be specified\033[0m"
exit 1
fi

ADMIN=""
if [[ ${NAME} == "admin" ]]; then
ADMIN="role: admin"
fi

NODE=""
if [[ ! -z "${2-}" ]]; then
NODE="nodeSelector: { name: ${2} }"
fi

cat << EOF
apiVersion: v1beta1
namespace: rethinkdb
kind: Pod
id: rethinkdb-${NAME}-${VERSION}
${NODE}
labels:
db: rethinkdb
${ADMIN}
desiredState:
manifest:
version: v1beta1
id: rethinkdb
containers:
- name: rethinkdb
image: antmanler/rethinkdb:${VERSION}
ports:
- name: admin-port
containerPort: 8080
- name: driver-port
containerPort: 28015
- name: cluster-port
containerPort: 29015
volumeMounts:
- name: rethinkdb-storage
mountPath: /data/rethinkdb_data
volumes:
- name: rethinkdb-storage
source:
hostDir:
path: /data/db/rethinkdb
restartPolicy:
always: {}
EOF
14 changes: 14 additions & 0 deletions examples/rethinkdb/image/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
FROM rethinkdb:1.16.0

MAINTAINER BinZhao <wo@zhaob.in>

RUN apt-get update && \
apt-get install -yq curl && \
rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/* && \
curl -L http://stedolan.github.io/jq/download/linux64/jq > /usr/bin/jq && \
chmod u+x /usr/bin/jq

COPY ./run.sh /usr/bin/run.sh
RUN chmod u+x /usr/bin/run.sh

CMD ["/usr/bin/run.sh"]
23 changes: 23 additions & 0 deletions examples/rethinkdb/image/run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/bin/bash

set -o pipefail

IP=""
if [[ -n "${KUBERNETES_RO_SERVICE_HOST}" ]]; then

: ${NAMESPACE:=rethinkdb}
# try to pick up first different ip from endpoints
MYHOST=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1 -d'/')
URL="${KUBERNETES_RO_SERVICE_HOST}/api/v1beta1/endpoints/rethinkdb-driver?namespace=${NAMESPACE}"
IP=$(curl -s ${URL} | jq -s -r --arg h "${MYHOST}" '.[0].endpoints | [ .[]? | split(":") ] | map(select(.[0] != $h)) | .[0][0]') || exit 1
[[ "${IP}" == null ]] && IP=""
fi

if [[ -n "${IP}" ]]; then
ENDPOINT="${IP}:29015"
echo "Join to ${ENDPOINT}"
exec rethinkdb --bind all --join ${ENDPOINT}
else
echo "Start single instance"
exec rethinkdb --bind all
fi
36 changes: 36 additions & 0 deletions examples/rethinkdb/rc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
apiVersion: v1beta1
namespace: rethinkdb
kind: ReplicationController
id: rethinkdb-rc-1.16.0
labels:
db: rethinkdb
desiredState:
replicas: 1
replicaSelector:
db: rethinkdb
role: replicas
podTemplate:
labels:
db: rethinkdb
role: replicas
desiredState:
manifest:
version: v1beta1
id: rethinkdb
containers:
- name: rethinkdb
image: antmanler/rethinkdb:1.16.0
ports:
- name: admin-port
containerPort: 8080
- name: driver-port
containerPort: 28015
- name: cluster-port
containerPort: 29015
volumeMounts:
- name: rethinkdb-storage
mountPath: /data/rethinkdb_data
volumes:
- name: rethinkdb-storage
source:
emptyDir: {}

0 comments on commit de25c0a

Please sign in to comment.