Simple and Scalable Microservices: NATS and the Docker tooling

Provisioning

Goal

5 VMs in Google Compute Engine

gcloud compute instances list

NAME            ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP      STATUS
docker-swarm-0  us-central1-a  n1-standard-1               10.240.0.10  AAA.BBB.CCC.DDD  RUNNING
docker-swarm-1  us-central1-a  n1-standard-1               10.240.0.11  AAA.BBB.CCC.DDD  RUNNING
docker-swarm-2  us-central1-a  n1-standard-1               10.240.0.12  AAA.BBB.CCC.DDD  RUNNING
docker-swarm-3  us-central1-a  n1-standard-1               10.240.0.13  AAA.BBB.CCC.DDD  RUNNING
docker-swarm-4  us-central1-a  n1-standard-1               10.240.0.14  AAA.BBB.CCC.DDD  RUNNING

Set region to use

gcloud config set compute/region us-central1
gcloud config set compute/zone   us-central1-a

Create a custom network

gcloud compute networks create nats-docker --mode custom

Create subnet

gcloud compute networks subnets create nats-docker \
  --network nats-docker \
  --range 10.240.0.0/24

Firewalls

Allow ping from anywhere

gcloud compute firewall-rules create nats-docker-allow-icmp \
  --allow icmp \
  --network nats-docker \
  --source-ranges 0.0.0.0/0

Allow internal traffic

gcloud compute firewall-rules create nats-docker-allow-internal \
  --allow tcp:0-65535,udp:0-65535,icmp \
  --network nats-docker \
  --source-ranges 10.240.0.0/24

Allow ssh from anywhere

gcloud compute firewall-rules create nats-docker-allow-ssh \
  --allow tcp:22 \
  --network nats-docker \
  --source-ranges 0.0.0.0/0

Allow public ip health check monitoring

gcloud compute firewall-rules create nats-docker-allow-healthz \
  --allow tcp:8080 \
  --network nats-docker \
  --source-ranges 130.211.0.0/22

Allow API server

gcloud compute firewall-rules create nats-docker-allow-api-server \
  --allow tcp:80 \
  --network nats-docker \
  --source-ranges 0.0.0.0/0

NATS Docker network

gcloud compute firewall-rules list --filter "network=nats-docker"

Public IP Address

gcloud compute addresses create nats-docker --region us-central1

Confirm

gcloud compute addresses list nats-docker

Infrastructure

NAME	ROLE(S)	IP
docker-swarm-0	leader	10.240.0.10
docker-swarm-1	follower	10.240.0.11
docker-swarm-2	follower	10.240.0.12
docker-swarm-3	follower	10.240.0.13
docker-swarm-4	follower	10.240.0.14

Creating the nodes

for i in `seq 0 4`; do
  gcloud compute instances create docker-swarm-$i \
   --boot-disk-size 200GB \
   --can-ip-forward \
   --image ubuntu-1604-xenial-v20160627 \
   --image-project ubuntu-os-cloud \
   --machine-type n1-standard-1 \
   --private-network-ip 10.240.0.1$i \
   --zone us-central1-a \
   --subnet nats-docker
done

Confirm

gcloud compute instances list

Installing Docker in all the nodes

Emacs

(setq cluster-domain ".us-central1-a.YOUR_PROJECT_NAME_HERE")

(defun docker:infra-ssh(node &optional path)
  (or path (setq path ""))
  (message (concat "/ssh:" node cluster-domain ":" path)))

Node 0

Installing Docker

curl -O -L https://get.docker.com/builds/Linux/x86_64/docker-1.12.1.tgz

tar -xvf docker-1.12.1.tgz

sudo cp docker/docker* /usr/bin/

Setting up Docker systemd service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io

[Service]
ExecStart=/usr/bin/docker daemon \
  --iptables=false \
  --ip-masq=false \
  --host=unix:///var/run/docker.sock \
  --host=tcp://127.0.0.1:2375 \
  --log-level=error \
  --storage-driver=overlay
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo cp /tmp/docker.service /etc/systemd/system/docker.service

sudo systemctl daemon-reload
sudo systemctl enable docker
sudo systemctl start docker

Confirm

sudo docker ps

Node 1

Installing Docker

curl -O -L https://get.docker.com/builds/Linux/x86_64/docker-1.12.1.tgz

tar -xvf docker-1.12.1.tgz

sudo cp docker/docker* /usr/bin/

Setting up Docker systemd service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io

[Service]
ExecStart=/usr/bin/docker daemon \
  --iptables=false \
  --ip-masq=false \
  --host=unix:///var/run/docker.sock \
  --host=tcp://127.0.0.1:2375 \
  --log-level=error \
  --storage-driver=overlay
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo cp /tmp/docker.service /etc/systemd/system/docker.service

sudo systemctl daemon-reload
sudo systemctl enable docker
sudo systemctl start docker

Confirm

sudo docker ps

Node 2

Installing Docker

curl -O -L https://get.docker.com/builds/Linux/x86_64/docker-1.12.1.tgz

tar -xvf docker-1.12.1.tgz

sudo cp docker/docker* /usr/bin/

Setting up Docker systemd service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io

[Service]
ExecStart=/usr/bin/docker daemon \
  --iptables=false \
  --ip-masq=false \
  --host=unix:///var/run/docker.sock \
  --host=tcp://127.0.0.1:2375 \
  --log-level=error \
  --storage-driver=overlay
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo cp /tmp/docker.service /etc/systemd/system/docker.service

sudo systemctl daemon-reload
sudo systemctl enable docker
sudo systemctl start docker

Confirm

sudo docker ps

Node 3

Installing Docker

curl -O -L https://get.docker.com/builds/Linux/x86_64/docker-1.12.1.tgz

tar -xvf docker-1.12.1.tgz

sudo cp docker/docker* /usr/bin/

Setting up Docker systemd service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io

[Service]
ExecStart=/usr/bin/docker daemon \
  --iptables=false \
  --ip-masq=false \
  --host=unix:///var/run/docker.sock \
  --host=tcp://127.0.0.1:2375 \
  --log-level=error \
  --storage-driver=overlay
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo cp /tmp/docker.service /etc/systemd/system/docker.service

sudo systemctl daemon-reload
sudo systemctl enable docker
sudo systemctl start docker

Confirm

sudo docker ps

Node 4

Installing Docker

curl -O -L https://get.docker.com/builds/Linux/x86_64/docker-1.12.1.tgz

tar -xvf docker-1.12.1.tgz

sudo cp docker/docker* /usr/bin/

Setting up Docker systemd service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io

[Service]
ExecStart=/usr/bin/docker daemon \
  --iptables=false \
  --ip-masq=false \
  --host=unix:///var/run/docker.sock \
  --host=tcp://127.0.0.1:2375 \
  --log-level=error \
  --storage-driver=overlay
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo cp /tmp/docker.service /etc/systemd/system/docker.service

sudo systemctl daemon-reload
sudo systemctl enable docker
sudo systemctl start docker

When updating we need to restart:

sudo systemctl daemon-reload
sudo systemctl restart docker

Confirm

sudo docker ps

Docker Swarm mode setup

Node 0

sudo docker swarm init --advertise-addr 10.240.0.10

Swarm initialized: current node (93cb4tot9jvcvn3o2u60fntlz) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-3dp2fr5m5ycz9daobryhi4kzv82ol3skahy7oqdyczgmjbj0ru-7ltk8rkyei28lz7d07l0fajjr \
    10.240.0.10:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Node 1

sudo docker swarm join \
--token SWMTKN-1-3dp2fr5m5ycz9daobryhi4kzv82ol3skahy7oqdyczgmjbj0ru-7ltk8rkyei28lz7d07l0fajjr \
10.240.0.10:2377

Node 2

sudo docker swarm join \
--token SWMTKN-1-3dp2fr5m5ycz9daobryhi4kzv82ol3skahy7oqdyczgmjbj0ru-7ltk8rkyei28lz7d07l0fajjr \
10.240.0.10:2377

Node 3

sudo docker swarm join \
--token SWMTKN-1-3dp2fr5m5ycz9daobryhi4kzv82ol3skahy7oqdyczgmjbj0ru-7ltk8rkyei28lz7d07l0fajjr \
10.240.0.10:2377

Node 4

sudo docker swarm join \
--token SWMTKN-1-3dp2fr5m5ycz9daobryhi4kzv82ol3skahy7oqdyczgmjbj0ru-7ltk8rkyei28lz7d07l0fajjr \
10.240.0.10:2377

Confirm Docker Swarm cluster

sudo docker node ls

ID                           HOSTNAME        STATUS  AVAILABILITY  MANAGER STATUS
16bvu8ecwg38gwxe15gqnic2c    docker-swarm-2  Ready   Active        
2p73mixe2doka3wj55away1nh    docker-swarm-4  Ready   Active        
93cb4tot9jvcvn3o2u60fntlz *  docker-swarm-0  Ready   Active        Leader
dobqk55qesnbdzdfod12pl07e    docker-swarm-1  Ready   Active        
ef6lbguddbexk61x5ntm73th5    docker-swarm-3  Ready   Active

Prepare nats-cluster-network and smoke test

Create the nats-cluster network and service

sudo docker network create --driver overlay nats-cluster-example

sudo docker service create --network nats-cluster-example \
                             --name nats-cluster-node-1 nats:0.9.4 -DV -cluster nats://0.0.0.0:6222

# Update
# sudo docker service update --args "-DV -cluster nats://0.0.0.0:6222" nats-cluster-node-1

Confirm

sudo docker ps

Create sample ruby-nats service

sudo docker service create --name ruby-nats --network nats-cluster-example wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0 -e '
  NATS.on_error do |e|
    puts "ERROR: #{e}"
  end
  NATS.start(:servers => ["nats://nats-cluster-node-1:4222"]) do |nc|
    inbox = NATS.create_inbox
    puts "[#{Time.now}] Connected to NATS at #{nc.connected_server}, inbox: #{inbox}"

    nc.subscribe(inbox) do |msg, reply|
      puts "[#{Time.now}] Received reply - #{msg}"
    end

    nc.subscribe("hello") do |msg, reply|
      next if reply == inbox
      puts "[#{Time.now}] Received greeting - #{msg} - #{reply}"
      nc.publish(reply, "world")
    end

    EM.add_periodic_timer(1) do
      puts "[#{Time.now}] Saying hi (servers in pool: #{nc.server_pool}"
      nc.publish("hello", "hi", inbox)
    end
  end
'

Inspect running services

sudo docker service ls

Increase number of ruby-nats services

sudo docker service scale ruby-nats=5

sudo docker service ls

echo "--- NODES"
for i in `seq 0 4`; do
  echo
  echo "● docker-swarm-$i"
  sudo docker node ps docker-swarm-$i
done
echo
echo "--- SERVICES"
sudo docker service ls

--- NODES

● docker-swarm-0
ID                         NAME                   IMAGE                    NODE            DESIRED STATE  CURRENT STATE              ERROR
5oq8kh059me3yra5gnc86mwws  nats-ops.1             wallyqs/nats-ops:latest  docker-swarm-0  Running        Running 10 minutes ago     
90ugss9jbm9jdfqplc6p07bmj  nats-cluster-node-1.1  nats:0.9.4               docker-swarm-0  Running        Running about an hour ago  

● docker-swarm-1
ID                         NAME         IMAGE                                     NODE            DESIRED STATE  CURRENT STATE           ERROR
9u4r6yp85spj85jvt7s8x6qmd  ruby-nats.2  wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0  docker-swarm-1  Running        Running 21 minutes ago  

● docker-swarm-2
ID                         NAME         IMAGE                                     NODE            DESIRED STATE  CURRENT STATE              ERROR
f3a08hc0hz9y61zolz6r3mibk  ruby-nats.1  wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0  docker-swarm-2  Running        Running about an hour ago  

● docker-swarm-3
ID                         NAME         IMAGE                                     NODE            DESIRED STATE  CURRENT STATE           ERROR
erkiytraly8atxqy03vvxqy0z  ruby-nats.4  wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0  docker-swarm-3  Running        Running 21 minutes ago  
13fmolsohdvh39juysb580pk8  ruby-nats.5  wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0  docker-swarm-3  Running        Running 21 minutes ago  

● docker-swarm-4
ID                         NAME         IMAGE                                     NODE            DESIRED STATE  CURRENT STATE           ERROR
02037tsgeyst9qqclbnn1urld  ruby-nats.3  wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0  docker-swarm-4  Running        Running 21 minutes ago  

--- SERVICES
ID            NAME                 REPLICAS  IMAGE                                     COMMAND
1qu447mo0fqe  nats-cluster-node-1  1/1       nats:0.9.4                                -DV
35g5krwa8q7v  nats-ops             1/1       wallyqs/nats-ops:latest                   tail -f /dev/null
5m2u2jp20cnl  ruby-nats            5/5       wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0  -e 
  NATS.on_error do |e|
    puts "ERROR: #{e}"
  end
  NATS.start(:servers => ["nats://nats-cluster-node-1:4222"]) do |nc|
    inbox = NATS.create_inbox
    puts "[#{Time.now}] Connected to NATS at #{nc.connected_server}, inbox: #{inbox}"

    nc.subscribe(inbox) do |msg, reply|
      puts "[#{Time.now}] Received reply - #{msg}"
    end

    nc.subscribe("hello") do |msg, reply|
      next if reply == inbox
      puts "[#{Time.now}] Received greeting - #{msg} - #{reply}"
      nc.publish(reply, "world")
    end

    EM.add_periodic_timer(1) do
      puts "[#{Time.now}] Saying hi (servers in pool: #{nc.server_pool}"
      nc.publish("hello", "hi", inbox)
    end
  end

Create nats-ops container and NATS nodes

Ops server for monitoring within the nats-cluster-example network

# sudo docker run --network nats-cluster-example wallyqs/nats-ops:latest /bin/bash
sudo docker service create --network nats-cluster-example \
                             --name nats-ops wallyqs/nats-ops:latest tail -f /dev/null

Add a new NATS cluster node which will be discovered dynamically

sudo docker service create --network nats-cluster-example \
               --name nats-cluster-node-2 nats:0.9.4 -DV -cluster nats://0.0.0.0:6222 -routes nats://nats-cluster-node-1:6222

Add another NATS cluster node

sudo docker service create --network nats-cluster-example \
               --name nats-cluster-node-3 nats:0.9.4 -DV -cluster nats://0.0.0.0:6222 -routes nats://nats-cluster-node-1:6222,nats://nats-cluster-node-2:6222

Scaling down the original container to rebalance

To test out manual operations:

sudo docker service scale nats-cluster-node-1=0

Run API Server and Workers onto Docker Swarm

Create seed NATS cluster

sudo docker service create --network nats-cluster-example \
              --name nats-cluster-node-1 nats:0.9.4 -DV -cluster nats://0.0.0.0:6222 -routes nats://nats-cluster-node-1:6222

Increase number of NATS nodes

Subsequent NATS cluster nodes could only connect to initial node

sudo docker service create --network nats-cluster-example \
              --name nats-cluster-node-2 nats:0.9.4 -DV -cluster nats://0.0.0.0:6222 -routes nats://nats-cluster-node-1:6222

sudo docker service create --network nats-cluster-example \
              --name nats-cluster-node-3 nats:0.9.4 -DV -cluster nats://0.0.0.0:6222 -routes nats://nats-cluster-node-1:6222

Run `wallyqs/nats-ops` image to inspect current status

sudo docker service create --network nats-cluster-example \
             --name nats-ops wallyqs/nats-ops:latest tail -f /dev/null

Confirm

sudo docker node ls
sudo docker service ls

Start API Server

sudo docker service create --network nats-cluster-example \
             --name api-server -e NATS_URI="nats://nats-cluster-node-1:4222" wallyqs/sample-nats-api-server:latest

Start single worker

sudo docker service create --network nats-cluster-example \
             --name worker -e NATS_URI="nats://nats-cluster-node-1:4222" wallyqs/sample-nats-worker:latest

Within nats-ops container

Make a request through NATS

while true; do 
  nats-req -s nats://nats-cluster-node-1:4222 tasks help
  sleep 0.1
done

Make a request through the API server

while true; do
  curl -X POST http://api-server:8080/createTask; 
  sleep 1; 
done

Make a request to the distribution queue

while true; do
  curl -X POST http://api-server:8080/createWorkerTask; 
  sleep 1; 
done

Files

infra-setup.org

Latest commit

History

infra-setup.org

File metadata and controls

Simple and Scalable Microservices: NATS and the Docker tooling

Provisioning

Goal

Set region to use

Create a custom network

Create subnet

Firewalls

Public IP Address

Infrastructure

Creating the nodes

Installing Docker in all the nodes

Emacs

Node 0

Installing Docker

Setting up Docker systemd service

Confirm

Node 1

Installing Docker

Setting up Docker systemd service

Confirm

Node 2

Installing Docker

Setting up Docker systemd service

Confirm

Node 3

Installing Docker

Setting up Docker systemd service

Confirm

Node 4

Installing Docker

Setting up Docker systemd service

Confirm

Docker Swarm mode setup

Node 0

Node 1

Node 2

Node 3

Node 4

Confirm Docker Swarm cluster

Prepare nats-cluster-network and smoke test

Create the nats-cluster network and service

Confirm

Create sample ruby-nats service

Inspect running services

Increase number of ruby-nats services

Create nats-ops container and NATS nodes

Scaling down the original container to rebalance

Run API Server and Workers onto Docker Swarm

Create seed NATS cluster

Increase number of NATS nodes

Run wallyqs/nats-ops image to inspect current status

Confirm

Start API Server

Start single worker

Within nats-ops container

Run `wallyqs/nats-ops` image to inspect current status