DevOps HW 4: Docker Part Deux

This HW implementation includes advanced docker techniques like using docker-compose, docker deployment and file IO using socat. The HW specification can be found here. For all the three parts of this assignment, I have separate folders which contain individual code for that particular part.

Docker Compose (docker-compose)

I have three containers which are started by docker-compose:

1. redis: This container has a based Ubuntu image and simply installs redis and starts off the redis service at port 6379.
2. webapp: This holds the HW3 app with certain changes needed for spawning new containers instead of new servers. It depends on the redis container. The Dockerfile for this also has an ARG predicate which takes in an argument while building the docker image. This holds a default value of 3000, so the webapp starts off at this port. Later on when new containers will be spawned, they will be give new port numbers and that argument will be used to create an environment variable using the ENV predicate. The node application in webapp when started, will extract this environment variable using process.env.start_port and accordingly start the express server in that port. As I refer to all my containers using the VM's IP, I need different port numbers to distinguished between the newly spawned containers.
3. proxy: This depends on redis in the docker-compose.yml as well. It starts a simple http-proxy at port 3456 and on any request, does a rpoplpush on the servers list maintained in redis and delegates the request to that server. This servers list is updated on request and periodically by webapp and infrastructure.js respectively.

The infrastructure.js sits outside the containers as a simple node application, which has a setInterval method and periodically checks for new server URLs being added to a temporary server list in redis. This new server URL will be added by webapp when /spawn will be called by the user from a browser. webapp will basically get all the current servers in the servers list in redis, find the maximum of the port numbers and then allot a port number for the new server which is 1 more than the maximum and then push this value (as a complete formed URL) to the temporary servers list. infrastructure.js will get this server URL and run spawn_docker.sh giving the to-be-deployed container a new app name based on the port number and the port number itself. After spawning the new container, it removes the server URL from the temporary servers redis list and pushes it to the servers list. Thus the servers list has a collection of all the stable and newly created containers. The shell script spawn_docker.sh does the following things:

1. Stops the app referred by the provided app name.
2. Removes it.
3. Removes the corresponding image.
4. Builds a new image and passes the port number as a --build-args value.
5. Starts the container with the new app name based on the image just created.

Similarly, when /destroy is called, a random server is chosen other than the first server that was spawned at port 3000 and is put in a delete servers temporary redis list. The infrastructure.js file also polls this list and sees if there is anything to destroy. If so, it stops the app, removes it and removes the image associated with it. Accordingly, it also removes the value from the main servers redis list.

Docker Deployment (docker-deploy)

I have my app in the app subdirectory. I have also created a deploy subdirectory which holds the blue.git and green.git bare repositories and the blue-www and green-www deployed docker container contents. This folder is not present in the github repository as I don't want to commit irrelevant *.git files to another git repository. So I have committed the hooks in the post-receive_blue and post-receive_green files. The complete implementation is present in these post-receive hooks for both the git repositories. The app holds both the repositories as remote. A private docker registry has been started at port 5000 with:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

When you push to either of the repositories, the post-receive hook gets fired and it does the following things:

1. It sets the Git work directory as the respective www directory, depending upon where it has been pushed (green.git or blue.git).
2. It checks out the files to that specific work directory.
3. It stops the green-app or the blue-app container.
4. It removes the specific container.
5. It removes green-image or the blue-image.
6. It builds the image again.
7. It tags the image with the private registry and a specific name.
8. It pushes the current image to the registry.
9. It pulls the latest image from the registry.
10. It runs the node application in a container called green-app or blue-app at port 50100 or 50101.

File IO (fileio)

I have started this part using docker-compose as well. This has two docker containers - one for writing contents to a file and exposing it via socat and another container for doing curl from that port on a HTTP request and sending the contents back as response.

The writer container has a setInterval method which writes random stuff in a file every 2 minutes. Other than that, it has an exec node command which starts the socat listener. The socat command used is:

socat TCP-LISTEN:7000,fork,reuseaddr,crlf OPEN:random,rdonly

where random is the name of the file. 7000 is th port where it will open the TCP listener. Hence, in the Dockerfile, this container has to expose the port 7000.

The webapp container has a simple HTTP server which runs on port 8000 and listens for incoming requests. Whenever it gets a request, it does a curl on the IP write (this comes as a hostname as the write container has been linked from the docker-compose.yml file) and sends the response back to the browser.

Screencast

https://www.youtube.com/watch?v=xhUhF-lea18&edit=vd