A productive development environment with Docker on OS X

Docker and Boot2Docker are awesome for running containers on OS X, but if you try to use them to do iterative development by mounting a source folder from OS X into your Docker container, you will run into two major problems:

1. Mounted volumes on VirtualBox use vboxsf, which is extremely slow, so compilation and startup times for code in mounted folders is 10-20x slower.
2. File watching is broken since vboxsf does not trigger the inotify file watching mechanism. The only workaround is to enable polling, which is much slower to pick up changes and eats up a lot of resources.

I tried many different solutions (see Alternatives) that didn't work until I finally stumbled across one that does: Vagrant and Rsync. Using this combination, build and compilation performance in mounted folders is on par with native OS X and normal file watching works properly too. However, setting it up correctly is a painful process that involves nearly a dozen steps, so to make life easier, I've packaged this process up in this docker-osx-dev project.

For more info, check out the blog post A productive development environment with Docker on OS X.

Status

Alpha. I've tested it on my own computer and am able to code productively, but I welcome others to try it and give me feedback (or submit pull requests!).

Note: this project is inherently a temporary workaround. I hope that in the future, someone will build a better alternative to vboxsf for mounting source code from OS X, and makes this entire project obsolete. Until that day comes, I will continue to use these hacky scripts to keep myself productive.

Install

Prerequisite: HomeBrew must be installed.

To install docker-osx-dev and all of its dependencies, run:

curl https://raw.githubusercontent.com/brikis98/docker-osx-dev/master/setup.sh | bash

Two notes about the setup.sh script:

1. It is idempotent, so if you have some of the dependencies installed already, it will not overwrite them. If you already have boot2docker installed, see the troubleshooting section below.
2. When setup.sh completes, it prints out instructions for one source command you have to run to pick up important environment variables in your current shell, so make sure not to skip that step!

To setup docker-osx-dev for a new project, run:

docker-osx-dev init

This will create a Vagrantfile in the same folder. You should commit this file to source control. You only need to do this once per project.

Usage

Once you've setup a project with docker-osx-dev, use the following command to start Vagrant, Docker, and file syncing:

docker-osx-dev start

You can now run whatever Docker containers you like. For example, here is how you can fire up the tiny Alpine Linux image and get a Linux console in seconds:

> echo "I'm running in $(uname)"
I'm running in Darwin

> docker run -it --rm gliderlabs/alpine:3.1 sh
/ # echo "Now I'm running in $(uname)!"
Now I'm running in Linux!

You can use the -v flag to mount a source folder. For example, here is how you can mount the current directory on OS X so it shows up under /src in the Docker container:

> ls -al
total 16
drwxr-xr-x  4 brikis98  staff  136 May 16 14:05 .
drwxr-xr-x  7 brikis98  staff  238 May 16 14:04 ..
-rw-r--r--  1 brikis98  staff   12 May 16 14:05 bar
-rw-r--r--  1 brikis98  staff    4 May 16 14:05 foo

> docker run -it --rm -v $(pwd):/src gliderlabs/alpine:3.1 sh
/ # cd /src
/src # ls -al
total 12
drwxrwxrwx    2 1000     users           80 May 16 21:06 .
drwxr-xr-x   25 root     root          4096 May 16 21:07 ..
-rw-rw-rw-    1 1000     users           12 May 16 21:06 bar
-rw-rw-rw-    1 1000     users            4 May 16 21:06 foo

docker-osx-dev uses rsync to keep the files in sync between OS X and your Docker containers with virtually no performance penalty. In the example above, any build you run in the /src folder of the Docker container should work just as quickly as if you ran it in OS X. Also, file watchers should work normally for any development environment that supports hot reload (i.e. make a change and refresh the page)—no polling required!

If you are using Docker Compose, docker-osx-dev will automatically use rsync to mount any folders marked as volumes. For example, let's say you had the following docker-compose.yml file:

web:  
  image: training/webapp
  volumes:
    - .:/src
  ports:
    - "5000:5000"
db:
  image: postgres    

You could run this file as follows:

docker-compose up

This would fire up a Postgres database and the training webapp (a simple "Hello, World" Python app), mount the current directory into /src in the webapp container (using rsync, so it'll be fast), and expose port 5000. You can now test this webapp by going to:

http://dockerhost:5000

When you install docker-osx-dev, it adds an entry to your /etc/hosts file so that http://dockerhost works as a URL for testing your Docker containers.

Finally, to shut down Docker and Vagrant, you can run:

docker-osx-dev stop

Troubleshooting

Docker TLS connection errors

If you already had boot2docker installed, you may get the following error:

FATA[0000] Get http:///var/run/docker.sock/v1.18/containers/json: dial unix /var/run/docker.sock: no such file or directory. 
Are you trying to connect to a TLS-enabled daemon without TLS?

Part of the boot2docker install process is to add several environment variables (based on the output of the boot2docker shellinit command), including DOCKER_CERT_PATH and DOCKER_TLS_VERIFY. You do not want these set when using docker-osx-dev, so make sure to:

1. Remove them from ~/.bash_profile and ~/.bashrc
2. Run unset DOCKER_CERT_PATH DOCKER_TLS_VERIFY in your current shell.

How do I ssh into the boot2docker VM?

The docker-osx-dev project uses vagrant under the hood, so most vagrant commands will work. For example, to SSH to the VM, you run:

vagrant ssh

How it works

The setup.sh script installs all the software you need:

1. Docker
2. Docker Compose
3. VirtualBox
4. Vagrant
5. vagrant-gatling-rsync
6. The docker-osx-dev script which you can use to start/stop Docker and Vagrant

The setup.sh also:

1. Adds the $DOCKER_HOST environment variable to ~/.bash_profile or ~/.bashrc file so it is available at startup.
2. Adds an entry to /etc/hosts so that http://dockerhost works as a valid URL for your docker container for easy testing.

Instead of using vboxsf, docker-osx-dev keeps files in sync by running the vagrant-gatling-rsync in the background, which uses rsync to quickly copy changes from OS X to your Docker container. By default, the current source folder (i.e. the one with the Vagrantfile) is synced. If you use docker-compose, docker-osx-dev will sync any folders marked as volumes.

Limitations and known issues

1. File syncing is currently one way only. That is, changes you make on OS X will be visible very quickly in the Docker container. However, changes in the Docker container will not be propagated back to OS X. This isn't a problem for most development scenarios, but time permitting, I'll be looking into using Unison to support two-way sync.
2. Too may technologies. I'd prefer to not have to use Vagrant, but it makes using rsync very easy. Time permitting, I'll be looking into using rsync directly with Boot2Docker.

Alternatives

Below are some of the other solutions I tried to make Docker productive on OS X (I even created a StackOverflow Discussion to find out what other people were doing.) With most of them, file syncing was still too slow to be usable, but they were useful to me to learn more about the Docker ecosystem, and perhaps they will be useful for you if docker-osx-dev doesn't work out:

1. boot2docker-vagrant: Docker, Vagrant, and the ability to choose between NFS, Samba, rsync, and vboxsf for file syncing. A lot of the work in this project inspired docker-osx-dev.
2. dinghy: Docker + Vagrant + NFS. I found NFS was 2-3x slower than running builds locally, which was much faster than the 10-20x slowness of vboxsf, but still too slow to be usable.
3. docker-unison: Docker + Unison. The Unison File Synchronizer should be almost as fast as rsync, but I ran into strange connection errors when I tried to use it with Docker.
4. Polling in Jekyll and Polling in SBT/Play. Some of the file syncing solutions, such as vboxsf and NFS, don't work correctly with file watchers that rely on inotify, so these are a couple examples of how to switch from file watching to polling. Unfortunately, this eats up a fair amount of resources and responds to file changes slower, especially as the project gets larger.
5. Hodor. Uses the Unison File Synchronizer to sync files. I have not had a chance to try this project out yet.

License

This code is released under the MIT License. See LICENSE.txt.