[DEPRECATED] About

I can finally deprecate this as Docker finally released Docker Desktop for Linux. Also, Podman became more useful as well, making Docker a bit redundant.

Docker host as Vagrant box created with kitchen-docker in mind. Provides a Docker host and Squid as caching proxy. It doesn't have anything Test Kitchen specific per-se, but it provides a consistent development environment. This document also contains a list of speed hacks to make Test Kitchen a lot more faster compared to its defaults.

The machine is also implemented using Test Kitchen using the kitchen-vagrant driver. It used to be implemented with Vagrant and vagrant-berkshelf, however, vagrant-berkshelf has the bad habit of breaking quite often.

All the required gems are supplied by CINC Workstation (i.e a free distribution of Chef Workstation without the corporate nonsense).

Dependencies

• CINC Workstation 21.3.346
• VirtualBox 6.1+
• Vagrant 2.2.15+
• Docker 20.10+

How to use

For starting the VM, simply issue a rake up command in the root directory of this project.

By default, it uses 2 virtual cores and 4096 GB of RAM and zram support. Export VB_CPUS and / or VB_MEM environment variables with the desired values to customize.

To customised the sync folders, create a volumes.yml file containing an array of paths. For example:

# volumes.yml
- /foo/bar
- /baz/qux

The paths on the host are mounted as sync folders in the VM on a 1:1 mapping (i.e the same paths are kept inside the VM). This allows using the Docker volumes feature as if the service is running locally, for as long as the path is declared in volumes.yml.

The VM itself uses a host-only network adapter with the IP address 192.168.99.100. This makes it sort of a drop-in replacement for docker-machine. The Docker socket isn't TLS enabled though.

To tell the Docker client where to find the host, simply:

export DOCKER_HOST=tcp://192.168.99.100:2375

To check whether the Docker connection is OK:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.5
 Storage Driver: devicemapper
  Pool Name: docker-252:0-134938214-pool
  Pool Blocksize: 65.54kB
  Base Device Size: 21.47GB
  Backing Filesystem: xfs
  Udev Sync Supported: true
  Data file: /dev/loop0
  Metadata file: /dev/loop1
  Data loop file: /var/lib/docker/devicemapper/devicemapper/data
  Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
  Data Space Used: 11.73MB
  Data Space Total: 107.4GB
  Data Space Available: 71.63GB
  Metadata Space Used: 17.36MB
  Metadata Space Total: 2.147GB
  Metadata Space Available: 2.13GB
  Thin Pool Minimum Free Space: 10.74GB
  Deferred Removal Enabled: true
  Deferred Deletion Enabled: true
  Deferred Deleted Device Count: 0
  Library Version: 1.02.171-RHEL8 (2020-05-28)
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.17-2036.104.5.el8uek.x86_64
 Operating System: Oracle Linux Server 8.3
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.561GiB
 Name: kdh-generic-oracle8.vagrantup.com
 ID: NZHB:L2GI:ZOFQ:L7A4:G5BF:X4PJ:ADTJ:VHRW:SD4D:US6T:UVXK:LDY6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

To use it with Test Kitchen, you need to install the kitchen-docker gem and to specify docker as Kitchen driver.

To use the Squid caching proxy you need to tell Test Kitchen to use http_proxy.

driver:
  name: docker
  http_proxy: http://192.168.99.100:3128

provisioner:
  chef_omnibus_url: http://www.opscode.com/chef/install.sh
  client_rb:
    http_proxy: http://192.168.99.100:3128

The only thing that doesn't seem to belong here is chef_omnibus_url. However, the Omnibus installer defaults to HTTPS, unless the URL to install.sh uses HTTP. This allows Squid to cache the Chef package which is quite large at ~40 MB.

The whole configuration may be bit smarter as in this example.

Speed up the Kitchen file transfer

By default, Test Kitchen with the kitchen-docker driver, uses SCP as transport backend. SCP is painfully slow for most of the tasks regarding file transfer to a Kitchen container. This cancels any speed gains from using containers instead of virtual machines provisioned by Vagrant. The rsync transport saves the day.

gem install kitchen-transport-rsync # or add it to your Gemfile

You need to install rsync inside the Kitchen container. These .kitchen.yml bits show you how to do it for Red Hat and Debian based distributions without a custom Dockerfile:

driver:
  name: docker
  provision_command:
  - if [ -x /usr/bin/yum ]; then yum -y install rsync; fi; if [ -x /usr/bin/apt-get ]; then apt-get -y install rsync; fi

transport:
  name: rsync

While provision_command accepts an array, doing a one liner gets the job done in one stage instead of two, which makes the Kitchen provisioning to be bit faster. Also, the commands themselves need to be wrapped in if statements as a non-zero exit stops the provisioning (equivalent to a script running with set -e).

kitchen-transport-rsync works with Test Kitchen 1.4.2, probably 1.4.x.

Better usage of Docker caching

Docker itself saves each stage (layer) which is built from the Dockerfile which is generated by kitchen-docker. Unfortunately, there's a major cache buster which is the SSH key pair. A simple solution is to use public_key and private_key configuration options for the docker driver. By using static keys, the generated Dockerfile has identical layers which roughly translates in a provisioning that takes little over one second instead of more than twenty seconds. The Docker image commits are fairly cheap, but they don't come for free.

Example:

driver:
  name: docker
  public_key: ../kitchen_id_rsa.pub
  private_key: ../kitchen_id_rsa

If the provisioning fails, it means that you're using a kitchen-docker version that isn't patched to strip the whitespace from the public_key. You'll need to remove the newline at the end of the key.

Caching the Chef Omnibus installation also brings performance improvements as it removes one more redundant step.

Example:

driver:
  name: docker
  provision_command:
    - curl -L http://www.opscode.com/chef/install.sh -o /tmp/install.sh && bash /tmp/install.sh -v 13.2.20

provisioner:
  name: chef_zero
  require_chef_omnibus: true # just checks the presence of a Chef Omnibus installation instead of passing a Chef version

Thick containers for Docker

Even though this isn't the usual use case, Docker is perfectly capable of running traditional containers (i.e. OpenVZ like). These thick containers behave more like a virtual machine, but they are very quick to provision unlike actual VM's.

Some of the goals for these Dockerfile templates:

• Have an actual init system as PID 1.
• The init should actually start services inside the container. Sometimes, this may be rather difficult with upstart / systemd.
• Have a working SSH service.
• The containers should respond to shutdown commands in a consistent way.
• Have all the basic bits baked into the images (rsync, Chef Omnibus).

kitchen-docker supports custom Dockerfiles via the dockerfile driver configuration option.

This is the list of supported distributions with these Dockerfiles:

• CentOS 6.8
• CentOS 7.2 (may be used for targeting Amazon Linux)
• Ubuntu 15.10
• Debian 8.2

For systemd to work, it requires at least CAP_SYS_ADMIN. For the shutdown support to work, I had to run the containers in privileged mode. There's too much work to figure out an exact list of capabilities and there's no guarantee as privileged provides more privileges than enabling all the supported capabilities.

The Dockerfiles are ERB templates which are rendered by kitchen-docker. There's a couple of variables:

• public_key - kitchen-docker already has this defined, whether you're using the generated keys or you're using static keys
• chef_version

chef_version by itself isn't defined in kitchen-docker, but the ERB context includes all the variables passed to the driver config, therefore you have a lot of flexibility.

Example:

driver:
  name: docker
  chef_version: 13.2.20

platforms:
- name: centos-6.8
  driver_config:
    dockerfile: "../centos-6.8"

For a development machine, I use Docker in a VM even for a host that supports it natively, therefore the SSH inside the container is a hard dependency. The reason for this statement is the fact that the volumes feature essentially provide root access to the host for all the users who have access to the Docker socket.

Monkey-patching the docker driver

This article explains the basics of speeding up kitchen-docker. Even though patching the driver isn't necessary, docker exec is much faster than SSH, and the containers are removed in a clean way. I think Docker got better regarding the resource leaks, but I wouldn't put that to the test.

require 'kitchen/driver/docker'

module Kitchen
  module Driver
    class Docker < Kitchen::Driver::SSHBase
      # monkey-patch kitchen login to use docker exec instead of ssh
      def login_command(state)
        LoginCommand.new 'docker', ['exec', '-it', state[:container_id], 'su', '-', 'kitchen']
      end

      # monkey-patch kitchen destroy
      def rm_container(state)
        cont_id = state[:container_id]
        docker_command "exec #{cont_id} poweroff"
        docker_command "wait #{cont_id}"
        docker_command "rm #{cont_id}"
      end
    end
  end
end

It can be easily loaded with something like:

# <% load "#{File.dirname(__FILE__)}/../kitchen_docker.rb" %>
---
driver:
  name: docker

How to turn your Test Kitchen into a fast-food joint

Having complex cookbooks with multiple code paths to test means you have to declare multiple test suites. Same applies if you target multiple platforms where you're looking for consistency. The issue is that by default Test Kitchen runs everything sequentially, therefore it means you don't get any benefit from a multi-core CPU.

Test Kitchen also supports a concurrent mode, but Unfortunately this isn't documented in a very visible way. Going at ludicrous speed is easy as the only thing you need to do is to pass the "-c" flag.

Example:

kitchen create -c
kitchen converge -c
kitchen verify -c
kitchen destroy -c

By default, the concurrency limit is at 9999 which is a reasonable value given the fact that it's unlikely to have so many cores. The concurrency flag accepts a numeric value to indicate the number of threads to run if the number of instances is too large for your CPU to handle.

The only drawback of the concurrent mode is the fact that the console output becomes virtually unreadable. However, the logs from .kitchen/logs are really valuable in this case and you may run a single suite at any point.

Example:

# run only the 'default' suite
kitchen converge default
kitchen verify default

Caching commonly used chef_gem resources

If you have a commonly used gem installed by the chef_gem resource, it pays off to leverage the Dockerfiles to have that gem preinstalled in a Docker image. For example, interfacing with AWS may require aws-sdk-core to be available in your Chef cookbooks.

Adding something like this RUN command to your Dockerfile's speeds up the kitchen converge:

RUN /opt/chef/embedded/bin/gem install --no-user-install --install-dir \
  /opt/chef/embedded/lib/ruby/gems/2.1.0 aws-sdk-core

If you need a specific gem version, it may be specified like aws-sdk-core:2.2.34.

Another advantage of baking Ruby gems into Docker images is the fact that it removes the need to download stuff from rubygems.org which is useful when the service is experiencing hiccups.

Preinstalling busser, busser-serverspec, and serverspec

Using Test Kitchen verifier with a busser is a repetitive and time wasting activity. It also depends on rubygems.org. In this example I'm using the serverspec busser, but it should be applicable for the rest as well.

Drop another layer using a RUN command like this:

# setup busser/serverspec to speed up kitchen verify
RUN su - kitchen -c 'BUSSER_ROOT="/tmp/verifier"; export BUSSER_ROOT; \
  GEM_HOME="/tmp/verifier/gems"; export GEM_HOME; \
  GEM_PATH="/tmp/verifier/gems"; export GEM_PATH; \
  GEM_CACHE="/tmp/verifier/gems/cache"; export GEM_CACHE; \
  /opt/chef/embedded/bin/gem install --no-rdoc --no-ri \
  --no-format-executable -n /tmp/verifier/bin --no-user-install \
  busser busser-serverspec serverspec'

This should setup all the required Ruby gems to have them ready for a kitchen verify.