A container debugging tool based on FUSE
Clone or download
Latest commit 8de9a36 Aug 16, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
examples fix cntrfs example Jun 15, 2018
scripts add pre-commit script Dec 19, 2017
src alias --type parameter to -t Jun 22, 2018
.gitignore first commit Jun 1, 2016
.travis.yml travis: also upload binary uncompressed May 13, 2018
Cargo.lock bump version to 1.2.0 Jun 22, 2018
Cargo.toml bump version to 1.2.0 Jun 22, 2018
Dockerfile.example add docker example Aug 16, 2018
LICENSE.md add license Mar 12, 2018
Makefile move other binaries to examples Mar 13, 2018
README.md add bibtex Jul 18, 2018
rustfmt.toml first commit Jun 1, 2016

README.md

cntr

Build Status

Say no to $ apt install vim in containers! cntr is a replacement for docker exec that brings all your developers tools with you. This allows to ship minimal runtime image in production and limit the surface for exploits.

Cntr was also published in Usenix ATC 2018. See bibtex for citation.

Demo

In this two minute recording you learn all the basics of cntr:

asciicast

Features

  • For convenience cntr supports container names/identifier for the following container engines natively:
    • docker
    • LXC
    • LXD
    • rkt
    • systemd-nspawn
  • For other container engines cntr also takes process ids (PIDs) instead of container names.

Installation

Pre-build static-linked binary

For linux x86_64 we build static binaries for every release. More platforms can added on request. See the release tab for pre-build tarballs. At runtime only commandline utils of the container engine in questions are required.

Build from source

All you need for compilation is rust + cargo. Checkout rustup.rs on how to get a working rust toolchain. Then run:

Either:

$ cargo install cntr

Or the latest master:

$ cargo install --git https://github.com/Mic92/cntr

For offline builds we also provided a tarball with all dependencies bundled here for compilation with cargo-vendor.

Usage

At a high-level cntr provides two subcommands: attach and exec:

  • attach: Allows you to attach to a container with your own native shell/commands. Cntr will mount the container at /var/lib/cntr. The container itself will run unaffected as the mount changes are not visible to container processes.
    • Example: cntr attach <container_id> where container_id can be a container identifier or process id (see examples below).
  • exec: Once you are in the container, you can also run commands from the container filesystem itself. Since those might need their native mount layout at / instead of /var/lib/cntr, cntr provides exec subcommand to chroot to container again and also resets the environment variables that might have been changed by the shell.
    • Example: cntr exec <command> where command is an executable in the container

Note: Cntr needs to run on the same host as the container. It does not work if the container is running in a virtual machine while cntr is running on the hypervisor.

$ cntr --help
Usage:
    cntr COMMAND [ARGUMENTS ...]
Enter or executed in container
positional arguments:
  command               Command to run (either "attach" or "exec")
  arguments             Arguments for command
optional arguments:
  -h,--help             show this help message and exit
$ cntr attach --help
Usage:
    subcommand attach [OPTIONS] ID [COMMAND] [ARGUMENTS ...]
Enter container
positional arguments:
  id                    container id, container name or process id
  command               command to execute after attach (default: $SHELL)
  arguments             arguments passed to command
optional arguments:
  -h,--help             show this help message and exit
  --effective-user EFFECTIVE_USER
                        effective username that should be owner of new created
                        files on the host
  --type TYPE           Container type (docker|lxc|rkt|process_id|nspawn,
                        default: all)
$ cntr exec --help
Usage:
    subcommand exec [COMMAND] [ARGUMENTS ...]
Execute command in container filesystem
positional arguments:
  command               command to execute (default: $SHELL)
  arguments             Arguments to pass to command
optional arguments:
  -h,--help             show this help message and exit

Docker

1: Find out the container name/container id:

$ docker run --name boxbusy -ti busybox
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
55a93d71b53b        busybox             "sh"                22 seconds ago      Up 20 seconds                           boxbusy

Either provide a container id...

$ cntr attach 55a93d71b53b
[root@55a93d71b53b:/var/lib/cntr]# echo "I am in a container!"
[root@55a93d71b53b:/var/lib/cntr]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
40: eth0@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
[root@55a93d71b53b:/var/lib/cntr]# vim etc/resolv.conf

...or the container name. Use cntr exec to execute container native commands (while running in the cntr shell).

$ cntr attach boxbusy
[root@55a93d71b53b:/var/lib/cntr]# cntr exec sh -c 'busybox | head -1'

LXD

1: Create a container and start it

$ lxc image import images:/alpine/edge
$ lxc launch images:alpine/edge
$ lxc list
+-----------------+---------+------+------+------------+-----------+
|      NAME       |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+-----------------+---------+------+------+------------+-----------+
| amazed-sailfish | RUNNING |      |      | PERSISTENT | 0         |
+-----------------+---------+------+------+------------+-----------+

2: Attach to the container with cntr

$ cntr attach amazed-sailfish
$ cat etc/hostname
amazed-sailfish

LXC

1: Create a container and start it

$ lxc-create --name ubuntu -t download -- -d ubuntu -r xenial -a amd64
$ lxc-start --name ubuntu -F
...
Ubuntu 16.04.4 LTS ubuntu console
ubuntu login:
$ lxc-ls
ubuntu

2: Attach to container with cntr:

$ cntr attach ubuntu
[root@ubuntu2:/var/lib/cntr]# cat etc/os-release
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

rkt

1: Find out the container uuid:

$ rkt run --interactive=true docker://busybox
$ rkt list
UUID            APP     IMAGE NAME                                      STATE   CREATED         STARTED         NETWORKS
c2d2e87e        busybox registry-1.docker.io/library/busybox:latest     running 6 minutes ago   6 minutes ago   default:ip4=172.16.28.3

2: Attach with cntr

# make sure your container is still running!
$ cntr attach c2d2e87e
# Finally not the old ugly top!
[gen0@rkt-c2d2e87e-e798-4341-ae93-26f6cbb7c017:/var/lib/cntr]# htop
...

With cntr you can also debug stage1 of rkt - even there is no support from rkt itself.

$ ps aux | grep stage1
joerg    13546  0.0  0.0 120808  1608 pts/12   S+   11:10   0:00 grep --binary-files=without-match --directories=skip --color=auto stage1
root     22232  0.0  0.0  54208  2656 pts/7    S+   10:54   0:00 stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --notify-ready=yes --register=true --link-journal=try-guest --quiet --uuid=c2d2e87e-e798-4341-ae93-26f6cbb7c017 --machine=rkt-c2d2e87e-e798-4341-ae93-26f6cbb7c017 --directory=stage1/rootfs --capability=CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FSETID,CAP_FOWNER,CAP_KILL,CAP_MKNOD,CAP_NET_RAW,CAP_NET_BIND_SERVICE,CAP_SETUID,CAP_SETGID,CAP_SETPCAP,CAP_SETFCAP,CAP_SYS_CHROOT -- --default-standard-output=tty --log-target=null --show-status=0

Therefore we use the process id instead of the container uuid:

$ cntr attach 22232
# new and exiting territory!
[root@turingmachine:/var/lib/cntr]# mount | grep pods
sysfs on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)

systemd-nspawn

1: Start container

$ wget https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-root.tar.xz
$ mkdir /var/lib/machines/ubuntu
$ tar -xf ubuntu-16.04-server-cloudimg-amd64-root.tar.xz -C /var/lib/machines/ubuntu
$ systemd-nspawn -b -M ubuntu
$ machinectl list
MACHINE CLASS     SERVICE        OS     VERSION ADDRESSES
ubuntu  container systemd-nspawn ubuntu 16.04   -

2: Attach

$ cntr attach ubuntu

Generic process id

The minimal information needed by cntr is the process id of a container process you want to attach to.

# Did you now chromium uses namespaces too?
$ ps aux | grep 'chromium --type=renderer'
joerg    17498 11.7  1.0 1394504 174256 ?      Sl   15:16   0:08 /usr/bin/chromium

In this case 17498 is the pid we are looking for.

$ cntr attach 17498
# looks quite similar to our system, but with less users
[joerg@turingmachine cntr]$ ls -la /
total 240
drwxr-xr-x   23 nobody nogroup    23 Mar 13 15:05 .
drwxr-xr-x   23 nobody nogroup    23 Mar 13 15:05 ..
drwxr-xr-x    2 nobody nogroup     3 Mar 13 15:14 bin
drwxr-xr-x    4 nobody nogroup 16384 Jan  1  1970 boot
drwxr-xr-x   24 nobody nogroup  4120 Mar 13 14:56 dev
drwxr-xr-x   52 nobody nogroup   125 Mar 13 15:14 etc
drwxr-xr-x    3 nobody nogroup     3 Jan  8 16:17 home
drwxr-xr-x    8 nobody nogroup     8 Feb  9 22:10 mnt
dr-xr-xr-x  306 nobody nogroup     0 Mar 13 09:38 proc
drwx------   22 nobody nogroup    43 Mar 13 15:09 root
...

How it works

Cntr is container-agnostic: Instead of interfacing with container engines, it implements the underlying operating system API. It treats every container as a group of processes, that it can inherit properties from.

Cntr inherits the following container properties:

  • Namespaces (mount, uts, pid, net, cgroup, ipc)
  • Cgroups
  • Apparamor/selinux
  • Capabilities
  • User/group ids
  • Environment variables
  • The following files: /etc/passwd, /etc/hostname, /etc/hosts, /etc/resolv.conf

Under the hood it spawns a shell or user defined program that inherits the full context of the container and mount itself as a fuse filesystem.

We extensively evaluated the correctness and performance of cntr's filesystem using xfstests and a wide range of filesystem performance benchmarks (iozone, pgbench, dbench, fio, fs-mark, postmark, ...)

Related projects

  • nsenter
    • Only covers linux namespaces and the user is limited to tools installed in the containers
  • toolbox
    • Does attach from a container to the host, this is the opposite of what Cntr is doing

Bibtex

We published a paper with all technical details about Cntr in Usenix ATC 2018.

@inproceedings{cntr-atc18,
  author = {J{\"o}rg Thalheim and Pramod Bhatotia and Pedro Fonseca and Baris Kasikci},
  title = {Cntr: Lightweight {OS} Containers},
  booktitle = {2018 {USENIX} Annual Technical Conference ({USENIX} {ATC} 18)},
  year = {2018},
}