Container doesn't have the expected behaviour after using commit --change

[jfusterm]

Description

Recently Packer has released the version 0.12.1 with support for the Docker builder to introduce changes via commit --change.

I have tested that functionality and the resulting containers doesn't work as expected. All the containers I have built with Packer uses tini as ENTRYPOINT, and the problem is that commit --change appends /bin/sh -c to both the ENTRYPOINT and the CMD, getting sh PID 1 instead of tini.

Steps to reproduce the issue:

1. Using an image built from a Dockerfile

FROM debian:jessie
ENV TINI_VERSION=v0.13.1
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/local/bin/tini
RUN chmod +x /usr/local/bin/tini
ENTRYPOINT ["tini", "--"]
CMD ["top"]

Build the image from that Dockerfile.

$ docker build -t tini:ok .
$ docker inspect -f "{{ .Config.Cmd }}" tini:ok
[top]
$ docker inspect -f "{{ .Config.Entrypoint }}" tini:ok
[tini --]
$ docker run -it —rm tini:ok 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0    4224    628    552 S   0.0  0.0   0:00.03 tini
    5 root      20   0   21948   2308   2000 R   0.0  0.1   0:00.00 top

Both CMD and ENTRYPOINT are defined and the container works as expected, having tini PID 1.

2. Using an image resulting from a commit

FROM debian:jessie
ENV TINI_VERSION=v0.13.1
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/local/bin/tini
RUN chmod +x /usr/local/bin/tini
CMD ["top"]

Build the image without ENTRYPOINT.

$ docker build -t tini:ko .
$ docker inspect -f "{{ .Config.Cmd }}" tini:ko
[top]
$ docker inspect -f "{{ .Config.Entrypoint }}" tini:ko
[]
$ docker run -it —rm tini:ko 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0   21948   2280   1968 R   0.0  0.1   0:00.03 top

The resulting image doesn't have ENTRYPOINT but works as expected without tini.

Now using commit --change, append the ENTRYPOINT and the CMD instruction.

$ docker run -d --name container-top tini:ko
ca8a054ef75e534563aa172d16642b6706ae127b6c0b660f503431f122860c16

$ docker commit -c "ENTRYPOINT tini --" -c "CMD top" container-top
sha256:df73193be07aede5155ecba5714b3802eef67e8a57bc40d625af3f8119b74f2f
$ docker inspect -f "{{ .Config.Cmd }}" df73193be07
[/bin/sh -c top]
$ docker inspect -f "{{ .Config.Entrypoint }}" df73193be07
[/bin/sh -c tini --]
$ docker run -it --rm df73193be07
tini (tini version 0.13.1 - git.ed34702)
Usage: tini [OPTIONS] PROGRAM -- [ARGS] | --version

Execute a program under the supervision of a valid init process (tini)

Command line options:

  --version: Show version and exit.
  -h: Show this help message and exit.
  -s: Register as a process subreaper (requires Linux >= 3.4).
  -v: Generate more verbose output. Repeat up to 3 times.
  -g: Send signals to the child's process group.
  -l: Show license and exit.

Environment variables:

  TINI_SUBREAPER: Register as a process subreaper (requires Linux >= 3.4)
  TINI_VERBOSITY: Set the verbosity level (default: 1)

After launching the container from the new image it doesn't start. Putting just the ENTRYPOINT with the command on it works, but not as expected.

$ docker commit -c "ENTRYPOINT tini -- top"  container-top
sha256:dfdd3c4c605851c6796d82cf0fa3fd85a21648634391c33182f4f20914574276
$ docker inspect -f "{{ .Config.Cmd }}" dfdd3c4c60
[]
$ docker inspect -f "{{ .Config.Entrypoint }}" dfdd3c4c60
[/bin/sh -c tini -- top]
$ docker run -it --rm dfdd3c4c60
 
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0    4336    752    668 S   0.0  0.0   0:00.02 sh
    5 root      20   0    4224    696    620 S   0.0  0.0   0:00.00 tini
    6 root      20   0   21920   2384   2060 R   0.0  0.1   0:00.00 top

tini doesn't have PID 1 but 5.

Describe the results you received:

If the image has been built using a Dockerfile, works as expected but if the image has been modified with commit --change the behaviour is different.

Describe the results you expected:

I would expect to start tini as PID 1 either building the image using Dockerfile or with commit --change. The problem is the appended /bin/sh -c.

Output of docker version:

Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 23:26:11 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 23:26:11 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 6
 Running: 3
 Paused: 0
 Stopped: 3
Images: 50
Server Version: 1.12.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 32
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.27-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.951 GiB
Name: moby
ID: IQER:325E:53RT:AVHJ:L5V2:4OLC:NOH7:UYXS:QRRA:AD4M:EUJV:MRNQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 99
 Goroutines: 130
 System Time: 2016-12-17T11:47:35.041308973Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Username: jfusterm
Registry: https://index.docker.io/v1/
WARNING: No kernel memory limit support
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

Using Docker for Mac.

[thaJeztah]

That's expected with the commands you're using; the "working" image uses the "exec form" (JSON) notation, therefore does not use the default /bin/sh -c, but when committing, you're using the "shell form", which will add the /bin/sh -c

If you change your --change to use the exec form as well, you'll see that things work as expected;

$ docker build -t tini:ko .
...

$ docker inspect -f "{{ json .Config.Cmd }}" tini:ko
["top"]

$ docker inspect -f "{{ json .Config.Entrypoint }}" tini:ko
null

Using exec-form notation;

$ docker rm -fv container-top
container-top

$ docker run -d --name container-top tini:ko
636e8b46f5fa354e202df817f365cd3d73e030b375a16644ba67d6e05acf040a

$ docker commit -c 'ENTRYPOINT ["tini", "--"]' -c 'CMD ["top"]' container-top
sha256:2e26d803d88f6fc46acd7632fa20a832b2962054bc1ec3e67a20b7b6e3b58a21

$ docker inspect -f "{{ json .Config.Cmd }}" 2e26d803d88f6fc46acd7632fa20a832b2962054bc1ec3e67a20b7b6e3b58a21
["top"]

$ docker inspect -f "{{ json .Config.Entrypoint }}" 2e26d803d88f6fc46acd7632fa20a832b2962054bc1ec3e67a20b7b6e3b58a21
["tini","--"]

$ docker run -it --rm 2e26d803d88f6fc46acd7632fa20a832b2962054bc1ec3e67a20b7b6e3b58a21

top - 18:44:06 up  4:18,  0 users,  load average: 0.03, 0.02, 0.00
Tasks:   2 total,   1 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.2 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.2 st
KiB Mem:   8175164 total,  1321532 used,  6853632 free,    35192 buffers
KiB Swap:        0 total,        0 used,        0 free.  1063944 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0    4224    652    576 S   0.0  0.0   0:00.03 tini
    7 root      20   0   21920   2408   2084 R   0.0  0.0   0:00.00 top

New --init feature in Docker 1.13

Note that Docker 1.13 has a new --init flag to automaticaly inject an init/reaper (also tini by default) into the container when starting, this allows you to do this without modifying the image. It's still recommended to include the init in your image if the process inside the container requires an init, but if you don't have control over the image you're running, it's a great way to run it without having to modify / commit the image. (see the pull requests #26061, and #28037)

For example, using the same tini:ko image;

$ docker run --init -it --rm tini:ko

Tasks:   2 total,   1 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.0 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8175164 total,  1320436 used,  6854728 free,    35648 buffers
KiB Swap:        0 total,        0 used,        0 free.  1063976 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0    1144      4      0 S   0.0  0.0   0:00.04 init
    7 root      20   0   21920   2408   2080 R   0.0  0.0   0:00.00 top

As you can see, init is PID 1 in the container now

I'll close this issue, because this is not a bug, but I hope this is useful to you.

[jfusterm]

Thanks @thaJeztah for the explanation and for the 1.13 sneak peek! I haven't noticed the exec form in the command.

Now CMD and ENTRYPOINT works as expected, but I have tried to extend an environmental variable using ENV and does not work :(

$ docker exec top env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=1f3da396cb31
no_proxy=*.local, 169.254/16
HOME=/root
$ docker commit --change 'ENV GOPATH=$HOME/go' top
sha256:f940a6b77df830b34fd9c9cdf3e8d32ac02c3cb666d2375323e74297101cab8d
$ docker inspect -f "{{ json .Config.Env }}" f940a6b77df83
["GOPATH=/go","no_proxy=*.local, 169.254/16","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]

I would expect GOPATH to be /root/go not /go, but $HOME was not evaluated. I doesn't work either with PATH=$PATH:$GOPATH/bin.

If I use a Dockerfile it works as expected. I have tried both formats, nor ENV GOPATH=$HOME/go neither ENV GOPATH $HOME/go work. Is there any special way to extend environmental variables using commit --change?

[thaJeztah]

@jfusterm no, I don't think it's possible. Docker build makes sure these are evaluated but docker commit does not take these steps; https://github.com/docker/docker/blob/ca6c6f0765aeccdb2730d03c05bd965906df8cd4/daemon/commit.go#L44

Is there a reason you're not using a Dockerfile?

[jfusterm]

Got it. I was just testing new Packer's functionality to add instructions like CMD, ENTRYPOINT or ENV via commit --change when building Docker images and I found out this behaviour.

[thaJeztah]

Feel free to open a separate issue for that though