chown not persisted between RUN steps in Dockerfile #46161

eest · 2023-08-06T01:25:10Z

Description

Trying to chown a directory in a Dockerfile RUN statement is not persisted in later RUN statements.

Reproduce

Create Dockerfile:

FROM debian:bookworm-slim

RUN ls -l /home
RUN mkdir /home/testdir && chown -v 1337:1337 /home/testdir && ls -l /home
RUN ls -l /home

Build it and inspect output:

# docker build -t chown-test:latest --no-cache .
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM debian:bookworm-slim
 ---> 7016e0cd8b19
Step 2/4 : RUN ls -l /home
 ---> Running in 456047c5dd90
total 0
Removing intermediate container 456047c5dd90
 ---> 82347e01fefa
Step 3/4 : RUN mkdir /home/testdir && chown -v 1337:1337 /home/testdir && ls -l /home
 ---> Running in a14df1b593af
changed ownership of '/home/testdir' from root:root to 1337:1337
total 4
drwxr-xr-x 2 1337 1337 4096 Aug  6 01:11 testdir
Removing intermediate container a14df1b593af
 ---> 3d5a3cf1549a
Step 4/4 : RUN ls -l /home
 ---> Running in 4b60305cb723
total 4
drwxr-xr-x 2 root root 4096 Aug  6 01:11 testdir
Removing intermediate container 4b60305cb723
 ---> bb8e4bd6b27f
Successfully built bb8e4bd6b27f
Successfully tagged chown-test:latest

Note that while chown reports changing the ownership to 1337:1337 and the follow-up && ls -l /home reports the expected ownership of the directory, the next RUN shows the file as owned by root:root.

Expected behavior

The change made by chown should be persisted in later steps.

docker version

Client:
 Version:           20.10.25
 API version:       1.41
 Go version:        go1.18.1
 Git commit:        20.10.25-0ubuntu1~22.04.1
 Built:             Fri Jul 14 21:58:09 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.25
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.1
  Git commit:       20.10.25-0ubuntu1~22.04.1
  Built:            Thu Jun 29 21:21:05 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~22.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

docker info

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 13
 Server Version: 20.10.25
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-78-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.918GiB
 Name: linux-test-1
 ID: YUF5:LCL5:ULRX:2HUX:TM2G:SKU4:6HLA:MUNW:VJDU:P6P7:ALP7:ANHI
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Looking at existing issues this seems similar to #39169 but from what I can tell I am not running with userns-remap enabled (this is just a default apt install docker.io on Ubuntu 22.04.3 LTS):

# cat /etc/docker/daemon.json
cat: /etc/docker/daemon.json: No such file or directory

# ps axuww | grep dockerd
root        1688  0.0  3.5 1373832 71544 ?       Ssl  00:55   0:01 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

The subuid/subgid files does exist on the host if relevant:

# cat /etc/subuid
ubuntu:100000:65536

# cat /etc/subgid
ubuntu:100000:65536

The text was updated successfully, but these errors were encountered:

eest · 2023-08-06T12:01:18Z

A friend (thanks @soysource!) mentioned not seeing the same problem on Ubuntu 22.04.3 LTS when running the upstream version of docker (instead of the ubuntu docker.io package), so I tried changing over based on https://docs.docker.com/engine/install/ubuntu/ and indeed the problem is not seen then (disabling buildkit to get the same type of output as my information above):

# DOCKER_BUILDKIT=0 docker build -t chown-test:latest --no-cache .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM debian:bookworm-slim
 ---> 7016e0cd8b19
Step 2/4 : RUN ls -l /home
 ---> Running in 4862fd7306cf
total 0
Removing intermediate container 4862fd7306cf
 ---> 2e367ceaa41a
Step 3/4 : RUN mkdir /home/testdir && chown -v 1337:1337 /home/testdir && ls -l /home
 ---> Running in e38f98f09fbd
changed ownership of '/home/testdir' from root:root to 1337:1337
total 4
drwxr-xr-x 2 1337 1337 4096 Aug  6 11:55 testdir
Removing intermediate container e38f98f09fbd
 ---> 490135952f88
Step 4/4 : RUN ls -l /home
 ---> Running in f5701d08da69
total 4
drwxr-xr-x 2 1337 1337 4096 Aug  6 11:55 testdir
Removing intermediate container f5701d08da69
 ---> 2bf330d52695
Successfully built 2bf330d52695
Successfully tagged chown-test:latest

For reference, here is when using buildkit (which also works):

# BUILDKIT_PROGRESS=plain docker build -t chown-test:latest --no-cache .
#0 building with "default" instance using docker driver

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 173B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/debian:bookworm-slim
#3 DONE 0.0s

#4 [1/4] FROM docker.io/library/debian:bookworm-slim
#4 CACHED

#5 [2/4] RUN ls -l /home
#5 0.145 total 0
#5 DONE 0.2s

#6 [3/4] RUN mkdir /home/testdir && chown -v 1337:1337 /home/testdir && ls -l /home
#6 0.289 changed ownership of '/home/testdir' from root:root to 1337:1337
#6 0.290 total 4
#6 0.290 drwxr-xr-x 2 1337 1337 4096 Aug  6 11:57 testdir
#6 DONE 0.3s

#7 [4/4] RUN ls -l /home
#7 0.168 total 4
#7 0.168 drwxr-xr-x 2 1337 1337 4096 Aug  6 11:57 testdir
#7 DONE 0.2s

#8 exporting to image
#8 exporting layers 0.1s done
#8 writing image sha256:3febd0f99c9b84daa39aa42138e84dab836e94444e41a47bc39ac9bccbc72c5e done
#8 naming to docker.io/library/chown-test:latest done
#8 DONE 0.1s

docker version

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:35:18 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:18 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 18
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-78-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.918GiB
 Name: linux-test-1
 ID: 84440848-055c-4046-905c-b86b771ce4bb
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

So whatever the problem I am seeing, it seems related to the docker.io package available for Ubuntu, not the upstream docker-ce package... Not sure it is still relevant as an issue here then.

cboitel · 2023-08-07T06:22:34Z

Bug reported in https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/2029564

You can revert to previous release of docker.io on Ubuntu 22.04. You can also apt-mark hold this package to prevent any update to occur until it is fixed.

thaJeztah · 2023-08-07T07:24:26Z

Yes, I tried to reproduce this with Docker's official packages (https://docs.docker.com/engine/install/ubuntu/). I installed v20.10.24 (note that Docker no longer builds packages for 20.10, so 20.10.24 is the last version, although I don't see anything related to this between v20.10.24 and v20.10.25 v20.10.24...v20.10.25);

DOCKER_BUILDKIT=0 docker build -t foo .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  19.97kB
Step 1/4 : FROM debian:bookworm-slim
bookworm-slim: Pulling from library/debian
648e0aadf75a: Pull complete
Digest: sha256:89468107e4c2b9fdea2f15fc582bf92c25aa4296a661ca0202f7ea2f4fc3f48c
Status: Downloaded newer image for debian:bookworm-slim
 ---> 7016e0cd8b19
Step 2/4 : RUN ls -l /home
 ---> Running in e0e517d4ca7a
total 0
Removing intermediate container e0e517d4ca7a
 ---> 1e256d338aa5
Step 3/4 : RUN mkdir /home/testdir && chown -v 1337:1337 /home/testdir && ls -l /home
 ---> Running in 9b95716f1563
changed ownership of '/home/testdir' from root:root to 1337:1337
total 4
drwxr-xr-x 2 1337 1337 4096 Aug  7 07:01 testdir
Removing intermediate container 9b95716f1563
 ---> 6c63a762b9ab
Step 4/4 : RUN ls -l /home
 ---> Running in a5ca9364c367
total 4
drwxr-xr-x 2 1337 1337 4096 Aug  7 07:01 testdir
Removing intermediate container a5ca9364c367
 ---> 9f96169c3e92
Successfully built 9f96169c3e92
Successfully tagged foo:latest

Using Ubuntu's docker.io v20.10.25 (20.10.25-0ubuntu1~22.04.1) package, the issue reproduces;

Client:
 Version:           20.10.25
 API version:       1.41
 Go version:        go1.18.1
 Git commit:        20.10.25-0ubuntu1~22.04.1
 Built:             Fri Jul 14 21:58:09 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.25
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.1
  Git commit:       20.10.25-0ubuntu1~22.04.1
  Built:            Thu Jun 29 21:21:05 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~22.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

DOCKER_BUILDKIT=0 docker build -t foo .
Sending build context to Docker daemon  20.99kB
Step 1/4 : FROM debian:bookworm-slim
bookworm-slim: Pulling from library/debian
648e0aadf75a: Pull complete
Digest: sha256:89468107e4c2b9fdea2f15fc582bf92c25aa4296a661ca0202f7ea2f4fc3f48c
Status: Downloaded newer image for debian:bookworm-slim
 ---> 7016e0cd8b19
Step 2/4 : RUN ls -l /home
 ---> Running in 8ced0784392c
total 0
Removing intermediate container 8ced0784392c
 ---> a09e63c02531
Step 3/4 : RUN mkdir /home/testdir && chown -v 1337:1337 /home/testdir && ls -l /home
 ---> Running in 6681a3df614e
changed ownership of '/home/testdir' from root:root to 1337:1337
total 4
drwxr-xr-x 2 1337 1337 4096 Aug  7 07:09 testdir
Removing intermediate container 6681a3df614e
 ---> 248d26dbb595
Step 4/4 : RUN ls -l /home
 ---> Running in 4cebe051cd53
total 4
drwxr-xr-x 2 root root 4096 Aug  7 07:09 testdir
Removing intermediate container 4cebe051cd53
 ---> c1fa835be601
Successfully built c1fa835be601
Successfully tagged foo:latest

After downgrading to 20.10.21-0ubuntu1~22.04.3, the issue goes away;

Client:
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.1
 Git commit:        20.10.21-0ubuntu1~22.04.3
 Built:             Thu Apr 27 05:57:17 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.1
  Git commit:       20.10.21-0ubuntu1~22.04.3
  Built:            Thu Apr 27 05:37:25 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~22.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

DOCKER_BUILDKIT=0 docker build -t foo .
Sending build context to Docker daemon  20.99kB
Step 1/4 : FROM debian:bookworm-slim
bookworm-slim: Pulling from library/debian
648e0aadf75a: Pull complete
Digest: sha256:89468107e4c2b9fdea2f15fc582bf92c25aa4296a661ca0202f7ea2f4fc3f48c
Status: Downloaded newer image for debian:bookworm-slim
 ---> 7016e0cd8b19
Step 2/4 : RUN ls -l /home
 ---> Running in ffbf811c2a38
total 0
Removing intermediate container ffbf811c2a38
 ---> 332243bc8604
Step 3/4 : RUN mkdir /home/testdir && chown -v 1337:1337 /home/testdir && ls -l /home
 ---> Running in 6416aa73ca5c
changed ownership of '/home/testdir' from root:root to 1337:1337
total 4
drwxr-xr-x 2 1337 1337 4096 Aug  7 07:18 testdir
Removing intermediate container 6416aa73ca5c
 ---> d9ebf7e92b20
Step 4/4 : RUN ls -l /home
 ---> Running in 0ae293b1459a
total 4
drwxr-xr-x 2 1337 1337 4096 Aug  7 07:18 testdir
Removing intermediate container 0ae293b1459a
 ---> 21a39d01f02a
Successfully built 21a39d01f02a
Successfully tagged foo:latest

thaJeztah · 2023-08-07T07:28:19Z

I'll close this ticket because this looks like a packaging issue in Ubuntu's packages.

If possible, I'd recommend using the official packages from download.docker.com (see https://docs.docker.com/engine/install/ubuntu), and a current version of the engine (24.0.x). Also note that the classic builder is being deprecated, so unless you have specific reasons to use it, I'd recommend using BuildKit as builder (you can set DOCKER_BUILDKIT=1 if you're using docker 20.10, but it's enabled by default in current versions)

damien-git · 2023-08-16T18:20:05Z

This bug is being tracked at https://bugs.launchpad.net/ubuntu/+source/docker.io-app/+bug/2029523 . A fix was committed on 8/11.

eest added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. status/0-triage labels Aug 6, 2023

eest mentioned this issue Aug 6, 2023

Docker root owned files ory/kratos-selfservice-ui-node#273

Open

6 tasks

thaJeztah added area/builder version/20.10 area/builder/classic-builder Issues affecting the classic builder and removed status/0-triage kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. labels Aug 7, 2023

thaJeztah closed this as not planned Won't fix, can't repro, duplicate, stale Aug 7, 2023

thaJeztah mentioned this issue Aug 7, 2023

RUN chown command does not work as expected docker/cli#4481

Closed

vvoland mentioned this issue Aug 10, 2023

[20.10] Discourage from building with Go older than 1.19 #46186

Closed

This was referenced Aug 14, 2023

docker build fails to persist owner:group of files #46233

Closed

Unable to chown after ADD or COPY any files. docker/cli#4498

Closed

thaJeztah mentioned this issue Aug 16, 2023

docker export container changes files ownership/groupship to root #46249

Closed

vvoland mentioned this issue Aug 17, 2023

Docker chown not working? docker/cli#4499

Closed

vvoland mentioned this issue Aug 29, 2023

USER command doesn't work as expected with v20.10.25 #46355

Closed

macbre mentioned this issue Sep 6, 2023

Error when using Github Actions and building with Docker 23 persists macbre/push-to-ghcr#33

Open

quic-yvasi mentioned this issue Oct 6, 2023

Bug in docker file quic/gunyah-support-scripts#6

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chown not persisted between RUN steps in Dockerfile #46161

chown not persisted between RUN steps in Dockerfile #46161

eest commented Aug 6, 2023 •

edited

Loading

eest commented Aug 6, 2023 •

edited

Loading

cboitel commented Aug 7, 2023 •

edited

Loading

thaJeztah commented Aug 7, 2023

thaJeztah commented Aug 7, 2023

damien-git commented Aug 16, 2023

chown not persisted between RUN steps in Dockerfile #46161

chown not persisted between RUN steps in Dockerfile #46161

Comments

eest commented Aug 6, 2023 • edited Loading

Description

Reproduce

Expected behavior

docker version

docker info

Additional Info

eest commented Aug 6, 2023 • edited Loading

docker version

docker info

cboitel commented Aug 7, 2023 • edited Loading

thaJeztah commented Aug 7, 2023

thaJeztah commented Aug 7, 2023

damien-git commented Aug 16, 2023

eest commented Aug 6, 2023 •

edited

Loading

eest commented Aug 6, 2023 •

edited

Loading

cboitel commented Aug 7, 2023 •

edited

Loading