Directory with .wh..wh..opaque extension is presented inside image

[SofyaTavrovskaya]

Deleted directory with .wh..wh..opaque extension is presented inside image

I am writing a program that consequentially parses layers of image and tries to find binaries inside image. Because images can have in some layers deleted files and directories, I try to implement some merge approach to take a final state of the file system. But now I have some confusing situation with image that have layers with deleted directories, but this directory and all her content are presented in the final state.

The most interesting situation is that I can reproduce this behavior when building image on ubuntu:bionic or ubuntu:focal. But when build the image on virtual machine with ubuntu:xenial the same directory doesn't have .wh..wh..opaque extensions.

Steps to reproduce the issue:
I built some tests images to reproduce the issue:

1. For image building was using https://github.com/bitnami/bitnami-docker-kube-state-metrics/blob/1.9.4-debian-10-r13/1/debian-10/Dockerfile and images were built on 2 systems:

• ubuntu:bionic - stavrovskaya/prometeus:test_bionic
• ubuntu:xenial: stavrovskaya/prometeus:test_xenial

2. In stavrovskaya/prometeus:test_bionic the 3-d layer looks like this:

 'opt'
 'opt/bitnami'
 'opt/bitnami/.wh..wh..opq'
 'opt/bitnami/kube-state-metrics'
 'opt/bitnami/kube-state-metrics/bin'
 'opt/bitnami/kube-state-metrics/bin/kube-state-metrics'
 'opt/bitnami/kube-state-metrics/licenses'
 'opt/bitnami/kube-state-metrics/licenses/kube-state-metrics-1.9.4.txt'
 'tmp'
 'tmp/bitnami'
 'tmp/bitnami/.wh..wh..opq'
 'tmp/bitnami/pkg'
 'tmp/bitnami/pkg/cache'

In stavrovskaya/prometeus:test_xenial the same layer is:
'opt'
'opt/bitnami'
'opt/bitnami/kube-state-metrics'
'opt/bitnami/kube-state-metrics/bin'
'opt/bitnami/kube-state-metrics/bin/kube-state-metrics'
'opt/bitnami/kube-state-metrics/licenses'
'opt/bitnami/kube-state-metrics/licenses/kube-state-metrics-1.9.4.txt'
'tmp', 'tmp/bitnami'
'tmp/bitnami/pkg'
'tmp/bitnami/pkg/cache'

Interesting fact that official image bitnami/kube-state-metrics:1.9.4-debian-10-r13 equals by structure with stavrovskaya/prometeus:test_xenial

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:36 2021
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 35
  Running: 7
  Paused: 0
  Stopped: 28
 Images: 89
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-166-generic
 Operating System: Ubuntu 18.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.29GiB
 Name: stavrovskaya-pc
 ID: WFM5:SY5Y:BJZC:MQNG:6O6M:2LEG:MJXU:V56S:HQVU:POEG:FHOD:7GF2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: stavrovskaya
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.240.0.0/16, Size: 24

[thaJeztah]

Are both docker installs (the xenial and bionic one) using the same storage driver?

[thaJeztah]

(could you provide the docker version and docker info for both setups?)

[SofyaTavrovskaya]

@thaJeztah I provide more info, when war in Ukraine will end

[thaJeztah]

@SofyaTavrovskaya of course, no problem! (we were curious what the exact differences were and if it's a bug or expected behavior)

Stay safe!! Sending good vibes 💜

[SofyaTavrovskaya]

@thaJeztah
Ubuntu Bionic:

docker version:

ubuntu@bionic:~$ sudo docker version
Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:47:10 2022
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:45:30 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  
  Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-173-generic
 Operating System: Ubuntu 18.04.6 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 1
 Total Memory: 983.5MiB
 Name: bionic
 ID: 57I2:LT6F:3K2L:YXYR:5OKI:CQ35:CSH7:CKK2:UKN4:464Z:DJBC:OHBL
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Ubuntu Xenial:
docker version:

root@devstack_pike:~# docker version     
Client:                                                                                                                                                                                                    
 Version:           18.09.7                                                                                                                                                                                
 API version:       1.39                                                                                                                                                                                   
 Go version:        go1.10.4                                                                                                                                                                               
 Git commit:        2d0083d                                                                                                                                                                                
 Built:             Fri Dec 18 12:33:04 2020                                                                                                                                                               
 OS/Arch:           linux/amd64                                                                                                                                                                            
 Experimental:      false                                                                                                                                                                                  
                                                                                                                                                                                                           
Server:                                                                                                                                                                                                    
 Engine:                                                                                                                                                                                                   
  Version:          18.09.7                                                                                                                                                                                
  API version:      1.39 (minimum version 1.12)                                                                                                                                                            
  Go version:       go1.10.4
  Git commit:       2d0083d
  Built:            Fri Dec  4 23:02:49 2020
  OS/Arch:          linux/amd64
  Experimental:     false

docker info:

root@devstack_pike:~# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.09.7
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 
runc version: N/A
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-210-generic
Operating System: Ubuntu 16.04.7 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.796GiB
Name: devstack_pike
ID: SQ3Q:64QY:N3AN:SXL3:CCU2:73PD:ZXGJ:FDB6:P3R6:6TQ6:PPTG:UZPM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[sify21]

what's the prefix .wh..wh.. used for? From the image archive spec, I only know that deleted files are prefixed with .wh.. Does it mean that a whiteout file in previous layer is deleted?

• original layer contains file .opaque
• second layer whiteout the file .wh..opaque
• third layer whiteout the "whiteout file" in second layer .wh..wh..opaque

[SofyaTavrovskaya]

@sify21 If I'm not mistaken, directory with .wh..wh..opaque extension is deleted directory, and all content inside this directory in deleted too.

[sify21]

@SofyaTavrovskaya But according to the specs, a single prefix .wh. is enough to indicate that a file/dir is deleted, where does the double prefix come from?

moby/image/spec/v1.2.md

Line 563 in 7d9c114

deleted file or directory prefixed with `.wh.`. The filenames prefixed with

[SofyaTavrovskaya]

@sify21 It's interesting, because I can find that directories with this extension are proceeded and it can be found in project code:
https://github.com/moby/moby/blob/1.12.x/pkg/archive/whiteouts.go#L21

If I understood correctly in overlay2 storage drivers this works in the same way as a whiteout file and effectively prevents the directory from being accessed, even though it still exists in the image.

[sify21]

Oh I see, maybe the specs should clarify the usages of those patterns defined in https://github.com/moby/moby/blob/e78f6f9c68fee84430b76738c328d5b51d9c853a/pkg/archive/whiteouts.go

[SofyaTavrovskaya]

@sify21 It would be great.
Because now I have situation when deleted directory has prefix .wh.
It would be better to understand from spec when deleted directory marked with .wh. prefix and when - .wh..wh..opaque and what does it depend on.