overlay-mode containerization breaks apt: Invalid cross-device link
#13123
Labels
kind/bug
Categorizes issue or PR as related to a bug.
locked - please file new issue/PR
Assist humans wanting to comment on an old issue or PR with locked comments.
Comments
|
@giuseppe PTAL |
|
that error is reported by overlay: You can more easily reproduce it with something like: and you get: apt lacks the fallback code that coreutils has to handle in fuse-overlayfs I've added the check slightly different, and if the source directory is not present in the lower layers, it still succeeds. There is nothing we can do in Podman, it has either to be changed in the linux kernel, or better in apt so to handle Could you try with a newer apt though? @rhvgoyal FYI |
Be it my will, I would use Fedora instead of Ubuntu :-) But I need exactly the old Ubuntu for work. I guess my best bet currently is to use fuse-overlayfs. |
|
Hiya, I think there's something worth taking a second look at here. I'm also getting stumped trying to build Ubuntu containers (for work 🙄) with I've written a reproduction script based on the very helpful clues from both @Hi-Angel and @giuseppe above. Thanks a lot for them! This was pretty arcane and I would have had no idea where to start without getting to stand on your shoulders.
|
| STORAGE_DRIVER=vfs | STORAGE_DRIVER=overlay | STORAGE_DRIVER=overlay + fuse-overlayfs | |
|---|---|---|---|
| rootless | 🟢 0 |
🔴 EXDEV |
🔴 EXDEV |
| rootful | 🟢 0 |
🟢 0 |
🔴 EXDEV |
Full logs
rootless, vfs
$ STORAGE_DRIVER=vfs STORAGE_OPTS="" sh repro.sh
+ podman version
Version: 3.4.4
API Version: 3.4.4
Go Version: go1.17.4
Git Commit: f6526ada1025c2e3f88745ba83b8b461ca659933
Built: Thu Dec 9 13:30:40 2021
OS/Arch: linux/amd64
+ podman info --debug
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.0-1
path: /usr/bin/conmon
version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
cpus: 4
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: requiem
idMappings:
gidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 100000
size: 65537
uidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 100000
size: 65537
kernel: 5.16.10-arch1-1
linkmode: dynamic
logDriver: journald
memFree: 9490178048
memTotal: 16446435328
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.4.2-1
path: /usr/bin/crun
version: |-
crun version 1.4.2
commit: f6fbc8f840df1a414f31a60953ae514fa497c748
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/user/1001/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
version: |-
slirp4netns version 1.1.12
commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.3
swapFree: 0
swapTotal: 0
uptime: 15h 37m 21.55s (Approximately 0.62 days)
plugins:
log:
- k8s-file
- none
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /home/kousu/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: vfs
graphOptions: {}
graphRoot: /home/kousu/.local/share/containers/storage
graphStatus: {}
imageStore:
number: 1
runRoot: /run/user/1001/containers
volumePath: /home/kousu/.local/share/containers/storage/volumes
version:
APIVersion: 3.4.4
Built: 1639074640
BuiltTime: Thu Dec 9 13:30:40 2021
GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
GoVersion: go1.17.4
OsArch: linux/amd64
Version: 3.4.4
+ findmnt --target=/home/kousu/.local/share/containers/storage
TARGET SOURCE FSTYPE OPTIONS
/home /dev/sda9 ext2 rw,relatime
+ df -h /home/kousu/.local/share/containers/storage
Filesystem Size Used Avail Use% Mounted on
/dev/sda9 391G 333G 38G 90% /home
+ findmnt --target=/run/user/1001/containers
TARGET SOURCE FSTYPE OPTIONS
/run/user/1001 tmpfs tmpfs rw,nosuid,nodev,relatime,size=14454876k,nr_inodes=401524,mode=700,uid=1001,gid=1001,inode64
+ df -h /run/user/1001/containers
Filesystem Size Used Avail Use% Mounted on
tmpfs 14G 1.4M 14G 1% /run/user/1001
+ podman build -t lower_layer .
STEP 1/5: FROM ubuntu:20.04
STEP 2/5: RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y strace
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [25.8 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1577 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [982 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [841 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1134 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [29.4 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1052 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2006 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.8 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [50.8 kB]
Fetched 21.2 MB in 9s (2482 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libunwind8
The following NEW packages will be installed:
libunwind8 strace
0 upgraded, 2 newly installed, 0 to remove and 10 not upgraded.
Need to get 428 kB of archives.
After this operation, 1844 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libunwind8 amd64 1.2.1-9build1 [47.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 strace amd64 5.5-3ubuntu1 [380 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 428 kB in 2s (227 kB/s)
Selecting previously unselected package libunwind8:amd64.
(Reading database ... 4127 files and directories currently installed.)
Preparing to unpack .../libunwind8_1.2.1-9build1_amd64.deb ...
Unpacking libunwind8:amd64 (1.2.1-9build1) ...
Selecting previously unselected package strace.
Preparing to unpack .../strace_5.5-3ubuntu1_amd64.deb ...
Unpacking strace (5.5-3ubuntu1) ...
Setting up libunwind8:amd64 (1.2.1-9build1) ...
Setting up strace (5.5-3ubuntu1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
--> df1731ea9d6
STEP 3/5: COPY rename /usr/bin/
--> 9c9d15b26ec
STEP 4/5: RUN mkdir -p /home/dir_lower
--> 6664e7d13e3
STEP 5/5: RUN touch /home/file_lower
COMMIT lower_layer
--> ffdadd2440f
Successfully tagged localhost/lower_layer:latest
ffdadd2440f0af24719c27d2efb02b333056c2c05be901260afd3c69ffa128c3
Run the built container image, on vfs (via (kernel)) (rootless: true):
+ mkdir /home/dir_upper
+ touch /home/file_upper
+ ls -l /home
total 8
drwxr-xr-x 2 root root 4096 Feb 22 09:13 dir_lower
drwxr-xr-x 2 root root 4096 Feb 22 09:13 dir_upper
-rw-r--r-- 1 root root 0 Feb 22 09:13 file_lower
-rw-r--r-- 1 root root 0 Feb 22 09:13 file_upper
+ strace -e rename rename /home/file_upper /home/file_upper.bak
rename("/home/file_upper", "/home/file_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_upper /home/dir_upper.bak
rename("/home/dir_upper", "/home/dir_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/file_lower /home/file_lower.bak
rename("/home/file_lower", "/home/file_lower.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_lower /home/dir_lower.bak
rename("/home/dir_lower", "/home/dir_lower.bak") = 0
+++ exited with 0 +++
+ ls /home
dir_lower.bak dir_upper.bak file_lower.bak file_upper.bak
rootful, vfs
$ sudo STORAGE_DRIVER=vfs STORAGE_OPTS="" sh repro.sh
[sudo] password for kousu:
+ podman version
Version: 3.4.4
API Version: 3.4.4
Go Version: go1.17.4
Git Commit: f6526ada1025c2e3f88745ba83b8b461ca659933
Built: Thu Dec 9 13:30:40 2021
OS/Arch: linux/amd64
+ podman info --debug
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- rdma
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.0-1
path: /usr/bin/conmon
version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
cpus: 4
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: requiem
idMappings:
gidmap: null
uidmap: null
kernel: 5.16.10-arch1-1
linkmode: dynamic
logDriver: journald
memFree: 9070465024
memTotal: 16446435328
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.4.2-1
path: /usr/bin/crun
version: |-
crun version 1.4.2
commit: f6fbc8f840df1a414f31a60953ae514fa497c748
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
version: |-
slirp4netns version 1.1.12
commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.3
swapFree: 0
swapTotal: 0
uptime: 15h 38m 10.27s (Approximately 0.62 days)
plugins:
log:
- k8s-file
- none
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: vfs
graphOptions: {}
graphRoot: /var/lib/containers/storage
graphStatus: {}
imageStore:
number: 0
runRoot: /run/containers/storage
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 3.4.4
Built: 1639074640
BuiltTime: Thu Dec 9 13:30:40 2021
GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
GoVersion: go1.17.4
OsArch: linux/amd64
Version: 3.4.4
+ findmnt --target=/var/lib/containers/storage
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda7 ext4 rw,relatime
+ df -h /var/lib/containers/storage
Filesystem Size Used Avail Use% Mounted on
/dev/sda7 30G 25G 3.1G 89% /
+ findmnt --target=/run/containers/storage
TARGET SOURCE FSTYPE OPTIONS
/run run tmpfs rw,nosuid,nodev,relatime,mode=755,inode64
+ df -h /run/containers/storage
Filesystem Size Used Avail Use% Mounted on
run 7.7G 1.3M 7.7G 1% /run
+ podman build -t lower_layer .
STEP 1/5: FROM ubuntu:20.04
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/00-shortnames.conf)
Trying to pull docker.io/library/ubuntu:20.04...
Getting image source signatures
Copying blob 08c01a0ec47e done
Copying config 54c9d81cbb done
Writing manifest to image destination
Storing signatures
STEP 2/5: RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y strace
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [25.8 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [841 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [982 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1577 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:12 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1052 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1134 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [29.4 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2006 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [50.8 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.8 kB]
Fetched 21.2 MB in 7s (2994 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libunwind8
The following NEW packages will be installed:
libunwind8 strace
0 upgraded, 2 newly installed, 0 to remove and 10 not upgraded.
Need to get 428 kB of archives.
After this operation, 1844 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libunwind8 amd64 1.2.1-9build1 [47.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 strace amd64 5.5-3ubuntu1 [380 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 428 kB in 2s (177 kB/s)
Selecting previously unselected package libunwind8:amd64.
(Reading database ... 4127 files and directories currently installed.)
Preparing to unpack .../libunwind8_1.2.1-9build1_amd64.deb ...
Unpacking libunwind8:amd64 (1.2.1-9build1) ...
Selecting previously unselected package strace.
Preparing to unpack .../strace_5.5-3ubuntu1_amd64.deb ...
Unpacking strace (5.5-3ubuntu1) ...
Setting up libunwind8:amd64 (1.2.1-9build1) ...
Setting up strace (5.5-3ubuntu1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
--> d7b7dadf630
STEP 3/5: COPY rename /usr/bin/
--> 80b2e2fd9d4
STEP 4/5: RUN mkdir -p /home/dir_lower
--> c76af0afca3
STEP 5/5: RUN touch /home/file_lower
COMMIT lower_layer
--> 0b177c2620c
Successfully tagged localhost/lower_layer:latest
0b177c2620c8c51ab2b541b6759c71798ddf4cbd5b7e1a54319dbe301f2b1f18
Run the built container image, on vfs (via (kernel)) (rootless: false):
+ mkdir /home/dir_upper
+ touch /home/file_upper
+ ls -l /home
total 8
drwxr-xr-x 2 root root 4096 Feb 22 09:14 dir_lower
drwxr-xr-x 2 root root 4096 Feb 22 09:14 dir_upper
-rw-r--r-- 1 root root 0 Feb 22 09:14 file_lower
-rw-r--r-- 1 root root 0 Feb 22 09:14 file_upper
+ strace -e rename rename /home/file_upper /home/file_upper.bak
rename("/home/file_upper", "/home/file_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_upper /home/dir_upper.bak
rename("/home/dir_upper", "/home/dir_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/file_lower /home/file_lower.bak
rename("/home/file_lower", "/home/file_lower.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_lower /home/dir_lower.bak
rename("/home/dir_lower", "/home/dir_lower.bak") = 0
+++ exited with 0 +++
+ ls /home
dir_lower.bak dir_upper.bak file_lower.bak file_upper.bak
rootless, overlay (pure kernel)
$ STORAGE_DRIVER=overlay STORAGE_OPTS="" sh repro.sh
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve
In order to switch graph drivers, you must erase your existing containers/images/cache.
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve
WARNING! This will remove:
- all containers
- all pods
- all images
- all build cache
WARNING! The following external containers will be purged:
- d0918f48dec5 (ubuntu-working-container)
Are you sure you want to continue? [y/N] y
ERRO[0005] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve
+ podman version
Version: 3.4.4
API Version: 3.4.4
Go Version: go1.17.4
Git Commit: f6526ada1025c2e3f88745ba83b8b461ca659933
Built: Thu Dec 9 13:30:40 2021
OS/Arch: linux/amd64
+ podman info --debug
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.0-1
path: /usr/bin/conmon
version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
cpus: 4
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: requiem
idMappings:
gidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 100000
size: 65537
uidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 100000
size: 65537
kernel: 5.16.10-arch1-1
linkmode: dynamic
logDriver: journald
memFree: 9199210496
memTotal: 16446435328
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.4.2-1
path: /usr/bin/crun
version: |-
crun version 1.4.2
commit: f6fbc8f840df1a414f31a60953ae514fa497c748
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/user/1001/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
version: |-
slirp4netns version 1.1.12
commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.3
swapFree: 0
swapTotal: 0
uptime: 15h 39m 33.39s (Approximately 0.62 days)
plugins:
log:
- k8s-file
- none
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /home/kousu/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/kousu/.local/share/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 0
runRoot: /run/user/1001/containers
volumePath: /home/kousu/.local/share/containers/storage/volumes
version:
APIVersion: 3.4.4
Built: 1639074640
BuiltTime: Thu Dec 9 13:30:40 2021
GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
GoVersion: go1.17.4
OsArch: linux/amd64
Version: 3.4.4
+ findmnt --target=/home/kousu/.local/share/containers/storage
TARGET SOURCE FSTYPE OPTIONS
/home /dev/sda9 ext2 rw,relatime
+ df -h /home/kousu/.local/share/containers/storage
Filesystem Size Used Avail Use% Mounted on
/dev/sda9 391G 333G 38G 90% /home
+ findmnt --target=/run/user/1001/containers
TARGET SOURCE FSTYPE OPTIONS
/run/user/1001 tmpfs tmpfs rw,nosuid,nodev,relatime,size=14454876k,nr_inodes=401524,mode=700,uid=1001,gid=1001,inode64
+ df -h /run/user/1001/containers
Filesystem Size Used Avail Use% Mounted on
tmpfs 14G 1.4M 14G 1% /run/user/1001
+ podman build -t lower_layer .
STEP 1/5: FROM ubuntu:20.04
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/00-shortnames.conf)
Trying to pull docker.io/library/ubuntu:20.04...
Getting image source signatures
Copying blob 08c01a0ec47e done
Copying config 54c9d81cbb done
Writing manifest to image destination
Storing signatures
STEP 2/5: RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y strace
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [841 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1577 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [982 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [25.8 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:11 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1052 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [29.4 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1134 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2006 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.8 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [50.8 kB]
Fetched 21.2 MB in 1min 5s (328 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libunwind8
The following NEW packages will be installed:
libunwind8 strace
0 upgraded, 2 newly installed, 0 to remove and 10 not upgraded.
Need to get 428 kB of archives.
After this operation, 1844 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libunwind8 amd64 1.2.1-9build1 [47.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 strace amd64 5.5-3ubuntu1 [380 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 428 kB in 2s (184 kB/s)
Selecting previously unselected package libunwind8:amd64.
(Reading database ... 4127 files and directories currently installed.)
Preparing to unpack .../libunwind8_1.2.1-9build1_amd64.deb ...
Unpacking libunwind8:amd64 (1.2.1-9build1) ...
Selecting previously unselected package strace.
Preparing to unpack .../strace_5.5-3ubuntu1_amd64.deb ...
Unpacking strace (5.5-3ubuntu1) ...
Setting up libunwind8:amd64 (1.2.1-9build1) ...
Setting up strace (5.5-3ubuntu1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
--> 5aa8fdfdccf
STEP 3/5: COPY rename /usr/bin/
--> aa7df45abce
STEP 4/5: RUN mkdir -p /home/dir_lower
--> 9aa09ffa23e
STEP 5/5: RUN touch /home/file_lower
COMMIT lower_layer
--> 6bbcc23768b
Successfully tagged localhost/lower_layer:latest
6bbcc23768bf52369b2339e4e5c7ed0df0fefb26530bc98978f3eaa3e296b6b5
Run the built container image, on overlay (via (kernel)) (rootless: true):
+ mkdir /home/dir_upper
+ touch /home/file_upper
+ ls -l /home
total 8
drwxr-xr-x 2 root root 4096 Feb 22 09:17 dir_lower
drwxr-xr-x 2 root root 4096 Feb 22 09:17 dir_upper
-rw-r--r-- 1 root root 0 Feb 22 09:17 file_lower
-rw-r--r-- 1 root root 0 Feb 22 09:17 file_upper
+ strace -e rename rename /home/file_upper /home/file_upper.bak
rename("/home/file_upper", "/home/file_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_upper /home/dir_upper.bak
rename("/home/dir_upper", "/home/dir_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/file_lower /home/file_lower.bak
rename("/home/file_lower", "/home/file_lower.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_lower /home/dir_lower.bak
rename("/home/dir_lower", "/home/dir_lower.bak") = -1 EXDEV (Invalid cross-device link)
rename: Invalid cross-device link
+++ exited with 2 +++
rootful, overlay (pure kernel)
$ sudo STORAGE_DRIVER=overlay STORAGE_OPTS="" sh repro.sh
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve
In order to switch graph drivers, you must erase your existing containers/images/cache.
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve
WARNING! This will remove:
- all containers
- all pods
- all images
- all build cache
Are you sure you want to continue? [y/N] y
ERRO[0001] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve
A storage.conf file exists at /etc/containers/storage.conf
You should remove this file if you did not modified the configuration.
A storage.conf file exists at /etc/containers/storage.conf
You should remove this file if you did not modified the configuration.
+ podman version
Version: 3.4.4
API Version: 3.4.4
Go Version: go1.17.4
Git Commit: f6526ada1025c2e3f88745ba83b8b461ca659933
Built: Thu Dec 9 13:30:40 2021
OS/Arch: linux/amd64
+ podman info --debug
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- rdma
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.0-1
path: /usr/bin/conmon
version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
cpus: 4
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: requiem
idMappings:
gidmap: null
uidmap: null
kernel: 5.16.10-arch1-1
linkmode: dynamic
logDriver: journald
memFree: 9665216512
memTotal: 16446435328
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.4.2-1
path: /usr/bin/crun
version: |-
crun version 1.4.2
commit: f6fbc8f840df1a414f31a60953ae514fa497c748
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
version: |-
slirp4netns version 1.1.12
commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.3
swapFree: 0
swapTotal: 0
uptime: 15h 41m 33.6s (Approximately 0.62 days)
plugins:
log:
- k8s-file
- none
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /var/lib/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "true"
imageStore:
number: 0
runRoot: /run/containers/storage
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 3.4.4
Built: 1639074640
BuiltTime: Thu Dec 9 13:30:40 2021
GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
GoVersion: go1.17.4
OsArch: linux/amd64
Version: 3.4.4
+ findmnt --target=/var/lib/containers/storage
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda7 ext4 rw,relatime
+ df -h /var/lib/containers/storage
Filesystem Size Used Avail Use% Mounted on
/dev/sda7 30G 25G 3.1G 89% /
+ findmnt --target=/run/containers/storage
TARGET SOURCE FSTYPE OPTIONS
/run run tmpfs rw,nosuid,nodev,relatime,mode=755,inode64
+ df -h /run/containers/storage
Filesystem Size Used Avail Use% Mounted on
run 7.7G 1.3M 7.7G 1% /run
+ podman build -t lower_layer .
STEP 1/5: FROM ubuntu:20.04
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/00-shortnames.conf)
Trying to pull docker.io/library/ubuntu:20.04...
Getting image source signatures
Copying blob 08c01a0ec47e done
Copying config 54c9d81cbb done
Writing manifest to image destination
Storing signatures
STEP 2/5: RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y strace
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [25.8 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [841 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [982 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2006 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1134 kB]
Get:14 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1577 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1052 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [29.4 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [50.8 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.8 kB]
Fetched 21.2 MB in 7s (3184 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libunwind8
The following NEW packages will be installed:
libunwind8 strace
0 upgraded, 2 newly installed, 0 to remove and 10 not upgraded.
Need to get 428 kB of archives.
After this operation, 1844 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libunwind8 amd64 1.2.1-9build1 [47.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 strace amd64 5.5-3ubuntu1 [380 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 428 kB in 1s (400 kB/s)
Selecting previously unselected package libunwind8:amd64.
(Reading database ... 4127 files and directories currently installed.)
Preparing to unpack .../libunwind8_1.2.1-9build1_amd64.deb ...
Unpacking libunwind8:amd64 (1.2.1-9build1) ...
Selecting previously unselected package strace.
Preparing to unpack .../strace_5.5-3ubuntu1_amd64.deb ...
Unpacking strace (5.5-3ubuntu1) ...
Setting up libunwind8:amd64 (1.2.1-9build1) ...
Setting up strace (5.5-3ubuntu1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
--> 1d1d8231dc8
STEP 3/5: COPY rename /usr/bin/
--> dcbe579fc68
STEP 4/5: RUN mkdir -p /home/dir_lower
--> 0ed6e090c37
STEP 5/5: RUN touch /home/file_lower
COMMIT lower_layer
--> a8c1a0981e7
Successfully tagged localhost/lower_layer:latest
a8c1a0981e75312e9b8689cb75eeda62625cbd28924707b1e3b354aac4be4db8
Run the built container image, on overlay (via (kernel)) (rootless: false):
+ mkdir /home/dir_upper
+ touch /home/file_upper
+ ls -l /home
total 8
drwxr-xr-x 2 root root 4096 Feb 22 09:18 dir_lower
drwxr-xr-x 2 root root 4096 Feb 22 09:18 dir_upper
-rw-r--r-- 1 root root 0 Feb 22 09:18 file_lower
-rw-r--r-- 1 root root 0 Feb 22 09:18 file_upper
+ strace -e rename rename /home/file_upper /home/file_upper.bak
rename("/home/file_upper", "/home/file_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_upper /home/dir_upper.bak
rename("/home/dir_upper", "/home/dir_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/file_lower /home/file_lower.bak
rename("/home/file_lower", "/home/file_lower.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_lower /home/dir_lower.bak
rename("/home/dir_lower", "/home/dir_lower.bak") = 0
+++ exited with 0 +++
+ ls /home
dir_lower.bak dir_upper.bak file_lower.bak file_upper.bak
rootless, overlay (w/ FUSE)
$ STORAGE_DRIVER=overlay STORAGE_OPTS="overlay.mount_program=/usr/bin/fuse-overlayfs" sh repro.sh
+ podman version
Version: 3.4.4
API Version: 3.4.4
Go Version: go1.17.4
Git Commit: f6526ada1025c2e3f88745ba83b8b461ca659933
Built: Thu Dec 9 13:30:40 2021
OS/Arch: linux/amd64
+ podman info --debug
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.0-1
path: /usr/bin/conmon
version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
cpus: 4
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: requiem
idMappings:
gidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 100000
size: 65537
uidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 100000
size: 65537
kernel: 5.16.10-arch1-1
linkmode: dynamic
logDriver: journald
memFree: 9578074112
memTotal: 16446435328
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.4.2-1
path: /usr/bin/crun
version: |-
crun version 1.4.2
commit: f6fbc8f840df1a414f31a60953ae514fa497c748
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/user/1001/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
version: |-
slirp4netns version 1.1.12
commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.3
swapFree: 0
swapTotal: 0
uptime: 15h 42m 36.29s (Approximately 0.62 days)
plugins:
log:
- k8s-file
- none
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /home/kousu/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /usr/bin/fuse-overlayfs
Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.8.2-1
Version: |-
fusermount3 version: 3.10.5
fuse-overlayfs: version 1.8.2
FUSE library version 3.10.5
using FUSE kernel interface version 7.31
graphRoot: /home/kousu/.local/share/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 5
runRoot: /run/user/1001/containers
volumePath: /home/kousu/.local/share/containers/storage/volumes
version:
APIVersion: 3.4.4
Built: 1639074640
BuiltTime: Thu Dec 9 13:30:40 2021
GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
GoVersion: go1.17.4
OsArch: linux/amd64
Version: 3.4.4
+ findmnt --target=/home/kousu/.local/share/containers/storage
TARGET SOURCE FSTYPE OPTIONS
/home /dev/sda9 ext2 rw,relatime
+ df -h /home/kousu/.local/share/containers/storage
Filesystem Size Used Avail Use% Mounted on
/dev/sda9 391G 333G 38G 90% /home
+ findmnt --target=/run/user/1001/containers
TARGET SOURCE FSTYPE OPTIONS
/run/user/1001 tmpfs tmpfs rw,nosuid,nodev,relatime,size=14454876k,nr_inodes=401524,mode=700,uid=1001,gid=1001,inode64
+ df -h /run/user/1001/containers
Filesystem Size Used Avail Use% Mounted on
tmpfs 14G 1.4M 14G 1% /run/user/1001
+ podman build -t lower_layer .
STEP 1/5: FROM ubuntu:20.04
STEP 2/5: RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y strace
--> Using cache 5aa8fdfdccf979dd68ec92ded415d56c7a6fea72a29a6523bcdb18750f5266d7
--> 5aa8fdfdccf
STEP 3/5: COPY rename /usr/bin/
--> Using cache aa7df45abceb9727efa5a87ea7a1ebbae8cbc4f8ea96fceda4f9a05bad543803
--> aa7df45abce
STEP 4/5: RUN mkdir -p /home/dir_lower
--> Using cache 9aa09ffa23e3b7ebe9aae02a959a601e1362c420d07fcb9a454e29065e7560d4
--> 9aa09ffa23e
STEP 5/5: RUN touch /home/file_lower
--> Using cache 6bbcc23768bf52369b2339e4e5c7ed0df0fefb26530bc98978f3eaa3e296b6b5
COMMIT lower_layer
--> 6bbcc23768b
Successfully tagged localhost/lower_layer:latest
6bbcc23768bf52369b2339e4e5c7ed0df0fefb26530bc98978f3eaa3e296b6b5
Run the built container image, on overlay (via fuse-overlayfs) (rootless: true):
+ mkdir /home/dir_upper
+ touch /home/file_upper
+ ls -l /home
total 8
drwxr-xr-x 2 root root 4096 Feb 22 09:17 dir_lower
drwxr-xr-x 2 root root 4096 Feb 22 09:18 dir_upper
-rw-r--r-- 1 root root 0 Feb 22 09:17 file_lower
-rw-r--r-- 1 root root 0 Feb 22 09:18 file_upper
+ strace -e rename rename /home/file_upper /home/file_upper.bak
rename("/home/file_upper", "/home/file_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_upper /home/dir_upper.bak
rename("/home/dir_upper", "/home/dir_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/file_lower /home/file_lower.bak
rename("/home/file_lower", "/home/file_lower.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_lower /home/dir_lower.bak
rename("/home/dir_lower", "/home/dir_lower.bak") = -1 EXDEV (Invalid cross-device link)
rename: Invalid cross-device link
+++ exited with 2 +++
rootful, overlay (w/ FUSE)
$ sudo STORAGE_DRIVER=overlay STORAGE_OPTS="overlay.mount_program=/usr/bin/fuse-overlayfs" sh repro.sh
+ podman version
Version: 3.4.4
API Version: 3.4.4
Go Version: go1.17.4
Git Commit: f6526ada1025c2e3f88745ba83b8b461ca659933
Built: Thu Dec 9 13:30:40 2021
OS/Arch: linux/amd64
+ podman info --debug
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- rdma
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.0-1
path: /usr/bin/conmon
version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
cpus: 4
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: requiem
idMappings:
gidmap: null
uidmap: null
kernel: 5.16.10-arch1-1
linkmode: dynamic
logDriver: journald
memFree: 9590231040
memTotal: 16446435328
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.4.2-1
path: /usr/bin/crun
version: |-
crun version 1.4.2
commit: f6fbc8f840df1a414f31a60953ae514fa497c748
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
version: |-
slirp4netns version 1.1.12
commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.3
swapFree: 0
swapTotal: 0
uptime: 15h 43m 1.04s (Approximately 0.62 days)
plugins:
log:
- k8s-file
- none
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /usr/bin/fuse-overlayfs
Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.8.2-1
Version: |-
fusermount3 version: 3.10.5
fuse-overlayfs: version 1.8.2
FUSE library version 3.10.5
using FUSE kernel interface version 7.31
graphRoot: /var/lib/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 5
runRoot: /run/containers/storage
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 3.4.4
Built: 1639074640
BuiltTime: Thu Dec 9 13:30:40 2021
GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
GoVersion: go1.17.4
OsArch: linux/amd64
Version: 3.4.4
+ findmnt --target=/var/lib/containers/storage
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda7 ext4 rw,relatime
+ df -h /var/lib/containers/storage
Filesystem Size Used Avail Use% Mounted on
/dev/sda7 30G 25G 3.0G 90% /
+ findmnt --target=/run/containers/storage
TARGET SOURCE FSTYPE OPTIONS
/run run tmpfs rw,nosuid,nodev,relatime,mode=755,inode64
+ df -h /run/containers/storage
Filesystem Size Used Avail Use% Mounted on
run 7.7G 1.4M 7.7G 1% /run
+ podman build -t lower_layer .
STEP 1/5: FROM ubuntu:20.04
STEP 2/5: RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y strace
--> Using cache 1d1d8231dc86b776c476cb472ebed709fe8eeaadb32b579c943fca0a28de6ef3
--> 1d1d8231dc8
STEP 3/5: COPY rename /usr/bin/
--> Using cache dcbe579fc6834ecb3e11a0467b2f9d623b27b370a74276a60503168b7acb9b07
--> dcbe579fc68
STEP 4/5: RUN mkdir -p /home/dir_lower
--> Using cache 0ed6e090c37db30b97641ac52366f7da3f703b3854c3a080f133d32ca5ed0c99
--> 0ed6e090c37
STEP 5/5: RUN touch /home/file_lower
--> Using cache a8c1a0981e75312e9b8689cb75eeda62625cbd28924707b1e3b354aac4be4db8
COMMIT lower_layer
--> a8c1a0981e7
Successfully tagged localhost/lower_layer:latest
a8c1a0981e75312e9b8689cb75eeda62625cbd28924707b1e3b354aac4be4db8
Run the built container image, on overlay (via fuse-overlayfs) (rootless: false):
+ mkdir /home/dir_upper
+ touch /home/file_upper
+ ls -l /home
total 8
drwxr-xr-x 2 root root 4096 Feb 22 09:18 dir_lower
drwxr-xr-x 2 root root 4096 Feb 22 09:19 dir_upper
-rw-r--r-- 1 root root 0 Feb 22 09:18 file_lower
-rw-r--r-- 1 root root 0 Feb 22 09:19 file_upper
+ strace -e rename rename /home/file_upper /home/file_upper.bak
rename("/home/file_upper", "/home/file_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_upper /home/dir_upper.bak
rename("/home/dir_upper", "/home/dir_upper.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/file_lower /home/file_lower.bak
rename("/home/file_lower", "/home/file_lower.bak") = 0
+++ exited with 0 +++
+ strace -e rename rename /home/dir_lower /home/dir_lower.bak
rename("/home/dir_lower", "/home/dir_lower.bak") = -1 EXDEV (Invalid cross-device link)
rename: Invalid cross-device link
+++ exited with 2 +++
Output of podman version:
$ podman version
Version: 3.4.4
API Version: 3.4.4
Go Version: go1.17.4
Git Commit: f6526ada1025c2e3f88745ba83b8b461ca659933
Built: Thu Dec 9 13:30:40 2021
OS/Arch: linux/amd64
Package info:
$ pacman -Qi podman
Name : podman
Version : 3.4.4-1
Description : Tool and library for running OCI-based containers in pods
Architecture : x86_64
URL : https://github.com/containers/podman
Licenses : Apache
Groups : None
Provides : None
Depends On : cni-plugins conmon containers-common crun fuse-overlayfs iptables libdevmapper.so=1.02-64 libgpgme.so=11-64 libseccomp.so=2-64 slirp4netns
Optional Deps : apparmor: for AppArmor support
btrfs-progs: support btrfs backend devices [installed]
catatonit: --init flag support [installed]
podman-docker: for Docker-compatible CLI [installed]
Required By : podman-docker
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 72.79 MiB
Packager : David Runge <dvzrv@archlinux.org>
Build Date : Thu 09 Dec 2021 01:30:40 PM
Install Date : Tue 14 Dec 2021 11:12:10 AM
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature
|
To test out what role I got identical results this way so btrfs does not seem to have an effect:
Curiously, only rootless overlay mode, the one OP originally insisted was buggy, reports But I'm not sure why @Hi-Angel's system is passing the third column where mine is failing. What did we do differently? @Hi-Angel, would you be willing to try my Full logsrootless, vfsrootful, vfsrootless, overlay (pure kernel)rootful, overlay (pure kernel)rootless, overlay (w/ FUSE)rootful, overlay (w/ FUSE) |
|
I downloaded and ran @Hi-Angel's
with STORAGE_DRIVER=overlay 🔴STORAGE_DRIVER=overlay + fuse-overlayfs 🟢I cut down my reproduction script to run against the container image you first saw it on, and got a surprise:
|
|
The
|
|
The reason I'm here in the first place is I am trying to run but I quickly hit STORAGE_DRIVER=overlay 🔴with this in STORAGE_DRIVER=overlay + fuse-overlayfs 🟠Weirdly, this one still failed, but it fails at a different place than the previous one, hence the 🟠 instead of 🔴 . This is the first time on my system I've seen a difference between using with this in STORAGE_DRIVER=vfs 🟢with this in
So maybe I'm just out of luck. If the root of the problem is lack of I'm still stuck where I was; the only reliable workaround is: |
Not sure if this question is still relevant, but running the script on my btrfs ends up with That is with |
Thank you! That's useful confirmation. I do not know what it means yet, but it's useful. I tried running your container on an older kernel: It uses Here's the successful part of If I try to force it to not use I'm going to try your container now on my other laptop which is also running Arch, but with |
Just to let you know: I tried but wasn't able to complete this. I wish I knew what it was about that image that makes it different from the |
I'm afraid there's no Dockerfile. It was originally a ubuntu 18.04, which was modified during 1.5 years by installing various packages and executing a |
This was referenced Feb 28, 2022
Merged
Thanks for this clue! I sat down with this again last night and was able to trace it out. There's an oversight in storage.drivers.overlay that is only triggered when using
Here's how I traced this out. SymptomsI took note that my but his fails on any I went searching for why. InspectingI read up on user_namespaces(7) and I figured out with both containers running I could inspect "under their floorboards" by with by What immediately stuck out:
This last point confused me, so I used Root CauseI read in https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html that
I skimmed the rest of those docs and I still don't really understand it all, except that indeed, overlayfs has to handle But I wondered why Tracing the Codeso podman/vendor/github.com/containers/storage/drivers/overlay/overlay.go Lines 1471 to 1484 in 86a057e and, 👏 to the podman team, just below there's this comment which precisely fits the symptoms above: podman/vendor/github.com/containers/storage/drivers/overlay/overlay.go Lines 1492 to 1495 in 86a057e
and the code that implements that comment overwrites podman/vendor/github.com/containers/storage/drivers/overlay/overlay.go Lines 1521 to 1529 in 86a057e which fits the final symptom
Ta-dahSo, that was the bug. #13375 should fix it, and you should be able to use your container without |
|
This is an amazing research, thank you very much! |
|
This is sort of beating the horse dead, but I have only been working on this in my spare time, and I want to be sure. I got stumped on https://github.com/containers/podman/issues/13432 too... but anyway now I've been able to work out a minimal reproducer the replicates your bug without having to download anything, which shows that the issue really is the many layers: #!/bin/sh
#
# reproduce https://github.com/containers/podman/issues/13123
#
# When:
#
# - using the non-fuse-overlayfs overlay storage driver
# - on a large number of container layers
# - in rootless mode
#
# then:
#
# - rename()ing any directory in the container fails.
#
# Usage: rename-repro.sh [LAYERS]
# LAYERS is the number of layers to create in the container.
#
set -eu
LAYERS=${1:-50}
# The bug is in the native overlayfs backend, the one without fuse-overlayfs
export STORAGE_DRIVER=overlay STORAGE_OPTS=""
# verify configured driver is actually active: if you switch the driver you
# have to erase the entire cache of containers/images/cached config. And sometimes even twice before it will stick?
if podman info --debug 2>&1 | grep -q 'delete libpod local files to resolve'; then
echo "In order to switch graph drivers, you must erase your existing containers/images/cache."
podman system reset
if podman info --debug 2>&1 | grep -q 'delete libpod local files to resolve'; then
echo "You chose to keep your existing containers. This reproduction script cannot work. Exiting."
exit 1
fi
# sometimes the above *does not work*, so force it a second time over
GRAPHROOT=$(podman info --format={{".Store.GraphRoot"}})
rm -f "$GRAPHROOT"/libpod/* # XXX dangerous
podman system reset -f # XXX dangerous
fi
cat <<EOF |
// rename.c: a direct interface to rename(2) without all that coreutils junk in the way
#include <stdio.h>
#include <errno.h>
int main(int argc, char* argv[]) {
if(argc != 3) { errno = EINVAL; perror("argv"); return 1; }
if(rename(argv[1], argv[2]) != 0) { perror("rename"); return 2; }
return 0;
}
EOF
gcc -static -x c - -o rename
# Make a container with many layers
LAYERS=$(($LAYERS - 3)) # the test container has $LAYERS plus the base layer plus the COPY layer plus the final actual layer
(
echo "FROM alpine"
echo "COPY rename /usr/bin"
for i in `seq $LAYERS`; do
echo 'RUN dd if=/dev/urandom of=$(mktemp) count=1'
done
) | podman build -f - -t layer .
podman --log-level=debug run -it --rm layer sh -c '
mkdir A &&
rename A B
'I made a username with 3 characters in it (because https://github.com/containers/podman/issues/13432 showed me that 5 wouldn't work at the moment). A shorter or longer name will change the precise number of layers where the problem appears. With Whereas layer 49 doesn't have the problem: To see that #13375 fixed it, build that version: Then test using the fixed version: and it finishes without error. |
kbaran1998
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 16, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
Signed-off-by: Krzysztof Baran <krysbaran@gmail.com>
keonchennl
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 18, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
Signed-off-by: Krzysztof Baran <krysbaran@gmail.com>
keonchennl
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 18, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
keonchennl
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 18, 2022
Signed-off-by: Krzysztof Baran <krysbaran@gmail.com> Signed-off-by: Calin Georgescu <caling@protonmail.com> Improve the error message for usused configMaps If you run `podman play kube` on a yaml file that only contains configMaps, podman will fail with the error: Error: YAML document does not contain any supported kube kind This is not strictly true; configMaps are a supported kube kind. The problem is that configMaps aren't a standalone entity. They have to be used in a container somewhere, otherwise they don't do anything. This change adds a new message in the case when there only configMaps resources. It would be helpful if podman reported which configMaps are unused on every invocation of kube play. However, even if that feedback were added, this new error messages still helpfully explains the reason that podman is not creating any resources. [NO NEW TESTS NEEDED] Signed-off-by: Jordan Christiansen <xordspar0@gmail.com> Bump github.com/containernetworking/plugins from 1.0.1 to 1.1.0 Bumps [github.com/containernetworking/plugins](https://github.com/containernetworking/plugins) from 1.0.1 to 1.1.0. - [Release notes](https://github.com/containernetworking/plugins/releases) - [Commits](containernetworking/plugins@v1.0.1...v1.1.0) --- updated-dependencies: - dependency-name: github.com/containernetworking/plugins dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Show version of the deb package in info output Previously just showing name of the package, followed by the path repeated again (already stated on the line above) [NO NEW TESTS NEEDED] Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com> copr packaging: use generic macros for tmpfiles and modules load dirs [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org> Vendor in containers/common@main Signed-off-by: Ashley Cui <acui@redhat.com> Allow setting binarypath from Makefile Packagers for other distributions and package managers may put their helper binaries in other location prefixes. Add HELPER_BINARIES_DIR to the makefile so packagers can set the prefix when building Podman. HELPER_BINARIES_DIR will be set at link-time. Example usage: make podman-remote HELPER_BINARIES_DIR=/my/location/prefix Signed-off-by: Ashley Cui <acui@redhat.com> Clarify v2 API testing for podman vs docker clients Fixes: containers#13273 Signed-off-by: Chris Evich <cevich@redhat.com> Add podman volume mount support Fixes: containers#12768 Signed-off-by: Daniel J Walsh <dwalsh@redhat.com> Add the names flag for pod logs Fixes containers#13261 Signed-off-by: Xueyuan Chen <X.Chen-47@student.tudelft.nl> Refactor docker-py compatibility tests * Add which python client is being used to run tests, see "python client" below. * Remove redundate code from test classes * Update/Add comments to modules and classes ======================================================= test session starts ======================================================== platform linux -- Python 3.10.0, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 python client -- DockerClient rootdir: /home/jhonce/Projects/go/src/github.com/containers/podman plugins: requests-mock-1.8.0 collected 33 items test/python/docker/compat/test_containers.py ...s.............. [ 54%] test/python/docker/compat/test_images.py ............ [ 90%] test/python/docker/compat/test_system.py ... [100%] Note: Follow-up PRs will verify the test results and expand the tests. Signed-off-by: Jhon Honce <jhonce@redhat.com> Revert "use GetRuntimeDir() from c/common" This reverts commit fc5cf81. [NO NEW TESTS NEEDED] Signed-off-by: Brent Baude <bbaude@redhat.com> Use storage that better supports rootless overlayfs overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned (as of linux 5.16.0, I believe) how to support rootless users. Previously, rootless users had to use these storage.conf(5) settings: * storage.driver=vfs (aka STORAGE_DRIVER=vfs), or * storage.driver=overlay (aka STORAGE_DRIVER=overlay), storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs (aka STORAGE_OPTS=/usr/bin/fuse-overlayfs) Now that a third backend is available, setting only: * storage.driver=overlay (aka STORAGE_DRIVER=overlay) containers#13123 reported EXDEV errors during the normal operation of their container. Tracing it out, the problem turned out to be that their container was being mounted without 'userxattr'; I don't fully understand why, but mount(8) mentions this is needed for rootless users: > userxattr > > Use the "user.overlay." xattr namespace instead of "trusted.overlay.". > This is useful for unprivileged mounting of overlayfs. containers/storage#1156 found and fixed the issue in podman, and this just pulls in that via go get github.com/containers/storage@ebc90ab go mod vendor make vendor Closes containers#13123 Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca> container: workdir resolution must consider symlink if explicitly configured While resolving `workdir` we mostly create a `workdir` when `stat` fails with `ENOENT` or `ErrNotExist` however following cases are not true when user explicitly specifies a `workdir` while `running` using `--workdir` which tells `podman` to only use workdir if its exists on the container. Following configuration is implicity set with other `run` mechanism like `podman play kube` Problem with explicit `--workdir` or similar implicit config in `podman play kube` is that currently podman ignores the fact that workdir can also be a `symlink` and actual `link` could be valid. Hence following commit ensures that in such scenarios when a `workdir` is not found and we cannot create a `workdir` podman must perform a check to ensure that if `workdir` is a `symlink` and `link` is resolved successfully and resolved link is present on the container then we return as it is. Docker performs a similar behviour. Signed-off-by: Aditya R <arajan@redhat.com> vendor: bump c/storage to main/d06b0f Bump c/storage to main/d06b0f so we podman could use new `race-free` `AddNames` and `RemoveNames` api Signed-off-by: Aditya R <arajan@redhat.com> vendor: bump c/image to main/9a9cd9 Bump c/image to upstream main/9a9cd9 so podman could use new race-free code. Signed-off-by: Aditya R <arajan@redhat.com> test: add a test to verify race free concurrent/parallel builds Invoking parallel/concurrent builds from podman race against each other following behviour was fixed in containers/storage#1153 and containers/image#1480 Test verifies if following bug is fixed in new race-free API or not. Read more about this issue, see bz 2055487 for more details. More details here: containers/buildah#3794 and containers#13339 Co-authored-by: Ed Santiago <santiago@redhat.com> Signed-off-by: Aditya R <arajan@redhat.com> RELEASE_PROCESS.md: build artifacts locally The current PR process for release bump has the HEAD commit which bumps version/version.go to the form `release+1-dev`. This makes Cirrus publish release artifacts with `release+1-dev` and not `release`. For example, the msi generated at https://cirrus-ci.com/task/5403901196238848 says podman-v4.0.3-dev.msi . Building locally by checking out the released tag would generate the correct artifacts and would also be faster and more convenient. [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org> libpod: pods do not use cgroups if --cgroups=disabled do not attempt to use cgroups with pods if the cgroups are disabled. A similar check is already in place for containers. Closes: containers#13411 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Throw an error if kube yaml has duplicate ctr names Error out if the kube yaml passed to play kube has more than one container or init container with the same name. Signed-off-by: Urvashi Mohnani <umohnani@redhat.com> Move all python tests to pytest * Add configuration to add report header for python client used in tests * Move report headers into the individual test runners vs runner.sh Signed-off-by: Jhon Honce <jhonce@redhat.com> Fixes: containers#13301 ("machine rm removes the mounted socket file on macos") [NO NEW TESTS NEEDED] Signed-off-by: Thibault Gagnaux <tgagnaux@gmail.com> [CI:DOCS] RELEASE_PROCESS.md: cosmetic fix Follow up to pr#13416 Insert line breaks to get rid of the horizontal scroll bar. Resolves: containers#13416 (comment) Co-authored-by: Ashley Cui <acui@redhat.com> Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org> [CI:DOCS] DISTRO_PACKAGE.md: List the packaging changes for v4 [NO NEW TESTS NEEDED] Co-authored-by: Ashley Cui <ashleycui16@gmail.com> Co-authored-by: Valentin Rothberg <vrothberg@redhat.com> Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org> [CI:DOCS] troubleshooting.md: Improve language and fix typos Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com> Add ExitCommandDelay configuration use in API exec handler [NO NEW TESTS NEEDED] Signed-off-by: Rover van der Noort <s.r.vandernoort@student.tudelft.nl> MacOS improvements * Enable support of virtfs in Podman and darwin. At the time of this writing, it requires a special patch not yet included in upstream qemu. * Prefer to use a specially built qemu to support virtfs. The qemu is installed under libexec/podman. [NO NEW TESTS NEEDED] Signed-off-by: Brent Baude <bbaude@redhat.com> Inspect network info of a joined network namespace Closes: containers#13150 Signed-off-by: 😎 Mostafa Emami <mustafaemami@gmail.com> Move secret-verify-leak containerfile into its own Directory Secret-verify-leak is causing flakes, when running in parallel tests. This is because remote secrets are copied into the context directory to send to the API server, and secret-verify-leak is doing a COPY * and then checking if the temporary secret file ends up in the container or not. Since all the temporary files are prefixed with "podman-build-secret", this test checks if podman-build-secret is in the image. However, when run in parallel with other tests, other temporary podman-build-secrets might be in the context dir. Moving secret-verify-leak into its own directory makes sure that the context dir is used only by this one test. Also renamed Dockerfile -> Containerfile and cleaned up unused Containerfiles. Signed-off-by: Ashley Cui <acui@redhat.com> Skip flaky pprof tests pprof tests are way too flaky, and are causing problems for community contributors who don't have privs to press Re-run. There has been no activity or interest in fixing the bug, and it's not something I can fix. So, just disable the test. Signed-off-by: Ed Santiago <santiago@redhat.com> [CI:DOCS] troubleshooting.md: mention "podman unshare chown 0:0 path" * Mention the command "podman unshare chown 0:0 dir1/a" that changes file ownership to the regular user's UID and GID on the host. Co-authored-by: Tom Sweeney <tsweeney@redhat.com> Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com> machine rm -f stops and removes machine If you want to remove a running machine, you can now pass the --force/-f to podman machine rm and the machine will be stopped and removed without confirmations. Fixes: containers#13448 [NO NEW TESTS NEEDED] Signed-off-by: Brent Baude <bbaude@redhat.com> Improve agent install message to add restart instructions Signed-off-by: Jason T. Greene <jason.greene@redhat.com> Bump github.com/docker/distribution Bumps [github.com/docker/distribution](https://github.com/docker/distribution) from 2.8.0+incompatible to 2.8.1+incompatible. - [Release notes](https://github.com/docker/distribution/releases) - [Commits](distribution/distribution@v2.8.0...v2.8.1) --- updated-dependencies: - dependency-name: github.com/docker/distribution dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Set default rule at the head of device configuration The default rule should be set at the head of device configuration. Otherwise, rules for user devices are overridden by the default rule so that any access to the user devices are denied. Signed-off-by: Hironori Shiina <shiina.hironori@jp.fujitsu.com> use gopkg.in/yaml.v2 instead of v3 Many dependencies already import gopkg.in/yaml.v2, podman is the only user of the v3 version except github.com/stretchr/testify but this is only a testing dependency so it will not end up in the binary. This change reduces the podman binary size from 54740 to 54260 KB on my system. [NO NEW TESTS NEEDED] Signed-off-by: Paul Holzinger <pholzing@redhat.com> Use github.com/vbauerster/mpb/v7 in pkg/machine We already use v7 in c/image so podman should use the same version to prevent duplication. This saves 170 KB binary size. [NO NEW TESTS NEEDED] Signed-off-by: Paul Holzinger <pholzing@redhat.com> Fix handling of tmpfs-mode for tmpfs creation in compat mode The permissions on disk were wrong since we were not converting to octal. Fixes: containers#13108 [NO NEW TESTS NEEDED] Since we don't currently test using the docker client Signed-off-by: Daniel J Walsh <dwalsh@redhat.com> Add --context-dir option to podman play kube This option was requested so that users could specify alternate locations to find context directories for each image build. It requites the --build option to be set. Partion Fix: containers#12485 Signed-off-by: Daniel J Walsh <dwalsh@redhat.com> Bump github.com/vbauerster/mpb/v7 from 7.3.2 to 7.4.1 Bumps [github.com/vbauerster/mpb/v7](https://github.com/vbauerster/mpb) from 7.3.2 to 7.4.1. - [Release notes](https://github.com/vbauerster/mpb/releases) - [Commits](vbauerster/mpb@v7.3.2...v7.4.1) --- updated-dependencies: - dependency-name: github.com/vbauerster/mpb/v7 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Bump github.com/containernetworking/plugins from 1.1.0 to 1.1.1 Bumps [github.com/containernetworking/plugins](https://github.com/containernetworking/plugins) from 1.1.0 to 1.1.1. - [Release notes](https://github.com/containernetworking/plugins/releases) - [Commits](containernetworking/plugins@v1.1.0...v1.1.1) --- updated-dependencies: - dependency-name: github.com/containernetworking/plugins dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> CI: fix golangci-lint installation avoid this warn: ``` golangci/golangci-lint info installed ./bin/golangci-lint golangci/golangci-lint err this script is deprecated, please do not use it anymore. check goreleaser/godownloader#207 ``` Signed-off-by: Pascal Bourdier <pascal.bourdier@gmail.com> [CI:DOCS] Remove "(1)" from web tab text * Remove the ending text "(1)" to avoid it from being displayed in the web tab title for a command man page on the web. Often such a text indicates that a web page got an update. For instance GitHub issues shows the number of new comments that have been written after the user's last visit. Fixes containers#13438 Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com> Add podman play kube --annotation Allow users to add annotions in the podman play kube command. This PR Also fixes the fact that annotations in the pod spec were not being passed down to containers. Fixes: containers#12968 Signed-off-by: Daniel J Walsh <dwalsh@redhat.com> Bump github.com/docker/docker Bumps [github.com/docker/docker](https://github.com/docker/docker) from 20.10.12+incompatible to 20.10.13+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Changelog](https://github.com/moby/moby/blob/master/CHANGELOG.md) - [Commits](moby/moby@v20.10.12...v20.10.13) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Bump github.com/spf13/cobra from 1.3.0 to 1.4.0 Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.3.0 to 1.4.0. - [Release notes](https://github.com/spf13/cobra/releases) - [Changelog](https://github.com/spf13/cobra/blob/master/CHANGELOG.md) - [Commits](spf13/cobra@v1.3.0...v1.4.0) --- updated-dependencies: - dependency-name: github.com/spf13/cobra dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> podman.spec.rpkg: enable rhel8 builds on copr fix conditionals and buildtags to enable rhel8 builds [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org> Fixes TTY & resizing on Mac and Windows Signed-off-by: Jason T. Greene <jason.greene@redhat.com> Bump golang to 1.17 in `vendor-in-container` Go 1.17 compiler got faster Signed-off-by: Pascal Bourdier <pascal.bourdier@gmail.com> Add support for --chrootdirs Signed-off-by: LStandman <65296484+LStandman@users.noreply.github.com> Fix typo [NO NEW TESTS NEEDED] Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org> slirp: fix setup on ipv6 disabled systems When enable_ipv6=true is set for slirp4netns (default since podman v4), we will try to set the accept sysctl. This sysctl will not exist on systems that have ipv6 disabled. In this case we should not error and just ignore the extra ipv6 setup. Also the current logic to wait for the slirp4 setup was kinda broken, it did not actually wait until the sysctl was set before starting slirp. This should now be fixed by using two `sync.WaitGroup`s. [NO NEW TESTS NEEDED] Fixes containers#13388 Signed-off-by: Paul Holzinger <pholzing@redhat.com> vendor: bump buildah, c/image and c/storage Bumps c/buildah to -> `v1.24.3-0.20220310160415-5ec70bf01ea5` c/storage to -> `v1.38.3-0.20220308085612-93ce26691863` c/image to -> `v5.20.1-0.20220310094651-0d8056ee346f` Signed-off-by: Aditya R <arajan@redhat.com> docs: podman-build add --no-hosts Add newly added `--no-hosts` to build docs and document its conflicting nature with `--add-host` Signed-off-by: Aditya R <arajan@redhat.com> apply-podman-deltas: skip modified test case for --add-host which adds anomaly All podman tests in CI expects exit code 125, which might not be true since exit code from runtime is relayed as it is without any modification both in `buildah` and `podman`. Following behviour is seen when PR containers/buildah#3809 added a test here https://github.com/containers/buildah/blob/main/tests/bud.bats#L3183 which relays exit code from runtime as it is, in case of both `podman` and `buildah`. However apart from this test case no other test case was able to trigger this behviour hence marking this test as an anomaly. Since its debatable if we should override this returned error number or not hence adding a note here. Signed-off-by: Aditya R <arajan@redhat.com> test/e2e: add aardvark specific tests Co-authored-by: Brent Baude <bbaude@redhat.com> Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org> Bump github.com/spf13/cobra from 1.3.0 to 1.4.0 Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.3.0 to 1.4.0. - [Release notes](https://github.com/spf13/cobra/releases) - [Changelog](https://github.com/spf13/cobra/blob/master/CHANGELOG.md) - [Commits](spf13/cobra@v1.3.0...v1.4.0) --- updated-dependencies: - dependency-name: github.com/spf13/cobra dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Bump github.com/docker/docker Bumps [github.com/docker/docker](https://github.com/docker/docker) from 20.10.12+incompatible to 20.10.13+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Changelog](https://github.com/moby/moby/blob/master/CHANGELOG.md) - [Commits](moby/moby@v20.10.12...v20.10.13) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> fix breaking change in pkg/bindings pkg/bindings is considered stable. We cannot make changes that would break any users. If someone uses this field their code would fail to compile. Since the fix is obviously correct we will keep it but also add the old field back in to keep compatibility with old code. When both fields are set ImportArchive is preferred over ImportAchive. Fixes changes from commit 2171973 [NO NEW TESTS NEEDED] Signed-off-by: Paul Holzinger <pholzing@redhat.com> Fix windows win-sshproxy build Github no longer supports the unauthenticated git protocol, so switch to using https instead. https://github.blog/2021-09-01-improving-git-protocol-security-github/ Signed-off-by: Paul Holzinger <pholzing@redhat.com> fix empty newline in version output When podman is build without git commit information it will print a empty newline instead. This is undesirable and a regression introduced in commit 7d22cc8. To test build podman with `go build -mod=vendor -o bin/podman ./cmd/podman` and check the output of bin/podman version with and without this commit. [NO NEW TESTS NEEDED] Signed-off-by: Paul Holzinger <pholzing@redhat.com> move k8s deps into podman We only need a small part of the k8s dependencies but they are the biggest dependencies in podman by far. Moving them into podman allows us to remove the unnecessary parts. Signed-off-by: Paul Holzinger <pholzing@redhat.com> remove unneeded k8s code There is a lot of unneeded code, k8s is the by far the biggest dependency in podman. We should remove as much as possible so that we only have the stuff left that we use. This is just a quick skim over the code which removes a lot of the generated code and many packages that are now unused. I know that this will be impossible to properly review. I will try to make smaller changes in follow up work. Right now this reduces about 8 MB in binary size!!! [NO NEW TESTS NEEDED] Hopefully existing tests will catch any problems. Signed-off-by: Paul Holzinger <pholzing@redhat.com> pkg/k8s.io/...: fix lint errors Fix linting errors. We use different/stricter linters, instead of skipping these packages we should fix it. Most errors are about naming conventions, since I do not want to change the names I added the nolint comment there. I also removed some unused fields where the linter complained. Signed-off-by: Paul Holzinger <pholzing@redhat.com> pkg/k8s.io/...: remove protobuf field tags Since we do not use protobuf we can remove these field tags. This will save some KB in the final binary size. This change was automated with the following commands: find pkg/k8s.io/ -type f -name "*.go" -exec sed -i -e 's/\sprotobuf\:\".*\"//g' {} + find pkg/k8s.io/ -type f -name "*.go" -exec sed -i -e 's/\s`protobuf\:\".*\"`//g' {} + Signed-off-by: Paul Holzinger <pholzing@redhat.com> pkg/k8s.io/...: remove more unneeded files We do not use the types defined in these fields. Signed-off-by: Paul Holzinger <pholzing@redhat.com> pkg/k8s.io/api/core/v1: remove unneeded types Remove types that are not applicable for podman. This are types I do not think we need, there is definitely more that could be removed but this should be handled by someone who knows the k8s code better than me. Signed-off-by: Paul Holzinger <pholzing@redhat.com> pkg/k8s.io: add small readme with copyright notice Signed-off-by: Paul Holzinger <pholzing@redhat.com> [CI:DOCS]: Mention netavark limitations for macvlan/ipvlan drivers The example is also improved to add the --subnet option, this option is required with netavark, else you get: Error: macvlan driver needs at least one subnet specified, DHCP is not supported with netavark Signed-off-by: Clayton Craft <clayton@craftyguy.net> Exit with 0 when receiving SIGTERM * systemctl stop podman.service will now return exit code 0 * Update test framework to support JSON boolean and numeric values Signed-off-by: Jhon Honce <jhonce@redhat.com> Add test for BZ #2052697 Signed-off-by: Jhon Honce <jhonce@redhat.com> Separator is no longer prepended when prefix is empty on podman generate systemd When podman generate systemd is invoked, it previously did not check if container-prefix or pod-prefix are empty. When these are empty, the file name starts with the separator, which is hyphen by default. This results in files like '-containername.service'. The code now checks if these prefixes are empty. If they are, the filename no longer adds a separator. Instead, it uses name or ID of the container or pod. Closes containers#13272 Signed-off-by: Nirmal Patel <npate012@gmail.com> Set rawimage for containers created via play kube This commit set the containers RawImageName to default image name specified in Pod YAML, so the containers could be used via autoupdate feature, which needs the RawImageName to be set. Currently RawImageName is set only for the create/run/clone podman commands. [NO NEW TESTS NEEDED] Signed-off-by: Ondra Machacek <omachace@redhat.com> podman create: building local pause image: do not read ignore files Make sure to ignore local {container,docker}ignore files when building a local pause image. Otherwise, we may mistakenly not be able to copy catatonit into the build container. Fixes: containers#13529 Signed-off-by: Valentin Rothberg <vrothberg@redhat.com> podman machine: remove hostip from port Inside the podman machine vm we always remove the hostip from the port mapping because this should only be used on the actual host. Otherwise you run into issues when we would bind 127.0.0.1 or try to bind a host address that is not available in the VM. This was already done for cni/netavark ports and slirp4netns but not for the port bindings inside libpod which are only used as root. [NO NEW TESTS NEEDED] We still do not have machine tests! Fixes containers#13543 Signed-off-by: Paul Holzinger <pholzing@redhat.com> go fmt: use go 1.18 conditional-build syntax Signed-off-by: Valentin Rothberg <vrothberg@redhat.com> Handle incompatible machines Start in a reduced mode for recovery, warn, and provide instructions to recreate them Signed-off-by: Jason T. Greene <jason.greene@redhat.com> logformatter: link to bats sources on error We already link to ginkgo sources, now add links to bats. Ugly, because we need to hardcode containers/podman (git repo) and test/system (test file path): those can't be determined from the log results like they can in ginkgo. Also, great suggestion from @Luap99: in addition to the 'Annotated results' link which we append to the basic log, include a short summary of failures. This should help a viewer see exactly which test(s) failed, which in turn can be helpful for diagnosing known-flake or real-problem. Signed-off-by: Ed Santiago <santiago@redhat.com> build(deps): bump github.com/stretchr/testify from 1.7.0 to 1.7.1 Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.7.0 to 1.7.1. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](stretchr/testify@v1.7.0...v1.7.1) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Deduplicate between Volumes and Mounts in compat API Docker Compose v2.0 passes mount specifications in two different places: Volumes (just the destination) and Mounts (full info provided - source, destination, etc). This was causing Podman to refuse to create containers, as the destination was used twice. Deduplicate between Mounts and Volumes, preferring volumes, to resolve this. Fixes containers#11822 Signed-off-by: Matthew Heon <mheon@redhat.com> Add tests with Docker Compose v2 Add a pair of new Cirrus test suites using Compose v2 instead of Compose v1 (as is currently packaged in Fedora). They work identically, and run the same tests, as the Compose v1 tests, but with the new v2 binary instead. [NO NEW TESTS NEEDED] This adds an entire Cirrus suite... Signed-off-by: Matthew Heon <mheon@redhat.com> Set names in compose tests based on version Compose v2 uses dashes as separators instead of hyphens. This broke some tests that relied upon container names. Set the name conditionally to make it safe for both. Signed-off-by: Matthew Heon <mheon@redhat.com>
keonchennl
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 18, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
keonchennl
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 18, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
kbaran1998
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 23, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
Signed-off-by: Krzysztof Baran <krysbaran@gmail.com>
kbaran1998
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 24, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
Signed-off-by: Krzysztof Baran <krysbaran@gmail.com>
kbaran1998
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 24, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
Signed-off-by: Krzysztof Baran <krysbaran@gmail.com>
kbaran1998
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 24, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
Signed-off-by: Krzysztof Baran <krysbaran@gmail.com>
keonchennl
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 24, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
keonchennl
pushed a commit
to gcalin/podman
that referenced
this issue
Mar 29, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
gbraad
pushed a commit
to gbraad-redhat/podman
that referenced
this issue
Jul 13, 2022
overlayfs -- the kernel's version, not fuse-overlayfs -- recently learned
(as of linux 5.16.0, I believe) how to support rootless users. Previously,
rootless users had to use these storage.conf(5) settings:
* storage.driver=vfs (aka STORAGE_DRIVER=vfs), or
* storage.driver=overlay (aka STORAGE_DRIVER=overlay),
storage.options.overlay.mount_program=/usr/bin/fuse-overlayfs
(aka STORAGE_OPTS=/usr/bin/fuse-overlayfs)
Now that a third backend is available, setting only:
* storage.driver=overlay (aka STORAGE_DRIVER=overlay)
containers#13123 reported EXDEV errors
during the normal operation of their container. Tracing it out, the
problem turned out to be that their container was being mounted without
'userxattr'; I don't fully understand why, but mount(8) mentions this is
needed for rootless users:
> userxattr
>
> Use the "user.overlay." xattr namespace instead of "trusted.overlay.".
> This is useful for unprivileged mounting of overlayfs.
containers/storage#1156 found and fixed the issue
in podman, and this just pulls in that via
go get github.com/containers/storage@ebc90ab
go mod vendor
make vendor
Closes containers#13123
Signed-off-by: Nick Guenther <nick.guenther@polymtl.ca>
github-actions
bot
added
the
locked - please file new issue/PR
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Is this a BUG REPORT or FEATURE REQUEST?
/kind bug
Description
Running
apt install libreadline-devin certain container results inaptbailing out with error:This is not reproducible with
fuse-overlayfs, it is only reproducible withoverlay. Image/container required to reproduce it is docker.io/kkharlamov/bugreport-enomem image.Steps to reproduce the issue:
overlaymode, e.g. runpodman info --debug | grep fuse-overlayfsand check that there's no output.overlayrequires kernel 5.16.0 or higherpodman run --rm -it docker.io/kkharlamov/bugreport-enomem /bin/zshapt install libreadline-devDescribe the results you received:
Describe the results you expected:
No errors
Workarounds:
Create
~/.config/containers/storage.confwith content:(WARNING: this step will remove all images) execute
podman system reset -fOutput of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Yes
The text was updated successfully, but these errors were encountered: