`--userns=keep-id` causes `chown [...]: invalid argument`. #7106

valentindavid · 2020-07-28T09:24:15Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running with --userns=keep-id causes chown [...]: invalid argument.

Steps to reproduce the issue:

I tried to run podman run -ti --log-level=debug --userns=keep-id docker.io/freedesktopsdk/sdk:20.08-beta /bin/bash

Describe the results you received:

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman run -ti --log-level=debug --userns=keep-id docker.io/freedesktopsdk/sdk:20.08-beta /bin/bash) 
DEBU[0000] Ignoring libpod.conf EventsLogger setting "/sysroot/home/valentin/.config/containers/containers.conf". Use "journald" if you want to change this setting and remove libpod.conf files. 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/valentin/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/valentin/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/valentin/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/valentin/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
WARN[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] Setting parallel job count to 25             
DEBU[0000] parsed reference into "[overlay@/var/home/valentin/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/freedesktopsdk/sdk:20.08-beta" 
DEBU[0000] parsed reference into "[overlay@/var/home/valentin/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/freedesktopsdk/sdk:20.08-beta" 
DEBU[0000] parsed reference into "[overlay@/var/home/valentin/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@4f3f90e23af0b9e8bf17cdee6d5f4df1dc24d51b2328be5d9c0298bacd9cda39" 
DEBU[0000] exporting opaque data as blob "sha256:4f3f90e23af0b9e8bf17cdee6d5f4df1dc24d51b2328be5d9c0298bacd9cda39" 
DEBU[0000] parsed reference into "[overlay@/var/home/valentin/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/freedesktopsdk/sdk:20.08-beta" 
DEBU[0000] parsed reference into "[overlay@/var/home/valentin/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@4f3f90e23af0b9e8bf17cdee6d5f4df1dc24d51b2328be5d9c0298bacd9cda39" 
DEBU[0000] exporting opaque data as blob "sha256:4f3f90e23af0b9e8bf17cdee6d5f4df1dc24d51b2328be5d9c0298bacd9cda39" 
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading default seccomp profile              
DEBU[0000] Allocated lock 1 for container c19467f5ed6c29d56d116813f67a82e96189ac03868fe7cc9561a4054537e3ff 
DEBU[0000] parsed reference into "[overlay@/var/home/valentin/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@4f3f90e23af0b9e8bf17cdee6d5f4df1dc24d51b2328be5d9c0298bacd9cda39" 
DEBU[0000] exporting opaque data as blob "sha256:4f3f90e23af0b9e8bf17cdee6d5f4df1dc24d51b2328be5d9c0298bacd9cda39" 
DEBU[0000] created container "c19467f5ed6c29d56d116813f67a82e96189ac03868fe7cc9561a4054537e3ff" 
DEBU[0000] container "c19467f5ed6c29d56d116813f67a82e96189ac03868fe7cc9561a4054537e3ff" has work directory "/var/home/valentin/.local/share/containers/storage/overlay-containers/c19467f5ed6c29d56d116813f67a82e96189ac03868fe7cc9561a4054537e3ff/userdata" 
DEBU[0000] container "c19467f5ed6c29d56d116813f67a82e96189ac03868fe7cc9561a4054537e3ff" has run directory "/run/user/1000/containers/overlay-containers/c19467f5ed6c29d56d116813f67a82e96189ac03868fe7cc9561a4054537e3ff/userdata" 
Error: chown /run/user/1000/containers/overlay-containers/c19467f5ed6c29d56d116813f67a82e96189ac03868fe7cc9561a4054537e3ff/userdata: invalid argument

Describe the results you expected:

It works.

Additional information you deem important (e.g. issue happens only occasionally):

/etc/subuid:

valentin:100000:65536

Same in /etc/subgid.

strace gives this interesting line:

[pid 184717] fchownat(AT_FDCWD, "/run/user/1000/containers/overlay-containers/9662a968d3c993e1aee49087318ecbefff2d1f30ff8edbc10d5e1abe257f0398/userdata", 1, 1, 0) = -1 EINVAL (Invalid argument)

Output of podman version:

Version:      2.0.2
API Version:  1
Go Version:   go1.14.6
Built:        Thu Nov 10 16:00:00 2011
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.19, commit: unknown'
  cpus: 8
  distribution:
    distribution: org.gnome.Platform
    version: master
  eventLogger: file
  hostname: valmont
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
  kernel: 5.7.7
  linkmode: dynamic
  memFree: 720683008
  memTotal: 25180336128
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 0
  swapTotal: 0
  uptime: 14h 55m 18.06s (Approximately 0.58 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /sysroot/home/valentin/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.2
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.2
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/valentin/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /var/home/valentin/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 1320937200
  BuiltTime: Thu Nov 10 16:00:00 2011
  GitCommit: ""
  GoVersion: go1.14.6
  OsArch: linux/amd64
  Version: 2.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

See https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/a0c728f2c37db5d93361fcfae77b08aa9804d6c1/elements/vm/podman/podman.bst

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical on GNOME OS.

The text was updated successfully, but these errors were encountered:

giuseppe · 2020-07-28T10:36:52Z

it works well here. What is the output for podman unshare cat /proc/self/uid_map ?

valentindavid · 2020-07-28T11:27:46Z

it works well here. What is the output for podman unshare cat /proc/self/uid_map ?

         0       1000          1

mheon · 2020-07-28T13:01:06Z

Can you try a podman system migrate then provide the results of podman unshare cat /proc/self/uid_map again? It looks like your user namespace doesn't have the mappings that are in /etc/subuid

valentindavid · 2020-07-28T13:03:33Z

Same result.

giuseppe · 2020-07-28T13:29:34Z

yes that is definitely the issue. It looks like the user namespace is created with a single mapping.

Can you try this command:

$ unshare -U sleep 100 &
$ newuidmap $! 0 100000 65536

Do you get any error?

valentindavid · 2020-07-28T13:33:08Z

newuidmap: write to uid_map failed: Operation not permitted

It can open /proc/.../uid_map O_WRONLY, the EPERM happens on when writing.

giuseppe · 2020-07-28T13:44:05Z

it is missing the file capabilities. newuidmap should have cap_setuid and newgidmap instead cap_setgid.

Alternatively, you can set the setuid bit (chmod +s newuidmap newgidmap)

rhatdan · 2020-07-29T11:59:37Z

sudo dnf -y reinstall shadow-utils

Should fix the permissions on newuidmap and newgidmap.

valentindavid · 2020-08-06T07:16:59Z

Sorry for not replying earlier. Setting +s or cap_setuid (resp. cap_setgid) fixes the issue.

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 28, 2020

valentindavid closed this as completed Aug 6, 2020

secondspass mentioned this issue Dec 3, 2020

Failing to run MPI Podman example on HPC cluster with subuid/subgid mapping restrictions #8580

Closed

89luca89 mentioned this issue Jan 3, 2022

Can't create box 89luca89/distrobox#92

Closed

johrstrom mentioned this issue May 25, 2022

Problems running dev container OSC/ondemand#2052

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`--userns=keep-id` causes `chown [...]: invalid argument`. #7106

`--userns=keep-id` causes `chown [...]: invalid argument`. #7106

valentindavid commented Jul 28, 2020

giuseppe commented Jul 28, 2020

valentindavid commented Jul 28, 2020

mheon commented Jul 28, 2020

valentindavid commented Jul 28, 2020

giuseppe commented Jul 28, 2020

valentindavid commented Jul 28, 2020

giuseppe commented Jul 28, 2020

rhatdan commented Jul 29, 2020

valentindavid commented Aug 6, 2020

--userns=keep-id causes chown [...]: invalid argument. #7106

--userns=keep-id causes chown [...]: invalid argument. #7106

Comments

valentindavid commented Jul 28, 2020

giuseppe commented Jul 28, 2020

valentindavid commented Jul 28, 2020

mheon commented Jul 28, 2020

valentindavid commented Jul 28, 2020

giuseppe commented Jul 28, 2020

valentindavid commented Jul 28, 2020

giuseppe commented Jul 28, 2020

rhatdan commented Jul 29, 2020

valentindavid commented Aug 6, 2020

`--userns=keep-id` causes `chown [...]: invalid argument`. #7106

`--userns=keep-id` causes `chown [...]: invalid argument`. #7106