Fedora gating test failures

[lsm5]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Fedora gating tests are currently failing.

Steps to reproduce the issue:

1. /usr/bin/podman run --uidmap 0:10000:10000 quay.io/libpod/testimage:20200902 true

2. Complete log at https://paste.centos.org/view/e6f829d2

Describe the results you received:

$ /usr/bin/podman run --uidmap 0:10000:10000 quay.io/libpod/testimage:20200902 true
Error: write to /proc/sys/net/ipv4/ping_group_range: Invalid argument: OCI runtime error

Describe the results you expected:

No error (I guess)

Additional information you deem important (e.g. issue happens only occasionally):

Consistent

Output of podman version:

Version:      2.1.0
API Version:  2.0.0
Go Version:   go1.15
Built:        Tue Sep 22 13:56:54 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: hashirama.nanadai.me
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.10-300.fc33.x86_64
  linkmode: dynamic
  memFree: 20476694528
  memTotal: 33087016960
  ociRuntime:
    name: crun
    package: crun-0.14.1-5.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.4+dev
      commit: eecccdb96f587b11d7764556ffacfeaffe4b6e11
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 16610488320
  swapTotal: 16610488320
  uptime: 17h 18m 38.16s (Approximately 0.71 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/lsm5/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 0
    stopped: 9
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/lsm5/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/lsm5/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1600797414
  BuiltTime: Tue Sep 22 13:56:54 2020
  GitCommit: ""
  GoVersion: go1.15
  OsArch: linux/amd64
  Version: 2.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.0-1.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes, happens on upstream master as well.

Additional environment details (AWS, VirtualBox, physical, etc.):

local f33 with btrfs, but happens on fedora's gating infra as well.

[vrothberg]

@rhatdan that's caused by

default_sysctls = [
 "net.ipv4.ping_group_range=0 65536",
]

in contianers.conf. Commenting out solves the issue on my end.

[rhatdan]

This is not an issue with the sysctls, but more of an issue with usernamespace support for them. Could be a kernel issue or in our code.
@giuseppe WDYT?

[rhatdan]

Ok Now I see it. The issue is that we have asked for a range of UIDs within the container that is not available.

I could change the default to be just one, which would be the safest, or could allocate 1000. The goal of this sysctl is just to allow non root users to be able to ping.

[rhatdan]

This should be a fedora bug not an issue.

[vrothberg]

Can you elaborate?

[rhatdan]

containers.conf needs to be changed that is shipped with Fedora. To not use 65k range.

[rhatdan]

This is basically a containers-common bug

[vrothberg]

Agreed.

[rhatdan]

https://koji.fedoraproject.org/koji/taskinfo?taskID=52228191

[lsm5]

https://koji.fedoraproject.org/koji/taskinfo?taskID=52228191

so would it suffice to have podman depend on containers-common from ^ or do we need any other change? I think we should add this skopeo build to the same podman bodhi.

[debarshiray]

Fly-by comment:

default_sysctls = [
 "net.ipv4.ping_group_range=0 65536",
]

in contianers.conf. Commenting out solves the issue on my end.

I just wanted to point out that net.ipv4.ping_group_range is already set by systemd in /usr/lib/sysctl.d/50-default.conf.

[rhatdan]

Sure but that does not get inherited by containers in new network namespaces.

[rhatdan]

-net.ipv4.ping_group_range = 0 2147483647

Would definitely not work within a user namespace.

I wonder if we should hack out the default SYSCTLS to allow the size to be customizable.

default_sysctls = [
 "net.ipv4.ping_group_range=0 MAXGID",
]

And then this would be set based on the user namespace of the container. Then it would always work

[debarshiray]

Sure but that does not get inherited by containers in new network
namespaces.

Ok, understood.