rootless podman 3.1.0-3 build fails

[adelton]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

With rootless podman 3.1.0-3, running podman build fails in the initial FROM step on GitHub Actions Ubuntu.

Steps to reproduce the issue:

1. Have Dockerfile.fedora-33 with content

FROM registry.fedoraproject.org/fedora:33
RUN dnf install -y --setopt=install_weak_deps=False httpd

2. Try to build image with rootless podman:

podman build -t local/image -f Dockerfile.fedora-33 

Describe the results you received:

STEP 1: FROM registry.fedoraproject.org/fedora:33
Getting image source signatures
Copying blob sha256:157ab801145489f145f258148bd135102a3294e420f1859a39e824e7cda56b2f
Copying config sha256:9f2a56037643a68ea81711a8eeb4501428eefd40b000c866ad9745a581c0464d
Writing manifest to image destination
Storing signatures
time="2021-04-16T11:53:03Z" level=error msg="error unmounting /home/runner/.local/share/containers/storage/overlay/781ca68fff15020c090b2b37d59bfb63e7018eeeaad718dab8e9309afece937b/merged: invalid argument"
Error: error mounting new container: error mounting build container "41bcce718d96add49663a05adae8a50558f4d4c517ee0692132e307ce8709673": error creating overlay mount to /home/runner/.local/share/containers/storage/overlay/781ca68fff15020c090b2b37d59bfb63e7018eeeaad718dab8e9309afece937b/merged, mount_data=",lowerdir=/home/runner/.local/share/containers/storage/overlay/l/YVYAFOL4CLGWMP6SYMAWEMWOMC,upperdir=/home/runner/.local/share/containers/storage/overlay/781ca68fff15020c090b2b37d59bfb63e7018eeeaad718dab8e9309afece937b/diff,workdir=/home/runner/.local/share/containers/storage/overlay/781ca68fff15020c090b2b37d59bfb63e7018eeeaad718dab8e9309afece937b/work,userxattr": invalid argument

Describe the results you expected:

No error, image built.

Additional information you deem important (e.g. issue happens only occasionally):

GitHub Actions reproducer:

• https://github.com/adelton/freeipa-container/runs/2361954818

It is a regression as it does not happen with podman 3.0.1-2:

• https://github.com/adelton/freeipa-container/actions/runs/755512636

This is a spin-off of #10049.

Output of podman version:
I did not run that command in GitHub Actions but the version is 3.1.0-3.

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.20.0
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 2
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: fv-az180-229
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 121
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.4.0-1043-azure
  linkmode: dynamic
  memFree: 5245566976
  memTotal: 7292145664
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-45fb5-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 3m 56.14s
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/runner/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /tmp/podman-run-1001/containers
  volumePath: /home/runner/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.0
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Listing...
podman/unknown,now 100:3.1.0-3 amd64 [installed]
podman/unknown 100:3.1.0-3 arm64
podman/unknown 100:3.1.0-3 armhf
podman/unknown 100:3.1.0-3 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes to the latest podman package available in the repositories; No to checking the troubleshooting guide.

Additional environment details (AWS, VirtualBox, physical, etc.):

[lsm5]

@mheon is this fixed in 3.1.1?

@adelton i'm in the process of building podman 3.1.1 along with dependencies for Kubic.

[mheon]

This seems like the c/storage issue that should have been fixed in 3.1.0 initial release?

[giuseppe]

is fuse-overlayfs installed?

[adelton]

Looking at https://github.com/adelton/freeipa-container/runs/2361954818 it says (in step Install podman and others)

fuse-overlayfs is already the newest version (1.4.0~1).
fuse-overlayfs set to manually installed.

[giuseppe]

PR here: #10115

[adelton]

GitHub Actions now upgraded podman to 3.1.0-3 in their Ubuntu environments so it's not even possible to stay with the original working 3.0.1.

What is the ETA of a new podman build (3.1.1?) which would address this problem? Or should I disable testing on rootless podman altogether?

[mheon]

@lsm5 should be pushing out a 3.1.2 build now, but it does not include that fix. We're probably going to need to cut a 3.1.3 next week.

[adelton]

For the record, podman 3.1.2-1 in https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/ seems to have fixed the problem for me. So if the fix should not have been in 3.1.2, maybe some more investigation to what the real cause was is needed.