Buildah in toolbox fails to list containers on Fedora 31 Beta #312

evelineraine · 2019-10-29T22:22:21Z

Description

Running buildah containers in freshly created (except dnf install) fedora-toolbox:31 container on Fedora Workstation 31 Beta produces a Permission Denied error.

Also there are multiple errors while to run buildah from step in the same container.

Rationale

Buildah works fully with chroot isolation in it's dedicated quay.io/buildah/stable container, or even in a generic fedora container in unprivileged rootless mode, allowing to build containers from inside a container. It makes sense it should also work in a comparatively very unconstrained toolbox container.

Steps to reproduce

$ fedora toolbox create
$ fedora toolbox enter
toolbox$ dnf install -y buildah

toolbox$ buildah containers
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
error reading build containers: error reading "/home/evelineraine/.local/share/containers/storage/overlay-containers/10d6a46bc6a7789bc944218f5e4201dbf99ace3e4c2a0d46259cf95114097c9f/userdata/buildah.json": open /home/evelineraine/.local/share/containers/storage/overlay-containers/10d6a46bc6a7789bc944218f5e4201dbf99ace3e4c2a0d46259cf95114097c9f/userdata/buildah.json: permission denied
ERRO exit status 1 

toolbox$ buildah from alpine
Getting image source signatures
Copying blob 89d9c30c1d48 done
Copying config 965ea09ff2 done
Writing manifest to image destination
Storing signatures
The following failures happened while trying to pull image specified by "alpine" based on search registries in /etc/containers/registries.conf:
* "localhost/alpine": Error initializing source docker://localhost/alpine:latest: error pinging docker registry localhost: Get https://localhost/v2/: dial tcp [::1]:443: connect: connection refused
* "docker.io/library/alpine": Error committing the finished image: error adding layer with blob "sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/shadow): lchown /etc/shadow: invalid argument
* "registry.fedoraproject.org/alpine": Error initializing source docker://registry.fedoraproject.org/alpine:latest: Error reading manifest latest in registry.fedoraproject.org/alpine: manifest unknown: manifest unknown
* "quay.io/alpine": Error initializing source docker://quay.io/alpine:latest: Error reading manifest latest in quay.io/alpine: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n"
* "registry.access.redhat.com/alpine": Error initializing source docker://registry.access.redhat.com/alpine:latest: Error reading manifest latest in registry.access.redhat.com/alpine: name unknown: Repo not found
* "registry.centos.org/alpine": Error initializing source docker://registry.centos.org/alpine:latest: Error reading manifest latest in registry.centos.org/alpine: manifest unknown: manifest unknown
ERRO exit status 1

Environment

Fact	Value
Hypervisor	VirtualBox 6.0.14 r133895
OS	Fedora 31 (Workstation Edition)
Kernel	5.3.7-301.fc31.x86_64
Host podman	1.6.2-2.fc31
Toolbox buildah	1.11.4-2.fc31.x86_64 (image-spec 1.0.1-dev, runtime-spec 1.0.1-dev)
Image	f31/fedora-toolbox:31-7
Image VCS	2823d72c9792be6c6cc0ae82d70c3f8f7d33f871

Host podman info:

host:
  BuildahVersion: 1.11.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.1-1.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.1, commit: 5e0eadedda9508810235ab878174dca1183f4013'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 2505
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2505
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 146006016
  MemTotal: 3137249280
  OCIRuntime:
    name: crun
    package: crun-0.10.2-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.10.2
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  eventlogger: journald
  hostname: fossil.raine.ai
  kernel: 5.3.7-301.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 25m 18.99s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/evelineraine/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.6.5-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.6.5
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/evelineraine/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /run/user/2505
  VolumePath: /home/evelineraine/.local/share/containers/storage/volumes

Toolbox buildah info:

{
    "host": {
        "CgroupVersion": "v2",
        "Distribution": {
            "distribution": "fedora",
            "version": "31"
        },
        "MemTotal": 3137249280,
        "MenFree": 96022528,
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 2,
        "hostname": "toolbox",
        "kernel": "5.3.7-301.fc31.x86_64",
        "os": "linux",
        "rootless": true,
        "uptime": "26m 59.65s"
    },
    "store": {
        "ContainerStore": {
            "number": 1
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.mount_program=/usr/bin/fuse-overlayfs"
        ],
        "GraphRoot": "/home/evelineraine/.local/share/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "btrfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 1
        },
        "RunRoot": "/run/user/2505"
    }
}

Toolbox container info: inspect.txt

The text was updated successfully, but these errors were encountered:

debarshiray · 2020-01-08T17:12:48Z

Umm... this can be addressed by #145

Or did you mean something else?

evelineraine · 2020-01-11T11:58:30Z

No, I don't think it's the same.

Buildah (in chroot mode, like I'm running it) is able to build containers even from inside unprivileged containers. So, a toolbox container should have everything for Buildah to work without a shim binary.

I think there is an issue with access permissions to ~/.local/share/containers/storage from inside the container, since unlike in an ordinary unprivileged container, it's mounted from the host.
And it's not a SELinux issue - putting it into permissive mode doesn't help.

debarshiray · 2020-01-13T14:57:40Z

Ok. Looking closely at the error messages, and based on what you wrote, this one stands out:

* "docker.io/library/alpine": Error committing the finished image: error adding layer with blob "sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/shadow): lchown /etc/shadow: invalid argument

This has to do with the user and group IDs available in the Toolbox container's namespace, plus the fact that $HOME is shared with the host. I don't know if there's an easy and generic way to fix this that doesn't involve tunnelling the buildah invocation on the host.

debarshiray · 2020-12-21T14:06:56Z

Duplicate of #145

HarryMichal added 1. Bug Something isn't working 5. Good First Issue Good for newcomers labels Nov 7, 2019

swick mentioned this issue May 23, 2020

buildah commit fails with 'permission denied' #455

Closed

HarryMichal added this to Needs triage in Priority Board Jul 28, 2020

HarryMichal removed the 5. Good First Issue Good for newcomers label Sep 9, 2020

debarshiray marked this as a duplicate of #145 Dec 21, 2020

debarshiray closed this as completed Dec 21, 2020

Priority Board automation moved this from Needs triage to Closed Dec 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Buildah in toolbox fails to list containers on Fedora 31 Beta #312

Buildah in toolbox fails to list containers on Fedora 31 Beta #312

evelineraine commented Oct 29, 2019

debarshiray commented Jan 8, 2020

evelineraine commented Jan 11, 2020

debarshiray commented Jan 13, 2020

debarshiray commented Dec 21, 2020

Buildah in toolbox fails to list containers on Fedora 31 Beta #312

Buildah in toolbox fails to list containers on Fedora 31 Beta #312

Comments

evelineraine commented Oct 29, 2019

Description

Rationale

Steps to reproduce

Environment

debarshiray commented Jan 8, 2020

evelineraine commented Jan 11, 2020

debarshiray commented Jan 13, 2020

debarshiray commented Dec 21, 2020