Build always fails: PermissionError: [Errno 13] Permission denied and alike

[nikonakoneko]

OS: Gentoo hardened amd64
Kubler version: 0.9.8

Build isn't always failing at the same stage. Sometime was when trying to download to /distfiles and sometime when writing to /var/tmp/portage. From my last try from scratch the error was: "portage.exception.PermissionDenied: [Errno 13] Permission denied: b'/var/tmp/portage/virtual/libcrypt-1-r1/.ipc/lock'"

I tried deleting everything (kubler clean -N; podman rm -a; podman rmi -a; rm -rf ~/.local/share/containers ~/.kubler) and start from scratch a few times.

I also tried to set BOB_FEATURES="-userfetch -userpriv"

podman info:

host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: app-containers/conmon-2.0.30
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: v2.0.30'
  cpuUtilization:
    idlePercent: 94.52
    systemPercent: 1.32
    userPercent: 4.16
  cpus: 20
  distribution:
    distribution: gentoo
    version: "2.8"
  eventLogger: file
  hostname: desktop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1000000
      size: 1000000000
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1000000
      size: 1000000000
  kernel: 5.10.76-gentoo-r1-x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 2456059904
  memTotal: 49300357120
  networkBackend: cni
  ociRuntime:
    name: crun
    package: app-containers/crun-1.4.4
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: app-containers/slirp4netns-1.2.0
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 513h 48m 39.92s (Approximately 21.38 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
    PullFromMirror: ""
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  search:
  - docker.io
  - registry.fedoraproject.org
store:
  configFile: /home/andrea/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/andrea/.local/share/containers/storage
  graphRootAllocated: 1992864915456
  graphRootUsed: 1618197934080
  graphStatus: {}
  imageCopyTmpDir: /tmp/.private/andrea
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  volumePath: /home/andrea/.local/share/containers/storage/volumes
version:
  APIVersion: 4.1.0
  Built: 1658075888
  BuiltTime: Sun Jul 17 18:38:08 2022
  GitCommit: e4b03902052294d4f342a185bb54702ed5bed8b1
  GoVersion: go1.18.3
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.0

[edannenberg]

Thanks for the report! Hmm at first glance, this seems to podman related, could you give it a try with Docker to narrow the issue down?

[nikonakoneko]

It's also happening with docker, I didn't have it I freshly installed it and I'm using default configuration.

Now I just saw that it tries to build with userpriv usersandbox features enabled. There's another issue talking about that iirc. I tried with BOB_FEATURES and FEATURES in my kubler.conf to disable these features but it's not changing anything

[nikonakoneko]

Confirmed. If I use interactive build mode and disable userpriv usersandbox in make.conf it works.

How can I disable them for every image build?

[edannenberg]

Hmm odd, I'm planning to do the monthly rebuild this Friday, let's see if I can replicate this.

Modifying BOB_FEATURES should be enough to unset userpriv and usersandbox. See man make.conf for all possible options.

[nikonakoneko]

I talked to fast. I tried again, I set -userpriv -usersandbox and it fails after doing kubler clean -N ; sudo rm -rf ~/.kubler ~/.local/share/containers ; kubler update && kubler build experiments/minimal with both docker and podman.... I try once again and send logs

[nikonakoneko]

docker info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 21
 Server Version: 20.10.12
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: f46b6ba2c9314cfc8caae24a32ec5fe9ef1059fe
 init version: de40ad007797e0dcd8b7126f27bb87401d224240
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.76-gentoo-r1-x86_64
 Operating System: Gentoo Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 45.91GiB
 Name: desktop
 ID: 236Q:XUCG:2OPI:OPOI:QEFX:UOCA:5HRC:ANUE:5TMX:JNY2:3SJT:KIQX
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

build.log

»»» jue 28 jul 2022 10:29:24 CEST »»» exec: run_image kubler/bob-musl:20220728 experiments/minimal false rootfs-builder-experiments-minimal-24563-10689
�]0;emerge�!!! It seems /run is not mounted. Process management may malfunction.
�]0;@experiments-minimal:/��]0;emerge�!!! It seems /run is not mounted. Process management may malfunction.
�]0;@experiments-minimal:/��]0;emerge�!!! It seems /run is not mounted. Process management may malfunction.

�[33;01m * IMPORTANT:�[39;49;00m 4 news items need reading for repository 'gentoo'.
�[33;01m *�[39;49;00m Use �[32;01meselect news read�[39;49;00m to view new items.


�[32mThese are the packages that would be merged, in order:�[39;49;00m

Calculating dependencies  �� -�� |�� /�� \�� \�� \�� -�� \�� -�� \�� \�� /�� |�� \�� \�� |�� /�� /�� -�� /�� \�� |�� \�� \�� |�� |�� /�� \�� -�� -�� |�� /�� \��... done!
[�[32mebuild�[39;49;00m  �[32;01mN�[39;49;00m     ] �[32msys-libs/musl-1.2.3::gentoo�[39;49;00m �[32mto /emerge-root/�[39;49;00m USE="�[34;01m-headers-only�[39;49;00m �[34;01m-verify-sig�[39;49;00m" 1060 KiB
[�[32mebuild�[39;49;00m   �[33;01mR�[39;49;00m    ] �[32mvirtual/libcrypt-1-r1:0/1::gentoo�[39;49;00m  USE="�[32;01mstatic-libs�[39;49;00m*" 0 KiB
[�[32mebuild�[39;49;00m  �[32;01mN�[39;49;00m     ] �[32mvirtual/libcrypt-1-r1:0/1::gentoo�[39;49;00m �[32mto /emerge-root/�[39;49;00m USE="�[31;01mstatic-libs�[39;49;00m" 0 KiB
[�[32;01mebuild�[39;49;00m  �[32;01mN�[39;49;00m     ] �[32;01msys-apps/busybox-1.34.1::gentoo�[39;49;00m �[32mto /emerge-root/�[39;49;00m USE="�[31;01mmake-symlinks�[39;49;00m �[31;01mstatic�[39;49;00m �[34;01m-debug�[39;49;00m �[34;01m-ipv6�[39;49;00m �[34;01m-livecd�[39;49;00m �[34;01m-math�[39;49;00m �[34;01m-mdev�[39;49;00m �[34;01m-pam�[39;49;00m �[34;01m-savedconfig�[39;49;00m (�[34;01m-selinux�[39;49;00m) �[34;01m-sep-usr�[39;49;00m �[34;01m-syslog�[39;49;00m (�[34;01m-systemd�[39;49;00m)" 2419 KiB

Total: 4 packages (3 new, 1 reinstall), Size of downloads: 3478 KiB

>>> Verifying ebuild manifests�E>>> Jobs: �[32m0�[39;49;00m of �[32m4�[39;49;00m complete                           Load avg: 25.0, 26.8, 25.8�]0;experiments-minimal: Jobs: 0 of 4 complete Load avg: 25.0, 26.8, 25.8�
�[K>>> Jobs: �[32m0�[39;49;00m of �[32m4�[39;49;00m complete, �[32m1�[39;49;00m running                Load avg: 25.0, 26.8, 25.8�]0;experiments-minimal: Jobs: 0 of 4 complete, 1 running Load avg: 25.0, 26.8, 25.8�
�[K>>> Emerging (�[33;01m1�[39;49;00m of �[33;01m4�[39;49;00m) �[32msys-libs/musl-1.2.3::gentoo�[39;49;00m for /emerge-root/�E>>> Jobs: �[32m0�[39;49;00m of �[32m4�[39;49;00m complete, �[32m1�[39;49;00m running                Load avg: 25.0, 26.8, 25.8�]0;experiments-minimal: Jobs: 0 of 4 complete, 1 running Load avg: 25.0, 26.8, 25.8�
�[K>>> �[31;01mFailed�[39;49;00m to emerge �[32msys-libs/musl-1.2.3�[39;49;00m for /emerge-root/, Log file:�E>>> Jobs: �[32m0�[39;49;00m of �[32m4�[39;49;00m complete, �[32m1�[39;49;00m running                Load avg: 25.0, 26.8, 25.8�]0;experiments-minimal: Jobs: 0 of 4 complete, 1 running Load avg: 25.0, 26.8, 25.8�
�[K>>>  '�[32m/var/tmp/portage/sys-libs/musl-1.2.3/temp/build.log�[39;49;00m'�E>>> Jobs: �[32m0�[39;49;00m of �[32m4�[39;49;00m complete, �[32m1�[39;49;00m running                Load avg: 25.0, 26.8, 25.8�]0;experiments-minimal: Jobs: 0 of 4 complete, 1 running Load avg: 25.0, 26.8, 25.8�
�[K>>> Jobs: �[32m0�[39;49;00m of �[32m4�[39;49;00m complete, �[32m1�[39;49;00m running, �[32m1�[39;49;00m failed      Load avg: 25.0, 26.8, 25.8�]0;experiments-minimal: Jobs: 0 of 4 complete, 1 running, 1 failed Load avg: 25.0, 26.8, 25.8�
�[K>>> Jobs: �[32m0�[39;49;00m of �[32m4�[39;49;00m complete, �[32m1�[39;49;00m failed                 Load avg: 25.0, 26.8, 25.8�]0;experiments-minimal: Jobs: 0 of 4 complete, 1 failed Load avg: 25.0, 26.8, 25.8��Ebash: line 1: /distfiles/.__portage_test_write__: Permission denied
!!! No write access to '/distfiles'
!!! No write access to '/distfiles'
!!! File .layout.conf.ftp.snt.utwente.nl isn't fetched but unable to get it.
!!! File musl-1.2.3.tar.gz isn't fetched but unable to get it.
�[31;01m * �[39;49;00mFetch failed for 'sys-libs/musl-1.2.3', Log file:
�[31;01m * �[39;49;00m '/var/tmp/portage/sys-libs/musl-1.2.3/temp/build.log'

�[32m * �[39;49;00mMessages for package �[32msys-libs/musl-1.2.3�[39;49;00m merged to /emerge-root/:

�[31;01m * �[39;49;00mFetch failed for 'sys-libs/musl-1.2.3', Log file:
�[31;01m * �[39;49;00m '/var/tmp/portage/sys-libs/musl-1.2.3/temp/build.log'
�]0;@experiments-minimal:/��[33m»[�[31m✘�[33m]»�(B�[m�[33m[�(B�[mexperiments/minimal�[33m]»�(B�[m fatal: Failed to run image kubler/bob-musl:20220728
�

Files in kubler's distfiles are owned by root:portage

[edannenberg]

Did you revert the userpriv and usersandbox changes? My distfiles folder looks like this:

drwxrwxr-x  3 ed portage 132K Jun 28 18:16 distfiles 

As it has write permissions for the group, portage has no issue downloading stuff. All files are owned by portage:portage in the folder. Can you double check the write permission for the folder?

[nikonakoneko]

portage group has write permission to ~/.kubler/distfiles directory and files inside it. I didn't change features. I even have them twice unset

grep -H userpri /etc/kubler.conf experiments/images/minimal/build.sh /etc/kubler.conf:BOB_FEATURES="${BOB_FEATURES:--parallel-fetch nodoc noinfo noman binpkg-multi-instance -ipc-sandbox -network-sandbox -pid-sandbox -userpriv -usersandbox}"
experiments/images/minimal/build.sh:    echo 'FEATURES="-userpriv -usersandbox"' >> /etc/portage/make.conf

[edannenberg]

Ok, so portage should be running as root but can't write anyways. Do you have some extra hardening on the host that might prevent docker/podman from writing to a host mount?

[nikonakoneko]

I'm using a gentoo hardened profile, but afaik i didnt change anything from defaults for security related config. SELinux is disabled, and I don't know what else I could have nor how i could debug it

[edannenberg]

Hmm let's try to narrow it down:

docker run -it --rm -v /path/to/distfiles:/distfiles busybox
# echo test > /distfiles/foo.txt

If that fails there is most likely some host related issue.

[nikonakoneko]

Hmm let's try to narrow it down:

docker run -it --rm -v /path/to/distfiles:/distfiles busybox
# echo test > /distfiles/foo.txt

If that fails there is most likely some host related issue.

It's working fine

[edannenberg]

Ok progess. :)

If I use interactive build mode and disable userpriv usersandbox in make.conf it works.

Let's check how the permissions for /distfiles look from inside the interactive build container.

[nikonakoneko]

kubler clean -N ; sudo rm -rf ~/.kubler ~/.local/share/containers ; kubler update && kubler build -i experiments/minimal

kubler-bob-musl / # ls -la /distfiles/
total 174684
drwxrwxr-x+  1 1000 portage      1052 Jul 28 15:57 .
drwxr-xr-x  24 root root            0 Jul 28 18:46 ..
-rw-rw-r--+  1 root portage        45 Nov  5  2019 .layout.conf.ftp.snt.utwente.nl
-rw-rw-r--+  1 root portage       119 Jul 28 07:22 .mirror-cache.json
-rw-rw-r--+  1 root portage    158456 Mar  8  2017 UnicodeData-10.0.0.txt.xz
-rw-rw-r--+  1 root portage    311004 Jul 25  2020 bash-completion-2.11.tar.xz
-rw-rw-r--+  1 root portage      3539 May 25  2019 bashcomp-2.0.3.tar.gz
-rw-rw-r--+  1 root portage   2105561 May 18 06:52 cython-0.29.30.gh.tar.gz
-rw-rw-r--+  1 root portage    639864 Jun  4 09:58 eix-0.36.3.tar.xz
-rw-rw-r--+  1 root portage      8543 Jan 13  2022 eselect-repository-12.tar.gz
-rw-rw-r--+  1 root portage     16767 May 24  2013 flaggie-0.2.1.tar.bz2
-rw-rw-r--+  1 root root            5 Jul 28 15:57 foo.txt
-rw-rw-r--+  1 root portage     21508 Feb 10  2019 gentoo-bashcomp-20190211.tar.bz2
-rw-rw-r--+  1 root portage   3203805 Mar  2  2021 gentoolkit-0.5.1.tar.gz
-rw-rw-r--+  1 root portage   6874520 Jan 29 01:46 git-2.35.1.tar.xz
-rw-rw-r--+  1 root portage    497284 Jan 29 01:46 git-manpages-2.35.1.tar.xz
-rw-rw-r--+  1 root portage 125758119 Jul 23  2021 go-linux-amd64-bootstrap-1.16.6.tbz
-rw-rw-r--+  1 root portage  22845866 Jul 12 19:40 go1.18.4.src.tar.gz
-rw-rw-r--+  1 root portage   1181867 Nov 10  2020 jq-1.7_pre20201109.tar.gz
-rw-rw-r--+  1 root portage    960663 Jul  2 05:52 lxml-4.9.1.gh.tar.gz
-rw-rw-r--+  1 root portage   1585293 Nov 17  2010 miscfiles-1.5.tar.gz
-rw-rw-r--+  1 root portage    944148 Apr 29 04:51 onig-6.9.8.tar.gz
-rw-rw-r--+  1 root portage   1820282 Feb 23 11:37 openssh-8.9p1.tar.gz
-rw-rw-r--+  1 root portage   9864061 Jul  5 10:09 openssl-1.1.1q.tar.gz
-rw-rw-r--+  1 root portage      2839 Sep  7  2020 push-3.4.tar.gz
-rw-rw-r--+  1 root portage     11128 Aug  9  2020 quoter-4.2.tar.gz

[edannenberg]

Sorry for the delay, I hope you could resolve the issue, it looked liked something specific to your setup as I couldn't replicate the problem. Feel free to reopen if you still need help with this.