No space left on device while building 3.26Gb image with 11Gb space available

[abitrolly]

Description

Can't build the image on my local machine. https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/main/support/gitpod

Steps to reproduce the issue:

1. git clone https://gitlab.com/gitlab-org/gitlab-development-kit
2. cd gitlab-development-kit
3. podman build -f Dockerfile -t gp . --logfile build.log

Describe the results you received:

Error: committing container for step {Env:[GIT_CHECKOUT_BRANCH=main PATH=/home/gitpod/.local/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=en_US.UTF-8 HOME=/home/gitpod TRIGGER_REBUILD=1] Command:run Args:[bash /tmp/bootstrap-gitpod-image.sh] Flags:[] Attrs:map[] Message:RUN bash /tmp/bootstrap-gitpod-image.sh Original:RUN bash /tmp/bootstrap-gitpod-image.sh}: copying layers and metadata for container "6e282120551db839cf0a7f0bb68e26e855126eb49f7c08191c5b4021d557f5b6": writing blob: storing blob to file "/var/tmp/storage3185272525/1": write /var/tmp/storage3185272525/1: no space left on device

Describe the results you expected:

I expected podman to report what it is doing, why there is not enough space. What takes that space and how to fix that.

Output of rpm -q buildah or apt list buildah:

$ rpm -q buildah
buildah-1.28.0-6.fc37.x86_64 

Output of buildah version:

$ buildah version
Version:         1.28.0
Go Version:      go1.19.2
Image Spec:      1.0.2-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.23.0
Git Commit:      
Built:           Tue Oct 25 17:14:21 2022
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of podman version if reporting a podman build issue:

$ podman version
Client:       Podman Engine
Version:      4.3.0
API Version:  4.3.0
Go Version:   go1.19.2
Built:        Fri Oct 21 11:09:51 2022
OS/Arch:      linux/amd64 

Output of cat /etc/*release:

$ cat /etc/*release
Fedora release 37 (Thirty Seven)
NAME="Fedora Linux"
VERSION="37 (Workstation Edition)"
ID=fedora
VERSION_ID=37
VERSION_CODENAME=""
PLATFORM_ID="platform:f37"
PRETTY_NAME="Fedora Linux 37 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:37"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f37/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=37
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=37
VARIANT="Workstation Edition"
VARIANT_ID=workstation
Fedora release 37 (Thirty Seven)
Fedora release 37 (Thirty Seven)

Output of uname -a:

$ uname -a
Linux blackred 6.0.8-300.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 11 15:09:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# This file is is the configuration file for all tools
# that use the containers/storage library. The storage.conf file
# overrides all other storage.conf files. Container engines using the
# container/storage library do not inherit fields from other storage.conf
# files.
#
#  Note: The storage.conf file overrides other storage.conf files based on this precedence:
#      /usr/containers/storage.conf
#      /etc/containers/storage.conf
#      $HOME/.config/containers/storage.conf
#      $XDG_CONFIG_HOME/containers/storage.conf (If XDG_CONFIG_HOME is set)
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver, Must be set for proper operation.
driver = "overlay"

# Temporary storage location
runroot = "/run/containers/storage"

# Primary Read/Write location of container storage
# When changing the graphroot location on an SELINUX system, you must
# ensure  the labeling matches the default locations labels with the
# following commands:
# semanage fcontext -a -e /var/lib/containers/storage /NEWSTORAGEPATH
# restorecon -R -v /NEWSTORAGEPATH
graphroot = "/var/lib/containers/storage"


# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to the UIDs/GIDs as they should appear outside of the container,
# and the length of the range of UIDs/GIDs.  Additional mapped sets can be
# listed and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and then a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped in-container ID,
# until all of the entries have been used for maps.
#
# remap-user = "containers"
# remap-group = "containers"

# Root-auto-userns-user is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid and /etc/subgid file.  These ranges will be partitioned
# to containers configured to create automatically a user namespace.  Containers
# configured to automatically create a user namespace can still overlap with containers
# having an explicit mapping set.
# This setting is ignored when running as rootless.
# root-auto-userns-user = "storage"
#
# Auto-userns-min-size is the minimum size for a user namespace created automatically.
# auto-userns-min-size=1024
#
# Auto-userns-max-size is the minimum size for a user namespace created automatically.
# auto-userns-max-size=65536

[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids.  Note multiple UIDs will be
# squashed down to the default uid in the container.  These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
#ignore_chown_errors = "false"

# Inodes is used to set a maximum inodes of the container image.
# inodes = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

# Set to skip a PRIVATE bind mount on the storage home directory.
# skip_mount_home = "false"

# Size is used to set a maximum size of the container image.
# size = ""

# ForceMask specifies the permissions mask that is used for new files and
# directories.
#
# The values "shared" and "private" are accepted.
# Octal permission masks are also accepted.
#
#  "": No value specified.
#     All files/directories, get set with the permissions identified within the
#     image.
#  "private": it is equivalent to 0700.
#     All files/directories get set with 0700 permissions.  The owner has rwx
#     access to the files. No other users on the system can access the files.
#     This setting could be used with networked based homedirs.
#  "shared": it is equivalent to 0755.
#     The owner has rwx access to the files and everyone else can read, access
#     and execute them. This setting is useful for sharing containers storage
#     with other users.  For instance have a storage owned by root but shared
#     to rootless users as an additional store.
#     NOTE:  All files within the image are made readable and executable by any
#     user on the system. Even /etc/shadow within your image is now readable by
#     any user.
#
#   OCTAL: Users can experiment with other OCTAL Permissions.
#
#  Note: The force_mask Flag is an experimental feature, it could change in the
#  future.  When "force_mask" is set the original permission mask is stored in
#  the "user.containers.override_stat" xattr and the "mount_program" option must
#  be specified. Mount programs like "/usr/bin/fuse-overlayfs" present the
#  extended attribute permissions to processes within containers rather then the
#  "force_mask"  permissions.
#
# force_mask = ""

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# metadata_size is used to set the `pvcreate --metadatasize` options when
# creating thin devices. Default is 128k
# metadata_size = ""

# Size is used to set a maximum size of the container image.
# size = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

[flouthoc]

Could you please share podman info and details if you are running this on a macOS or linux.

[abitrolly]

@flouthoc there is output of cat /etc/*release above and uname -a that shows it is Linux.

podman info

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.4-3.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.4, commit: '
  cpuUtilization:
    idlePercent: 77.74
    systemPercent: 5.46
    userPercent: 16.8
  cpus: 4
  distribution:
    distribution: fedora
    variant: workstation
    version: "37"
  eventLogger: journald
  hostname: blackred
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.0.8-300.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 490893312
  memTotal: 8020430848
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.7-1.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.7
      commit: 40d996ea8a827981895ce22886a9bac367f87264
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 13805871104
  swapTotal: 16216219648
  uptime: 66h 14m 34.00s (Approximately 2.75 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/anatoli/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/anatoli/.local/share/containers/storage
  graphRootAllocated: 188932792320
  graphRootUsed: 116227137536
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 29
  runRoot: /run/user/1000/containers
  volumePath: /home/anatoli/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.0
  Built: 1666339791
  BuiltTime: Fri Oct 21 11:09:51 2022
  GitCommit: ""
  GoVersion: go1.19.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.0

[rhatdan]

The issue is the space on /var/tmp. Are you sure there is a lot of space on /var/tmp?

$ df /var/tmp/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/luks-1d46fdff-f599-4589-80c6-ff54591a6a44 209596424 16548700 193047724 8% /var

[abitrolly]

@rhatdan

$ df /var/tmp/ 
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sdb3       51290592 37341852  11310916  77% /

[rhatdan]

That does not look like it is enough space to store the blobs for the 3.26Gb files.
You can set TMPDIR environment variable to a different directory to store the blobs, with more space.

[abitrolly]

That does not look like it is enough space to store the blobs for the 3.26Gb files.

How come that 11Gb is not enough for podman to store layers for 3.26Gb image?

[rhatdan]

I don't know, but is the 3.26Gb image compressed? Perhaps it is getting uncompressed causing the system to run out of space.
@vrothberg @mtrmac @nalind WDYT

[abitrolly]

@rhatdan I dk. Can you tell? registry.gitlab.com/gitlab-org/gitlab-development-kit/gitpod-workspace:gitpod-cleanup

In either way, adding a layer on top, should not result in uncompressing all layers and leaving them on disk, if I get the Docker model right.

[vrothberg]

It could very well be hitting containers/image#1187.

When pulling an image, podman first pulls the layers, then commits them to the storage. Once committed, the downloaded compressed data is removed.

[abitrolly]

@vrothberg is it possible to add debug level logging to trace that? I patched Google's crane for decompressing images for listing files on the fly, so I might be able to fix this one.

My idea is that committing layers as soon as they are ready and cleaning up immediately should same space. Here is the main issue for a bit of context https://gitlab.com/gitlab-org/gitlab-development-kit/-/merge_requests/2782

[vrothberg]

@vrothberg is it possible to add debug level logging to trace that?

You can add the --log-level=debug option but I don't see logs for that specific part.

My idea is that committing layers as soon as they are ready and cleaning up immediately should same space. Here is the main issue for a bit of context https://gitlab.com/gitlab-org/gitlab-development-kit/-/merge_requests/2782

Committing as soon as possible already happens but the downloaded compressed data isn't removed as soon as possible but after committing the image.

[abitrolly]

@vrothberg if the data is compressed, how it can take 11Gb is the final image is 3.70Gb max?

[vrothberg]

The compressed data is downloaded and then gets uncompressed for storing it in the local storage. I do not have another theory that may explain the observation.

[klautcomputing]

I am kind of running into this too:

writing blob: storing blob to file "/run/user/1000/storage4017465812/1": write /run/user/1000/storage4017465812/1: no space left on device

I have 1.6G for /run/user/1000 and watched it fill up while it was building until it crashed.
tmpfs 1.6G 396K 1.6G 1% /run/user/1000
But I have way more than enough space on my drive in general:
/dev/dm-2 460G 322G 115G 74% /
Is there a way to tell it where to put the intermediate objects during a build?

[klautcomputing]

Oh FYI I am running NixOS and I just "fixed" this by cranking up the size on the tmpfs by adding this to my services section:

    logind.extraConfig = ''
      RuntimeDirectorySize=6G
    '';

[abitrolly]

@vrothberg why store it uncompressed? Is it possible to add debug info to prove that?

[neuberfran]

@vrothberg @mtrmac @nalindWDYT @rhatdan

I have this installed on my rpi4 1gb:
https://github.com/neuberfran/seadog-gaia
I have alias podman/docker here.

When I run this:
docker run --rm -it --privileged debian:sid
I have issue:

🐶 seadog @bcm2711-rpi-4b~: docker run --rm -it --privileged debian:sid 
Resolving "debian" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/debian:sid...
Getting image source signatures
Error: copying system image from manifest list: writing blob: storing blob to file "/var/tmp/storage3298711326/1": write /var/tmp/storage3298711326/1: no space left on device

podman info:
https://gist.github.com/neuberfran/7a60df1bef913b559e966cf1d8ab41a7

🐶 seadog @bcm2711-rpi-4b~: df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               259.8M    209.1M     30.8M  87% /
devtmpfs                 10.0M         0     10.0M   0% /dev
tmpfs                    21.4M    292.0K     21.1M   1% /run
shm                      53.4M    168.0K     53.2M   0% /dev/shm
/dev/mmcblk0p1           46.3M     43.1M      3.2M  93% /boot
cgroup_root              10.0M         0     10.0M   0% /sys/fs/cgroup
df: /var/lib/containers/storage/overlay: Permission denied

🐶 seadog @bcm2711-rpi-4b~: cat /etc/*release
3.16.2
NAME="Seadog Linux"
ID=alpine
VERSION_CODENAME=war-never-changes-🪖-💀
VERSION_ID=0.0.1
PRETTY_NAME="Seadog Linux v0.0.1"

I tried podman volume prune, etc But not solved yet

How to solve

Pls

[mtrmac]

@neuberfran That indicates the total size of the root filesystem is 259.8 MB (and 30 MB free). Fitting an image with 44.8 MB compressed, and 122 MB uncompressed, is just impossible.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[abitrolly]

@rhatdan why did you close the issue? Did podman become space efficient when building images?

[vrothberg]

I reopened.

[neuberfran]

@mtrmac
Hi everyone,

I have rpi4 1gb (rev. 1.5) here.

I think that to run the command below, you need at least a 4gb or 8gb rpi4:
docker run --rm -it --privileged debian:sid

Issue I got:

Resolving "debian" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/debian:sid...
Getting image source signatures
Copying blob 541de84dec56 done  
[  202.048440] Out of memory: Killed process 1507 (podman) total-vm:751796kB, anon-rss:32320kB, file-rss:0kB, shmem-rss:84kB, UID:0 pgtables:200kB oom_score_adj:0
Killed

I take this opportunity to challenge you to put the seadog in an rpi4. I guarantee you won't be able to, even watching the videos from the @microhobby channel series on yt

[mtrmac]

[ 202.048440] Out of memory: Killed process 1507 (podman) total-vm:751796kB, anon-rss:32320kB, file-rss:0kB, shmem-rss:84kB, UID:0 pgtables:200kB oom_score_adj:0

@neuberfran please don’t mix different issues in the same report, even if they have something in common. This one was originally reporting being out of disk space — and still isn’t diagnosed (and that one is amd64, not a RPi). If we mix running out of memory to the same issue, the chance of either one making any progress will only get smaller.

[neuberfran]

@mtrmac
I solved the space problem (df -h result: before 87% used, now 55.9% used)
(After having done high level analysis in building the seadog-rpi4 image)
But the container has not yet been uploaded due to the limitation of my raspberry pi 4 (1gb only)

I've always been referring to rpi4 (arch64) which I'm using. I never referred to amd64. I hope I'm not disturbing.

[]s

[mtrmac]

@neuberfran That “out of memory” report would quite likely be welcome separately; just not in this issue.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[alexhaydock]

I'm still affected by this one and it can be reproduced quite dramatically by pulling the Silverblue container from Quay using Buildah.

Example Containerfile:

FROM quay.io/fedora-ostree-desktops/silverblue:39

This one 1.8GB image ends up filling my VM's disk before even finishing the pull:

[...]
Copying blob 5f9d3dcf5281 done  
Copying blob d319b2b962b0 done
Error: creating build container: writing blob: adding layer with blob "sha256:d53af960dcd80f95604f7ef40b93628610408aabe0418903b52966373e28be6e": creating layer with ID "b9b22bf8e1ef93e618f0d1502a9792948f2864cdc083f60c16f1db82f030e855": no space left on device

I've cleared out .local/share/containers by deleting it completely but whenever I try to pull the Silverblue container again it immediately fills my whole disk:

100G    /home/user/.local/share/containers/

I think this is a good container for trying to reproduce this issue, as it seems to contain thousands of tiny files, as it's a whole OS filesystem.

This is a basic Debian 12 installation in a QEMU VM with Podman installed from the Debian 12 repos, and the filesystem on ext4 with default settings. Buildah is being run rootless as an unprivileged user.

Some info:

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ apt list buildah
Listing... Done
buildah/stable,now 1.28.2+ds1-3+b1 amd64 [installed,automatic]

$ buildah version
Version:         1.28.2
Go Version:      go1.19.8
Image Spec:      1.1.0-rc2
Runtime Spec:    1.0.2-dev
CNI Spec:        1.0.0
libcni Version:  
image Version:   5.23.1
Git Commit:      
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Edit: I don't seem to be affected on Fedora 38 or 39 so it would seem like this has actually been fixed upstream and the issue is down to Debian continuing to ship the older Podman version.

[AYDEV-FR]

I'm still affected by this one and it can be reproduced quite dramatically by pulling the Silverblue container from Quay using Buildah.

Example Containerfile:

FROM quay.io/fedora-ostree-desktops/silverblue:39

This one 1.8GB image ends up filling my VM's disk before even finishing the pull:

[...]

Copying blob 5f9d3dcf5281 done  

Copying blob d319b2b962b0 done

Error: creating build container: writing blob: adding layer with blob "sha256:d53af960dcd80f95604f7ef40b93628610408aabe0418903b52966373e28be6e": creating layer with ID "b9b22bf8e1ef93e618f0d1502a9792948f2864cdc083f60c16f1db82f030e855": no space left on device

I've cleared out .local/share/containers by deleting it completely but whenever I try to pull the Silverblue container again it immediately fills my whole disk:

100G    /home/user/.local/share/containers/

I think this is a good container for trying to reproduce this issue, as it seems to contain thousands of tiny files, as it's a whole OS filesystem.

This is a basic Debian 12 installation in a QEMU VM with Podman installed from the Debian 12 repos, and the filesystem on ext4 with default settings. Buildah is being run rootless as an unprivileged user.

Some info:

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"

NAME="Debian GNU/Linux"

VERSION_ID="12"

VERSION="12 (bookworm)"

VERSION_CODENAME=bookworm

ID=debian

HOME_URL="https://www.debian.org/"

SUPPORT_URL="https://www.debian.org/support"

BUG_REPORT_URL="https://bugs.debian.org/"

$ apt list buildah

Listing... Done

buildah/stable,now 1.28.2+ds1-3+b1 amd64 [installed,automatic]

$ buildah version

Version:         1.28.2

Go Version:      go1.19.8

Image Spec:      1.1.0-rc2

Runtime Spec:    1.0.2-dev

CNI Spec:        1.0.0

libcni Version:  

image Version:   5.23.1

Git Commit:      

Built:           Thu Jan  1 00:00:00 1970

OS/Arch:         linux/amd64

BuildPlatform:   linux/amd64

Edit: I don't seem to be affected on Fedora 38 or 39 so it would seem like this has actually been fixed upstream and the issue is down to Debian continuing to ship the older Podman version.

Hello, I have exactly the same problem on quay.io/fedora/fedora-silverblue:39. With Buildah with the VFS storage driver. The idea is to build on a gitlab runner.

Have you found a solution?

[mtrmac]

Using the VFS storage driver simply is that costly with images that have that many layers. Arrange to use the overlay driver.

[alexhaydock]

Hello, I have exactly the same problem on quay.io/fedora/fedora-silverblue:39. With Buildah with the VFS storage driver. The idea is to build on a gitlab runner.

Have you found a solution?

Well a workaround but not a solution.

My fix was to move my GitLab Runner from a Debian 12 base to a CentOS Stream 9 base. Seems that CentOS ships a new enough version of Podman to avoid this issue, or maybe just a better default Podman config.

But it does sound based on other comments that there might be a solution available based on configuring alternative storage drivers?

[AYDEV-FR]

Using the VFS storage driver simply is that costly with images that have that many layers. Arrange to use the overlay driver.

I understand that VFS consumes, however if I'm using a Gitlab Runner Kubernetes, it's a container that runs Buildah and I don't want to have to mount /var/containers let alone give it SYS_ADMIN CAPS.
Am I wrong, or is this the only way to configure Buildah with overlay (not VFS) on a container runner?