Unable to push an image builded using buildah with overlayfs storage drive

[phozzy]

Description

I'm trying to make a CI build of a buildah image using a buildah image (recursive build for scheduled build to always have a fresh image). Everythin goes fine until the last step: pushing image to a registry.

Steps to reproduce the issue:

1. Clone this repo: https://gitlab.com/ph0zzy/fedbuilder
2. Switch to the bugreport branch to see a code of the pipeline
3. Run a job in gitlab-ci using this branch

Describe the results you received:
I recieve following error at the last step:

$ buildah push --creds ${CI_REGISTRY_USER}:${CI_JOB_TOKEN} ${CI_PROJECT_NAME} docker://${CI_REGISTRY_IMAGE}:${RELEASE}
Getting image source signatures
Copying blob sha256:3a056b419d6ac62ab8c4c4131788e95c89cee3c8aa20e8482c4b5af0b34f4136
Patch https://registry.gitlab.com/v2/ph0zzy/fedbuilder/blobs/uploads/ff2e1c5b-ae65-496e-a797-000a3c4d66f1?_state=85uei903ADoLVcHNpKt-Orb8ELG0E9Y0_s4av5Owtg17Ik5hbWUiOiJwaDB6enkvZmVkYnVpbGRlciIsIlVVSUQiOiJmZjJlMWM1Yi1hZTY1LTQ5NmUtYTc5Ny0wMDBhM2M0ZDY2ZjEiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDQtMDhUMjA6MzE6MDcuODEwMzgyMzkzWiJ9: open /var/lib/containers/storage/overlay/3a056b419d6ac62ab8c4c4131788e95c89cee3c8aa20e8482c4b5af0b34f4136/merged/etc/DIR_COLORS: no such file or directory
ERROR: Job failed: exit code 1

And as I understand this message it can not find it localy.
Full log is here https://gitlab.com/ph0zzy/fedbuilder/-/jobs/192938050
Describe the results you expected:
I expect to see something like this:

$ buildah --storage-driver vfs push --creds ${CI_REGISTRY_USER}:${CI_JOB_TOKEN} ${CI_PROJECT_NAME} docker://${CI_REGISTRY_IMAGE}:latest
Getting image source signatures
Copying blob sha256:1acd4249dbb9606f45d61e12ed53fecfcf2ebd0ae878f65a83108591284a28f6
Copying config sha256:89ce1fa76dee06697211ee9a827221799ccc153e58f2c3cdfac8ad0e5129c448
Writing manifest to image destination
Storing signatures
Successfully pushed //registry.gitlab.com/ph0zzy/fedbuilder:latest@sha256:ee81487f66bd6cf170d4d530a1b17a6397bcea51f75018fbbd70b0765d758d6d
Job succeeded

Output of rpm -q buildah or apt list buildah:

buildah-1.7-2.git873f001.fc29.x86_64

Output of buildah version:

Version:         1.7
Go Version:      go1.11.5
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Do not use podman

Output of cat /etc/*release:

Fedora release 29 (Twenty Nine)
NAME=Fedora
VERSION="29 (Twenty Nine)"
ID=fedora
VERSION_ID=29
VERSION_CODENAME=""
PLATFORM_ID="platform:f29"
PRETTY_NAME="Fedora 29 (Twenty Nine)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:29"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=29
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=29
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
Fedora release 29 (Twenty Nine)
Fedora release 29 (Twenty Nine)

Output of uname -a:

Linux runner-72989761-project-11601846-concurrent-0 4.19.23-coreos-r1 #1 SMP Mon Feb 25 23:40:01 -00 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
mount_program = "/usr/bin/fuse-overlayfs"

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base.
# device.
# mkfsarg = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver 
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

# If specified, use OSTree to deduplicate files with the overlay backend
ostree_repo = ""

# Set to skip a PRIVATE bind mount on the storage home directory.  Only supported by
# certain container storage drivers
skip_mount_home = "false"

[TomSweeneyRedHat]

@phozzy can you run your buildah push command again please, but use --debug and let us know the results?

buildah --debug push ...

Thanks!

[rhatdan]

Is this because docker.io is rejecting the OCI Format?
@phozzy Could you try your buildah bud command with --format docker and see if this solves the problem?

[phozzy]

Hi @TomSweeneyRedHat ! Here is a link to the log of a job with debug flag activated: https://gitlab.com/ph0zzy/fedbuilder/-/jobs/193077624

[phozzy]

Hi @rhatdan !

• I don't use docker.io, I use gitlab-ci registry to store my image and when I run similar job using --storage-driver vfs flag everything goes fine even withouth specifiing --format docker flag.
• I don use buildah bud command. My goal is to build an image using buildah commands only. But I do use --format docker flag with a buildah commit command. Here is a link to that line in the pipeline's code: https://gitlab.com/ph0zzy/fedbuilder/blob/bugreport/.gitlab-ci.yml#L21

[vrothberg]

@giuseppe, @nalind, do you have a suspicion regarding the source of the issue? I tried reproducing by following the instructions in the log (https://gitlab.com/ph0zzy/fedbuilder/-/jobs/193077624) but it's working for me.

[phozzy]

Hi @vrothberg do you use bugreport branch for a job? Since jobs from the master branch work fine. The difference between master and bugreport branch is that master uses --storage-driver vfs flag with a buildah command, and the bugreport branch uses default overlayfs storage driver.
@giuseppe , @nalind ☝️ FYI

[vrothberg]

@phozzy I was reproducing on my local machine with overlayfs.

[phozzy]

@vrothberg well... of course it runs well on local machine. actually the initial image that was used to lunch this pipeline was prepared localy using similar steps. The idea behind this pipeline is to have always updated image of buildah, so I'll be able to use it in other pipelines to build other images without any worrying if a image I use fresh or not.

[vrothberg]

well... of course it runs well on local machine.

This is not going to help unless you are clear why this cannot be reproducible in another environment.

[phozzy]

@vrothberg Did you try to reproduce it localy using buildah a local binary? Can you try to reproduce it using an image from this registry: https://gitlab.com/ph0zzy/fedbuilder/container_registry ?

[vrothberg]

@phozzy overlay in overlay is not supported. If we run Buildah/Podman in a container, we need to use vfs.

[vrothberg]

@giuseppe mentioned a trick though by using /var/lib/containers/storage as a volume, in case sharing the storage is acceptable for your use case.

[phozzy]

@vrothberg thanks! I'll try to check that.

[giuseppe]

@phozzy had a chance to try that out?

[phozzy]

HI @TomSweeneyRedHat @rhatdan @vrothberg @giuseppe !
Thanks for your help! Unfortunately I was not able find a way how to use /var/lib/containers/storage as a volume in CI's I've tried. I tried following CI's:

• gitlab-ci: The case is described in this issue topic. There is an open issue at gitlab-ci's issue-tracket, and I put a reference to the current issue in the issue at gitlab-ci's issue tracker.

• bitbacket: The same beheviour as in gitlab-ci. You can take a look at this project

• codeship: They promise the support of docker volumes in their pipelines. But I faced a problem with running buildah command. You can check it at this project.

If you know any public CI that supports the required feature please let me know.
If you'll have something to add to my project I'll appreciate that.
If you think that this issue is more issue of CIs, feel free to close it.

[vrothberg]

Thanks a lot for the great summary, @phozzy! I don't think we can do anything here to address the issues in the mentioned CI systems, so I'll close the issue. The safest and most portable approach may be using VFS inside the containers.