New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RUN --mount-type=cache,sharing=locked hangs in specific cases #4342
Comments
I'll check this, thanks for reporting. |
`--mount=type=cache` must not add internal lockfiles to cache directory created by users instead store it in a different central directory with path as `/base/buildah-cache/buildah-lockfiles`. There are use-cases where users can wipe cache between the builds so lockfiles will be removed in unexpected manner and also its not okay to mix buildah's internal construct with user's cache content. Helps in: containers#4342 Signed-off-by: Aditya R <arajan@redhat.com>
Single `RUN` can contain multiple `--mount` commands so lets append into `lockedTargets` so we collect `lockfiles` from all the `--mount` instructions. Helps in: containers#4342 Signed-off-by: Aditya R <arajan@redhat.com>
Use-cases as shown in below containerfile cleans cache between the builds, in previous commits we have ensured that buildah lockfiles will not be part of user's cache content hence following use-case must paas ``` FROM quay.io/centos/centos:7 ARG WIPE_CACHE RUN --mount=type=cache,target=/cache1,sharing=locked \ --mount=type=cache,target=/cache2 \ set -ex; \ ls -l /cache1; \ if [[ -v WIPE_CACHE ]]; then \ >&2 echo "Wiping cache"; \ find /cache1 -mindepth 1 -delete; \ fi; \ echo "foo" > /cache1/foo.txt; \ ls -l /cache1; \ chmod --recursive g=u /cache1; \ : ; RUN --mount=type=cache,target=/cache1,sharing=locked \ >&2 echo "Never get here" ``` Closes: containers#4342 Signed-off-by: Aditya R <arajan@redhat.com>
Wow, that was an insanely fast turnaround. I was still setting up a build environment for it! Thanks! |
`--mount=type=cache` must not add internal lockfiles to cache directory created by users instead store it in a different central directory with path as `/base/buildah-cache/buildah-lockfiles`. There are use-cases where users can wipe cache between the builds so lockfiles will be removed in unexpected manner and also its not okay to mix buildah's internal construct with user's cache content. Helps in: containers#4342 Signed-off-by: Aditya R <arajan@redhat.com>
Single `RUN` can contain multiple `--mount` commands so lets append into `lockedTargets` so we collect `lockfiles` from all the `--mount` instructions. Helps in: containers#4342 Signed-off-by: Aditya R <arajan@redhat.com>
Use-cases as shown in below containerfile cleans cache between the builds, in previous commits we have ensured that buildah lockfiles will not be part of user's cache content hence following use-case must paas ``` FROM quay.io/centos/centos:7 ARG WIPE_CACHE RUN --mount=type=cache,target=/cache1,sharing=locked \ --mount=type=cache,target=/cache2 \ set -ex; \ ls -l /cache1; \ if [[ -v WIPE_CACHE ]]; then \ >&2 echo "Wiping cache"; \ find /cache1 -mindepth 1 -delete; \ fi; \ echo "foo" > /cache1/foo.txt; \ ls -l /cache1; \ chmod --recursive g=u /cache1; \ : ; RUN --mount=type=cache,target=/cache1,sharing=locked \ >&2 echo "Never get here" ``` Closes: containers#4342 Signed-off-by: Aditya R <arajan@redhat.com>
`--mount=type=cache` must not add internal lockfiles to cache directory created by users instead store it in a different central directory with path as `/base/buildah-cache/buildah-lockfiles`. There are use-cases where users can wipe cache between the builds so lockfiles will be removed in unexpected manner and also its not okay to mix buildah's internal construct with user's cache content. Helps in: containers#4342 Signed-off-by: Aditya R <arajan@redhat.com>
Single `RUN` can contain multiple `--mount` commands so lets append into `lockedTargets` so we collect `lockfiles` from all the `--mount` instructions. Helps in: containers#4342 Signed-off-by: Aditya R <arajan@redhat.com>
Use-cases as shown in below containerfile cleans cache between the builds, in previous commits we have ensured that buildah lockfiles will not be part of user's cache content hence following use-case must paas ``` FROM quay.io/centos/centos:7 ARG WIPE_CACHE RUN --mount=type=cache,target=/cache1,sharing=locked \ --mount=type=cache,target=/cache2 \ set -ex; \ ls -l /cache1; \ if [[ -v WIPE_CACHE ]]; then \ >&2 echo "Wiping cache"; \ find /cache1 -mindepth 1 -delete; \ fi; \ echo "foo" > /cache1/foo.txt; \ ls -l /cache1; \ chmod --recursive g=u /cache1; \ : ; RUN --mount=type=cache,target=/cache1,sharing=locked \ >&2 echo "Never get here" ``` Closes: containers#4342 Signed-off-by: Aditya R <arajan@redhat.com>
Use-cases as shown in below containerfile cleans cache between the builds, in previous commits we have ensured that buildah lockfiles will not be part of user's cache content hence following use-case must paas ``` FROM quay.io/centos/centos:7 ARG WIPE_CACHE RUN --mount=type=cache,target=/cache1,sharing=locked \ --mount=type=cache,target=/cache2 \ set -ex; \ ls -l /cache1; \ if [[ -v WIPE_CACHE ]]; then \ >&2 echo "Wiping cache"; \ find /cache1 -mindepth 1 -delete; \ fi; \ echo "foo" > /cache1/foo.txt; \ ls -l /cache1; \ chmod --recursive g=u /cache1; \ : ; RUN --mount=type=cache,target=/cache1,sharing=locked \ >&2 echo "Never get here" ``` Closes: containers#4342 Signed-off-by: Aditya R <arajan@redhat.com>
@marwatk Thanks, PR is merged feel free to directly use |
Additional(?) simpler case: FROM quay.io/centos/centos:7
RUN \
--mount=type=cache,id=442870093b67597071f8e55,target=/cache1,sharing=locked \
--mount=type=cache,id=442870093b67597071f8e55,target=/cache2,sharing=locked \
>&2 echo "Never get here" @flouthoc - I'm not the best at go, and I haven't spent any real time in the code base, but I think this might not be covered by the existing fix? |
You want to mount the same files twice and lock them both times? What's the use case for that? If you just want the same files to show up under two different paths (though I'm still have trouble imagining why) and ensure other builds can't use them you can just lock the first one, no need to lock both. What is your expected behavior by locking both? |
Yes, you're right that the files do show up twice (I think I was expecting them to be separated by destination, for some reason). So no, it's not likely to be observed in real scenarios. That said, a hang in this case is still unfriendly. |
Description
When performing specific actions (
rm
orchmod
) on a cached volume mounted with locking enabled future RUN clauses will hang. (See Dockerfile below).We discovered the bug because we
chmod
cache folders in some builds to allow writes when we drop root. It looks likebuildah
writes the lock file in the volume sochmod
somehow disrupts the lock.In other builds we utilize an env var to wipe the cache folder so we don't have to prune everything for a single build (though we still haven't figured out how to prune cache volumes, neither
rmi --prune
norrm -a
seem to remove them), which you can replicate with theWIPE_CACHE
build arg.Interestingly the second volume is required to exhibit the bug when
chmod
ing, even though it's not locked and never touched. The second volume is not required for theWIPE_CACHE
case to hang.Without the second mount or
--build-arg WIPE_CACHE=1
a warning is issued that the lock file is missing, but the hang still occurs. The warning is not present if both volumes are supplied but the hang still occurs.I cannot reproduce the issue with
buildkit
.Steps to reproduce the issue:
Steps 1 and 2 may not be required but help avoid hitting cached layers. These steps reproduce the issue from both root and non-root users.
buildah rm -a
buildah rmi --prune
buildah build -f - < ./Dockerfile
Dockerfile:
Describe the results you received:
Build hangs indefinitely at:
Describe the results you expected:
Build completes
Output of
rpm -q buildah
orapt list buildah
:Output of
buildah version
:Output of
cat /etc/*release
:Output of
uname -a
:Output of
cat /etc/containers/storage.conf
:The text was updated successfully, but these errors were encountered: