Copying rootfs fails

Dockerfile to reproduce:

FROM debian:buster-slim AS importer

RUN apt-get update && apt-get install -y libguestfs-tools wget xz-utils
RUN wget -nv https://raspi.debian.net/verified/20210718_raspi_4_buster.img.xz && xz -d 20210718_raspi_4_buster.img.xz && mkdir /rootfs && guestfish --ro -a 20210718_raspi_4_buster.img -m /dev/sda2 copy-out / /rootfs/ && rm 20210718_raspi_4_buster.img

FROM scratch
COPY --from=importer /rootfs/ /

For v1.6.0 this hangs at "Saving file rootfs for later use". This is also described in #960 and the PR #1724 improves the situation. However, it now stops in the copy step reporting

error building image: error building stage: failed to execute command: copying dir: chown /sys: read-only file system

The workaround from #1007 seems to work though.

Except that I am then getting the following error when trying to use this image in a subsequent build step:

INFO[0001] Unpacking rootfs as cmd RUN ... requires it. 
error building image: error building stage: failed to get filesystem from image: removing whiteout .wh.proc: unlinkat //proc/fb: permission denied

[kvaps]

Hi isn't #1724 not solving your problem?
Copying /sys should be solved in d43b732

mkdir -p /tmp/123
cat > /tmp/123/Dockerfile <<EOT
FROM debian:buster-slim AS importer

RUN apt-get update && apt-get install -y libguestfs-tools wget xz-utils
RUN wget -nv https://raspi.debian.net/verified/20210718_raspi_4_buster.img.xz && xz -d 20210718_raspi_4_buster.img.xz && mkdir /rootfs && guestfish --ro -a 20210718_raspi_4_buster.img -m /dev/sda2 copy-out / /rootfs/ && rm 20210718_raspi_4_buster.img

FROM scratch
COPY --from=importer /rootfs/ /
EOT
docker run -ti --rm -v /tmp/123:/context ghcr.io/kvaps/kaniko-executor:v1.6.0 -c /context --no-push

I use the same approach to build VM images out of docker images :)
https://github.com/kvaps/opennebula-images/tree/da5bcbadbe2915167ef74fde7166ff4d75fb1151/new

So running the following (note: the download URL slightly changed)

mkdir -p /tmp/123
cat > /tmp/123/Dockerfile <<EOT
FROM debian:buster-slim AS importer

RUN apt-get update && apt-get install -y libguestfs-tools wget xz-utils
RUN wget -nv https://raspi.debian.net/tested/20210718_raspi_4_buster.img.xz && xz -d 20210718_raspi_4_buster.img.xz && mkdir /rootfs && guestfish --ro -a 20210718_raspi_4_buster.img -m /dev/sda2 copy-out / /rootfs/ && rm 20210718_raspi_4_buster.img

FROM scratch
COPY --from=importer /rootfs/ /
EOT
docker run -ti --rm -v /tmp/123:/context ghcr.io/kvaps/kaniko-executor:v1.7.0 -c /context --no-push

hangs for me at Saving file rootfs for later use. Same for kaniko-executor:v1.6.0. If I remeber correctly running it with kaniko-executor:v1.6.0-fix back then actually worked around the hang (see above). Did that work for you?

I struggeled using your image in our Gitlab CI/CD pipeline. I digged a little deeper and it turns out that multiple achitectures are mixed up there. When pulling from x64 I obtain an arm64 image where for example /busybox/sh is indeed arm64 but the executor is x64:

uname -m
# x86_64
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
mkdir -p /tmp/123
docker run --rm -it -v /tmp/123:/context --entrypoint /busybox/sh ghcr.io/kvaps/kaniko-executor:v1.7.0-debug
# WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
# cp /busybox/sh /context/
# cp /kaniko/executor /context/
# exit
file /tmp/123/sh
# sh: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped
file /tmp/123/executor
# executor: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=7BkQGRwXsNnIqlKjVroQ/0m_j6BzaX-FhkPpGcJaR/-9sTmtlaXltjhcGsg2mS/HMbpOMzbZstMMB7jfvs1, stripped

[kvaps]

Oh that's bad, I just bought a new Macbook with M1 chip, and get confused by amount of architectures 😅

Images are fixed now:

ghcr.io/kvaps/kaniko-executor:v1.7.0@sha256:63c8fddec65e5b48ab41657da537f9f2360d6fbc179ecc096c414597d739143a
ghcr.io/kvaps/kaniko-executor:v1.7.0-debug@sha256:2124a5418d126d8684ed03e539b08d1b4bc5f1cc23751abeff182802cb7906ad
ghcr.io/kvaps/kaniko-warmer:v1.7.0@sha256:a3f95ed186b37cf792f5dd0f648d8b2ba4907ddb8dffee3ab0cee97a01dbeb45

The same images are used in my kubectl-build plugin

[mav96]

Oh that's bad, I just bought a new Macbook with M1 chip, and get confused by amount of architectures 😅

Images are fixed now:

ghcr.io/kvaps/kaniko-executor:v1.7.0@sha256:63c8fddec65e5b48ab41657da537f9f2360d6fbc179ecc096c414597d739143a
ghcr.io/kvaps/kaniko-executor:v1.7.0-debug@sha256:2124a5418d126d8684ed03e539b08d1b4bc5f1cc23751abeff182802cb7906ad
ghcr.io/kvaps/kaniko-warmer:v1.7.0@sha256:a3f95ed186b37cf792f5dd0f648d8b2ba4907ddb8dffee3ab0cee97a01dbeb45

The same images are used in my kubectl-build plugin

Unfortunately it doesn't work either.

I tried this example:

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all

FROM quay.io/keycloak/keycloak
COPY --from=ubi-micro-build /mnt/rootfs /

[niekcandaele]

Oh that's bad, I just bought a new Macbook with M1 chip, and get confused by amount of architectures 😅
Images are fixed now:

ghcr.io/kvaps/kaniko-executor:v1.7.0@sha256:63c8fddec65e5b48ab41657da537f9f2360d6fbc179ecc096c414597d739143a
ghcr.io/kvaps/kaniko-executor:v1.7.0-debug@sha256:2124a5418d126d8684ed03e539b08d1b4bc5f1cc23751abeff182802cb7906ad
ghcr.io/kvaps/kaniko-warmer:v1.7.0@sha256:a3f95ed186b37cf792f5dd0f648d8b2ba4907ddb8dffee3ab0cee97a01dbeb45

The same images are used in my kubectl-build plugin

Unfortunately it doesn't work either.

I tried this example:

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all

FROM quay.io/keycloak/keycloak
COPY --from=ubi-micro-build /mnt/rootfs /

Heya, had the same issue with our Keycloak container

Worked around it with the following COPY statements instead of the full / COPY

COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/

Hope that helps 😊

[chary1112004]

Oh that's bad, I just bought a new Macbook with M1 chip, and get confused by amount of architectures 😅
Images are fixed now:

ghcr.io/kvaps/kaniko-executor:v1.7.0@sha256:63c8fddec65e5b48ab41657da537f9f2360d6fbc179ecc096c414597d739143a
ghcr.io/kvaps/kaniko-executor:v1.7.0-debug@sha256:2124a5418d126d8684ed03e539b08d1b4bc5f1cc23751abeff182802cb7906ad
ghcr.io/kvaps/kaniko-warmer:v1.7.0@sha256:a3f95ed186b37cf792f5dd0f648d8b2ba4907ddb8dffee3ab0cee97a01dbeb45

The same images are used in my kubectl-build plugin

Unfortunately it doesn't work either.
I tried this example:

FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all

FROM quay.io/keycloak/keycloak
COPY --from=ubi-micro-build /mnt/rootfs /

Heya, had the same issue with our Keycloak container

Worked around it with the following COPY statements instead of the full / COPY

COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/ /lib64/

Hope that helps 😊

Hi, I had same issue while trying to copy curl while building keycloak image. After I did work around like your suggestion then it passed for build image however there is error while deploy pod and below is error describe pod. Did you have same issue and any idea to resolve this? Thanks!

  Warning  Unhealthy  32s (x10 over 50s)  kubelet            Liveness probe failed: curl: error while loading shared libraries: libcurl.so.4: cannot open shared object file: No such file or directory
  Warning  Unhealthy  30s (x11 over 50s)  kubelet            Readiness probe failed: curl: error while loading shared libraries: libcurl.so.4: cannot open shared object file: No such file or directory

[niekcandaele]

Heya, I made an oopsie when copying my suggestion, my bad! 🙈

Give the following a try:

COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/

Tested like this

catalysm:px/ (develop✗) $ docker run -it kc '--&& bash'                                                                                                                                                                                                                      
bash-5.1$ curl --help
Usage: curl [options...] <url>
 -d, --data <data>   HTTP POST data
 -f, --fail          Fail silently (no output at all) on HTTP errors
 -h, --help <category>  Get help for commands
 -i, --include       Include protocol response headers in the output
 -o, --output <file>  Write to file instead of stdout
 -O, --remote-name   Write output to a file named as the remote file
 -s, --silent        Silent mode
 -T, --upload-file <file>  Transfer local FILE to destination
 -u, --user <user:password>  Server user and password
 -A, --user-agent <name>  Send User-Agent <name> to server
 -v, --verbose       Make the operation more talkative
 -V, --version       Show version number and quit

This is not the full help, this menu is stripped into categories.
Use "--help category" to get an overview of all categories.
For all options use the manual or "--help all".
bash-5.1$ curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

[sanderjochems-whyellow]

Heya, I made an oopsie when copying my suggestion, my bad! 🙈

Give the following a try:

COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/

Tested like this

This gives me the following error:

INFO[0035] COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/ 
INFO[0035] Resolving srcs [/mnt/rootfs/lib64/*]...      
error building image: error building stage: failed to execute command: resolving dest symlink: failed to eval symlinks: EvalSymlinks: too many links

I am installing curl using the following command

RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y && \
    dnf --installroot /mnt/rootfs clean all && \
    rpm --root /mnt/rootfs -e --nodeps setup