version 8.1.2 breaks container image jobs under arm64 architecture

[sveneisenmanndmde]

Hi,

the current 'latest' tag (version 8.1.2) breaks container image jobs under arm64 architecture.

We are using the image regularly in a gitlab ci/cd job and are receiving the following error message:

ERROR: Job failed (system failure): Error response from daemon: unable to find user curl_user: no matching entries in passwd file (exec.go:78:0s)

Please let me know, if I can support troubleshooting any further.

Best regards

[xquery]

thx for the report - are you able to manually test that env eg.

podman run -it docker.io/curlimages/curl:latest -V

should return something like

curl 8.1.2 (x86_64-pc-linux-musl) libcurl/8.1.2 OpenSSL/3.1.0 zlib/1.2.13 brotli/1.0.9 libidn2/2.3.4 libssh2/1.10.0 nghttp2/1.53.0
Release-Date: 2023-05-30
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largef

[xquery]

actually have reproduced locally

 podman run -it localhost/curl-linux-arm64:master -V
WARNING: image platform (linux/arm64/v8) does not match the expected platform (linux/amd64)
Error: unable to find user curl_user: no matching entries in passwd file

looks like somewhere the arch is assuming /v8 eg. before we built setting platform to 'linux/arm64' ... or some other new behaviour creating that curl_user specifically for that platform (which would be weird ...) investigating.

[spurin]

Just came by to report the same issue, I noticed it when referencing curlimages/curl:latest

% docker run -it curlimages/curl sh
docker: Error response from daemon: unable to find user curl_user: no matching entries in passwd file.
ERRO[0000] error waiting for container:

It threw me as I was seeing this reported by the kubelet -

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  26s                default-scheduler  Successfully assigned default/curl to control-plane
  Normal   Pulled     25s                kubelet            Successfully pulled image "curlimages/curl:latest" in 809.51675ms (809.529667ms including waiting)
  Warning  Failed     25s                kubelet            Error: failed to create containerd container: mount callback failed on /var/lib/rancher/k3s/agent/containerd/tmpmounts/containerd-mount4116084050: no users found

Thanks by the way, love your efforts with curlimages/curl 👍

[EricChen1248]

Came to report same issue.

On a podman machine I get

ERROR: Job failed (system failure): unable to upgrade to tcp, received 409 (exec.go:71:0s)

On a docker machine I get the same curl_user error as sveneisenmann

Thanks for the work though, this container is great for running super lightweight ci/cd jobs for web testing.

[JRemitz]

Noticed this line which seems unusual. Was this the output of something automated?

[xquery]

no that was an error but does not break anything - will be removing soon - thx for the quick 'eagle eyes'! Fix (#6) in flight.

[xquery]

reproduces:

docker run -it --platform linux/arm64 curlimages/curl:8.1.2 -V

other arches are fine, ex:

docker run -it --platform linux/amd64 curlimages/curl:8.1.2 -V

still investigating ...

[spurin]

Hi @xquery,

Looking at this, using https://explore.ggcr.dev/?image=curlimages/curl (current toplevel link)

We can see the hierarchy for each architecture -

AMD64: Link
ARM64: Link

Breaking this down further, each platform specific consists of 3 layers. It appears to be Alpine, Main App, then an Empty Layer (which can occur for the likes of environment variables or other container entities that dont write a layer).

AMD64: Link

• Alpine: Link
• Main App (curl): Link
• Empty Layer: Link

ARM64: Link

• Alpine: Link
• Main App (curl): Link
• Empty Layer: Link

Clicking the link and comparing the files in the 'Main App (curl)' layer between AMD64 and ARM64, suggests that the build process may have failed for this layer. For convenience, screenshot comparison -

AMD64 -

ARM64 -

Hopefully this helps with the troubleshooting.

[JRemitz]

Noticing that the CI is failing:

1 error occurred:
	* curl-base-multi:master: image not known


f56b1ae03f9e53bdf2e6ab3709ea4ec2c230a890975b43add2c3fe588a1e6548
1 error occurred:
	* curl-multi:master: image not known

And also later in the build, it appears to be running on the wrong arch, possibly related to the initial error not finding the image, but not sure:

exec container process `/sbin/apk`: Exec format error
error while running runtime: exit status 1
exec container process `/sbin/apk`: Exec format error
error while running runtime: exit status 1
her.
get curl source
exec container process `/bin/mkdir`: Exec format error
error while running runtime: exit status 1
error running container: error from crun creating container for [/usr/bin/curl -L -o curl.tar.gz https://github.com/curl/curl/archive/refs/heads/master.tar.gz]: executable file `/usr/bin/curl` not found in $PATH: No such file or directory
: exit status 1

[xquery]

image error is just side effect of attempting to remove manifest (which does not exist) ... thx for the debug info ... once I have a few free cycles will investigate more deeply

[davesmith00sky]

Just dropped by to report the same issue, thanks for investigating. 👍

➜  ~ docker run --rm curlimages/curl:8.1.2 --version
docker: Error response from daemon: unable to find user curl_user: no matching entries in passwd file.

[xquery]

fix coming soon - figured it out

[xquery]

in the short term have removed support for arm64 ... this is a deeper issue related to buildah (side effect of new container build systm)... once that is addressed arm64 support will come back (hopefully in a few days).

[spurin]

What's the issue that is being encountered with buildah and arm64 builds?

I'd like to offer support if possible with troubleshooting/investigating/fixing this if I can. It sounded like you had a potential fix but, your follow-up post suggests that something else might have occurred since then.

Many Thanks

[Jordan9500]

Hi, I am getting the same issue posted here but for s390x

docker run -it --platform linux/s390x curlimages/curl:8.1.2 -V
.......
docker: Error response from daemon: unable to find user curl_user: no matching entries in passwd file.

And get a mount faillure similar to: #3 (comment) for kubelet (Which also threw me off):

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  22s               default-scheduler  Successfully assigned kube-system/ext-res-updater-xsvrc to node
  Normal   Pulled     21s               kubelet            Successfully pulled image "curlimages/curl" in 298.088945ms (298.103129ms including waiting)
  Warning  Failed     21s               kubelet            Error: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount989058884: no users found

The workaround mentioned for using the prior tags (8.1.1) works.

Please let me know if this needs a different issue. With this issue focusing on the arm64 architecture.

Many Thanks!

[xquery]

Thx for any offer of help - anyone (and everyone) is welcome to raise a PR which we can review... my own blockers are the pedestrian and boring constraints of time and space ;)

As for the issue itself I think (once I have some free cycles) can fix ( which will include fixing s390x).

[xquery]

just to let everyone know - a few free cycles have appeared ... raising PR shortly

[xquery]

looks like that PR works ... the issue was using buildah in a container modulo qemu ... though the delay was me getting confused by a completely different qemu problem in my local dev env (and me thinking that was the issue in CI) ... release will happen in next 24 hrs (if not sooner).

[xquery]

latest release now publishing (will take a little time) -

[xquery]

everyone please test with latest releases (from quay/curl/curl:8.1.2 or docker.io/curlimages/curl:8.1.2)

apologies for the churn - migrating to a new build system came with some risks of disruption - hoping we are now past it.