Not finding local images that do exist, trying to pull instead #42893
Comments
|
@crazy-max @tonistiigi PTAL
I don't think that should be a problem; docker uses I'm trying to reproduce the issue, but so far haven't been able to (running on docker desktop for mac, but not on m1) @jeffparsons do you know if you're using |
Ah, neato. That's reassuring.
I don't think I have I do have I'll install buildx and see if the outcome changes. |
|
After installing |
|
Thanks! My reason for asking if you were using buildx, is that buildx allows you to create multiple "builders". Those builders can use the "container" driver, in which case they're running isolated in a container (which doesn't share the local image cache of the docker daemon). If you would have been using that, it could (e.g.) have been a case where the first image was built in a builder container, but not loaded back into the docker daemon (and because of that not found during the second build). The error is emitted in the Dockerfile "front-end"; https://github.com/moby/buildkit/blob/v0.8.3/frontend/dockerfile/dockerfile2llb/convert.go#L247-L251 dgst, dt, err := metaResolver.ResolveImageConfig(ctx, d.stage.BaseName, llb.ResolveImageConfigOpt{
Platform: platform,
ResolveMode: opt.ImageResolveMode.String(),
LogName: fmt.Sprintf("%s load metadata for %s", prefix, d.stage.BaseName),
})I'm wondering if the cause of this could be related to inconsistencies in how arm architectures are normalised. Arm architectures are "complicated" (various variants can be "compatible", but there's also multiple notations for the same architecture). e.g. see containerd/containerd#4932. It's possible that some code normalises the architecture, and other code tries to match the architecture (with different normalizing), therefore doesn't find a match, so tries to pull the "missing" architecture from the registry (docker hub) instead. Unfortunately the error message doesn't include what architecture it tried to match (and what it found), or what it is trying to pull. I was looking here at the difference between docker buildx imagetools inspect rockylinux/rockylinux:8.4@sha256:8122f31fbdd5c1368c6b7d5b9ae99fec2eb5966a5c967339d71e95c4a3ab7846
Name: docker.io/rockylinux/rockylinux:8.4@sha256:8122f31fbdd5c1368c6b7d5b9ae99fec2eb5966a5c967339d71e95c4a3ab7846
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest: sha256:8122f31fbdd5c1368c6b7d5b9ae99fec2eb5966a5c967339d71e95c4a3ab7846
Manifests:
Name: docker.io/rockylinux/rockylinux:8.4@sha256:e98dd481db7973d701f8695df7be1b2393261cf273aa68a89d873b7b134588df
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform: linux/arm64/v8
Name: docker.io/rockylinux/rockylinux:8.4@sha256:f0d7460b97156f6c8ea2ae73152bc11fe410d272387d60ddff36dfcea22ef689
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform: linux/amd64But looking at (e.g.) the I'd have to double-check if the architectures printed there are the raw ones, or if there's possibly some normalizing happening when presenting them though. |
|
A colleague of mine who has an m1 machine was able to confirm your issue (saw the same failure), so it looks to be specific to the platform (which could confirm my suspicion there's something wrong in platform normalisation / matching) |
thaJeztah
added
kind/bug
|
That sounds promising. 😃 Please do let me know if there's anything else I can do from this side to help narrow it down. (I'm not familiar with the Docker codebase though so I probably wouldn't be much help at actually trying to fix the bug.) |
|
I just ran into this issue on my M1 MacBook Pro. Even with Is there any reason not to disable buildkit as a workaround? |
@byronbowerman It depends on whether or not you're intentionally using its features. BuildKit brings some nice stuff for general use (like garbage collection of builder stages) but overall Docker works just fine without it. In our case, though, we're increasingly relying on features that BuildKite brings to the table, and plan to do so more in future. E.g. Docker's legacy handling of build context is a bit painful in large repositories. E.g., if I have a "common" directory and a "backend" directory, and I want an image to have access to both, then I might want to make my build context the root of the repository. But legacy Docker will then tar up the entire repository for every build, which may range from slow to pretty much impracticable. Workarounds include dynamically generating a ".dockerignore" file before each build, copying parts of the tree around in a "pre-build" step (we do this one), or making the build context yourself with a script and then sending that to Docker explicitly (very tempting). BuildKit, on the other hand, only tries to read the files you've actually chosen to It puts us in a slightly awkward position when things like this pop up, because the answer to problems with the legacy builder appears to be "just use BuildKit, that's where all development effort is focused now", and the answer to problems with BuildKit is "BuildKit's not stable yet, just use the default builder". 😅 |
|
Just to be sure (as the code I linked to is part of the Dockerfile front-end)l; could one of you try if the latest Dockerfile front-end (syntax) possibly resolves the issue? If you add this to the top of your dockerfiles: # syntax=docker/dockerfile:1(which should pull the latest Dockerfile syntax/front-end from Docker Hub during build). |
|
Also: @tonistiigi @crazy-max PTAL |
|
@thaJeztah I added the syntax specification to both the parent and child dockerfiles and rebuilt as per previous example. The second build still can't find the parent image: Let me know if there's anything else I can test to help resolve this :) |
|
Thanks! Was hoping that could be a workaround, but alas. I just got a message from @crazy-max that he received his m1 machine (which allows him to do some debugging) |
|
Ok so I made some tests on some hosts to be sure of reproducibility: WSL2 (Ubuntu 20.04.3 LTS x86_64)Was not able to repro on x86: Mac Mini M1Was able to repro on M1 too like @jeffjohnston: Raspberry 4 (Ubuntu 20.04.2 LTS aarch64)Also repro on this platform: Raspberry 3 (Raspbian GNU/Linux 10 armv7l)No repro for this one: SiFive HiFive Unmatched (Ubuntu 21.04 riscv64)I use ConclusionSo my guess is a misinterpretation of variants in the meta resolver with arm64 and arm64/v8 that is not picked up right. Is it familiar to you @tonistiigi? |
|
Wonder if this is not linked to #40790 or at least something that needs to be changed in this method: moby/builder/builder-next/adapters/containerimage/pull.go Lines 855 to 863 in 1430d84 |
|
@jeffjohnston As a workaround it should work with: docker build --platform linux/arm64 -f Dockerfile.parent -t local/foo-parent .
docker build -f Dockerfile.child -t local/foo-child .
|
|
@crazy-max That workaround works just fine for me. 👍 I could've sworn I'd tried that already, but I guess I must have only tried the other way around (i.e. specifying platform explicitly when building the child image)! Thanks for that. I'll hopefully have a chance to integrate this into our build system in the coming days/weeks and report on whether I run into any other M1-specific issues. |
|
@crazy-max Thanks :) |
Yes sorry I was not on the right host when I've done that test. |
|
Greetings @thaJeztah, et al.
I'm seeing an issue similar to the one described in this issue on the latest and greatest Docker Desktop v4.12.0 (85629) on Apple silicon Macbook, but I'm building parent and child images with The builder is created with the following command: docker buildx create \
--use \
--platform linux/arm64,linux/amd64 \
--name multiarch \
--bootstrap \
--buildkitd-flags '--allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host'These are example Dockerfiles I'm using: # Dockerfile.parent
FROM alpine:3.16
RUN echo "this is a parent" | tee > /parent# Dockerfile.child
FROM parent
RUN rm -f /parent; echo "this is a child" | tee > /childAnd here is my build test |
|
Yes, that's the expected behavior. Contrary to the BuildKit builder embedded in the "Docker Engine" (which is more targeted at local development), BuildKit (containerized) builders are designed to be sandboxed builder nodes that don't depend on prior state. Those build nodes may contain some amount of caching data to speed up builds, but don't store images (or other persistent data). This design allows builders to be part of a distributed system, where builds can be distributed over the cluster. To build an image that depends on another image, either;
# Dockerfile.parent
FROM alpine:3.16 AS parent
RUN echo "this is a parent" | tee > /parent
FROM parent AS child
RUN rm -f /parent; echo "this is a child" | tee > /childMore details can be found in this discussion in the BuildKit repository moby/buildkit#2343 |
|
Is this behavior documented anywhere? I've spent the last hour or two in a similar scenario (trying to use a locally-built parent image in a You might imagine that finding out this obtuse error is expected behavior is somewhat frustrating. |
|
... setting Example: |
|
@tech-team-rural-mda please open a new ticket instead with reproducible steps. As to that image; are you sure the image is available as arm64, because if it's a local copy of |
Description
When BuildKit is enabled, if you tag an image with
--tagand then refer to it when building another image, Docker will not find it and instead tries to pull it from Docker Hub.Disabling BuildKit makes the problem go away.
Steps to reproduce the issue:
Make these files:
Dockerfile.parent:
Dockerfile.child:
The run these commands:
Describe the results you received:
On building the parent image, the last line says:
On building the child image, it fails to find the local image and instead tries to pull from Docker Hub:
Describe the results you expected:
The tag I specified already had a
/in it, so Docker shouldn't have automatically prefixed it withdocker.io/. I guess that's at least part of the problem?However,
docker inspectstill finds the image just fine by either name, so I'm not sure why building the child image fails to find it.Additional information you deem important (e.g. issue happens only occasionally):
I'm running Docker Desktop on an M1 MacBook.
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
Physical MacBook with Apple Silicon (M1), default settings, relatively fresh install of everything including Docker. (This is my first time trying to get an existing project working on the new Macs.)
This is all pure ARM -- no x86_64 virtualisation or anything complicating it as far as I can tell:
The text was updated successfully, but these errors were encountered: