"no matching manifest" on cross v0.2.5 #1214

ConradIrwin · 2023-03-02T05:35:31Z

Checklist

I've looked through the issues and pull requests for similar reports

Describe your issue

I've been cross-compiling https://github.com/automerge/automerge-go for a while. Today I upgraded cross, and it broke cross-compilation:

$ cargo init --bin hello
$ cd hello
$ cross run --target aarch64-unknown-linux-gnu
Unable to find image 'ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/aarch64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.

The error message is the same as the one I get when I run docker with no --platform argument, when I do not have the image cached locally (once the image has run once, I no longer need to pass --platform to docker to make it work).

In case it helps tracking this down, I'm using Docker version 20.10.23, build 7155243 (as provided by Docker Desktop 4.17.0 (99724).

$ docker run -it  ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 /bin/bash
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.

$ docker run -it --platform linux/amd64 ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 /bin/bash
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
58690f9b18fc: Pull complete
b51569e7c507: Pull complete
da8ef40b9eca: Pull complete
fb15d46c38dc: Pull complete
7cd0932c59ac: Pull complete
b8099f45b9a5: Pull complete
640227c3e7fb: Pull complete
feac1a957a2e: Pull complete
2bb7cc963f55: Pull complete
69f3cca19ad2: Pull complete
9eae8e6ab0e2: Pull complete
f6948013d489: Pull complete
e63c3aabe8fe: Pull complete
f3f8758fddaa: Pull complete
e8df8713a13e: Pull complete
840b2ff81099: Pull complete
e52d8b455ef1: Pull complete
Digest: sha256:9e5b39c09874bc1816c675ed11afca2c2ed6cee0c4ed2b3c1d5763c346c9ae3f
Status: Downloaded newer image for ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5
root@ad6f0bc127c6:/#

docker run -it  ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 /bin/bash
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
root@9e3a9c7477c6:/# exit

What target(s) are you cross-compiling for?

aarch64-unknown-linux-gnu, x86_64-unknown-linux-gnu

Which operating system is the host (e.g computer cross is on) running?

macOS
Windows
Linux / BSD
other OS (specify in description)

What architecture is the host?

x86_64 / AMD64
arm32
arm64 (including Mac M1)

What container engine is cross using?

docker
podman
other container engine (specify in description)

cross version

0.2.5

Example

$ cargo init --bin hello
$ cd hello
$ cross run --target aarch64-unknown-linux-gnu
Unable to find image 'ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/aarch64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.

Additional information / notes

This is not a problem on the previous version with cargo install cross@0.2.4. Everything works fine!

The text was updated successfully, but these errors were encountered:

Emilgardis · 2023-03-02T16:53:33Z

I think this is a regression in docker, pretty sure it used to work

$ cross build --target x86_64-unknown-linux-gnu -v
+ cargo metadata --format-version 1 --filter-platform x86_64-unknown-linux-gnu
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/local/bin/docker
+ /usr/local/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=emil' --rm --user 501:20 -v /Users/emil/.xargo:/xargo:z -v /Users/emil/.cargo:/cargo:z -v /cargo/bin -v /Users/emil/workspace/cross:/project:z -v /Users/emil/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:z,ro -v /Users/emil/workspace/cross/target:/target:z -w /project -i -t ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo build --target x86_64-unknown-linux-gnu -v'
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu

the fix is as mentioned adding --platform, which was added in #817

for now, if you must use 0.2.5, use CROSS_CONTAINER_OPTS="--platform linux/amd64"

$ CROSS_CONTAINER_OPTS="--platform linux/amd64" cross build --target x86_64-unknown-linux-gnu -v
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/local/bin/docker
+ /usr/local/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=emil' --platform linux/amd64 --rm --user 501:20 -v /Users/emil/.xargo:/xargo:z -v /Users/emil/.cargo:/cargo:z -v /cargo/bin -v /Users/emil/workspace/cross:/project:z -v /Users/emil/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:z,ro -v /Users/emil/workspace/cross/target:/target:z -w /project -i -t ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo build --target x86_64-unknown-linux-gnu -v'
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
58690f9b18fc: Pulling fs layer 
b51569e7c507: Pulling fs layer 
da8ef40b9eca: Pulling fs layer 
fb15d46c38dc: Waiting 
7cd0932c59ac: Waiting 
b8099f45b9a5: Waiting 
640227c3e7fb: Waiting 
...

Emilgardis · 2023-03-02T17:06:33Z

Hmm, actually, is it a change in the oci images?

$ cross-v0.2.5 metadata --target x86_64-unknown-linux-gnu -v
+ cargo metadata --format-version 1 --filter-platform x86_64-unknown-linux-gnu
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/local/bin/docker
+ /usr/local/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=emil' --rm --user 501:20 -v /Users/emil/.xargo:/xargo:z -v /Users/emil/.cargo:/cargo:z -v /cargo/bin -v /Users/emil/workspace/cross:/project:z -v /Users/emil/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:z,ro -v /Users/emil/workspace/cross/target:/target:z -w /project -i -t ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo metadata -v'
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5' locally
0.2.5: Pulling from cross-rs/x86_64-unknown-linux-gnu
docker: no matching manifest for linux/arm64/v8 in the manifest list entries.
See 'docker run --help'.
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu

$ cross-v0.2.4 metadata --target x86_64-unknown-linux-gnu -v
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain stable-x86_64-unknown-linux-gnu
+ rustup component list --toolchain stable-x86_64-unknown-linux-gnu
+ /usr/local/bin/docker
+ /usr/local/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=emil' --rm --user 501:20 -v /Users/emil/.xargo:/xargo:Z -v /Users/emil/.cargo:/cargo:Z -v /cargo/bin -v /Users/emil/workspace/cross:/project:Z -v /Users/emil/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:Z,ro -v /Users/emil/workspace/cross/target:/target:Z -w /project -i -t ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.4 sh -c 'PATH=$PATH:/rust/bin cargo metadata -v'
Unable to find image 'ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.4' locally
0.2.4: Pulling from cross-rs/x86_64-unknown-linux-gnu
58690f9b18fc: Pulling fs layer 
b51569e7c507: Pulling fs layer 
da8ef40b9eca: Pulling fs layer 
fb15d46c38dc: Waiting 
ba4cfa0d02e9: Waiting 
9b3d6abb553f: Waiting 
7e4f4e01d8cc: Waiting 
7457ea83291e: Waiting 
385151f4cf4a: Waiting 
dd3a4857dad0: Waiting 
f4c64b50a179: Waiting 
634c7871468d: Waiting 
94489cc40d0c: Waiting 
f47e7b8372aa: Waiting 
70626baa3f44: Waiting 
8c9e9eeed2e0: Waiting 
de83618d0833: Waiting 
^C

there's no changes in the invocation between 0.2.4 and 0.2.5 except we're using the tag 0.2.5 and using :z on mounts

very strange

Emilgardis · 2023-03-02T18:05:44Z

ehm, what's this?

unknown/unknown

Emilgardis · 2023-03-02T18:15:42Z

the sha for unknown/unknown matches

https://github.com/cross-rs/cross/actions/runs/4092128923/jobs/7057264550#step:16:129


#23 exporting to image
#23 exporting layers done
#23 exporting manifest sha256:a2d924b0399f17432c3c8d6af2136903c5105bf71c726044a988427f849a8509 done
#23 exporting config sha256:77a3259dcb3ad2fec032d8d51dea3ae852ea488ff33a0261a3dc3d3a06f6b7fb done
#23 exporting attestation manifest sha256:a62738a88fd7403bf1fd5beb14d391013206ec8e7d3a4ba6522cf61a411b20a0 done
#23 exporting manifest list sha256:9e5b39c09874bc1816c675ed11afca2c2ed6cee0c4ed2b3c1d5763c346c9ae3f done
#23 pushing layers
#23 ...

I've never seen this before, but apparently it's https://docs.docker.com/build/attestations/attestation-storage/

not sure if it's worth disabling, since I definitely feel like this is an upstream issue in that, imo, if the manifest only has one valid and supported platform, it should use that, if there's multiple supported platforms, then prompt the user to pick.

Emilgardis · 2023-03-02T18:19:15Z

related: docker/build-push-action#820

we don't use the build-push-action, but I suspect the issue it the same here as we basically do the same thing as it.

Emilgardis · 2023-03-02T19:00:54Z

So to summarize, the problem is

cross <0.3.0 calls docker run --rm -t ghcr.io/cross-rs/x86_64-unknown-linux-gnu:$version, no --platform specified
the image for :0.2.5 was built with docker buildx build, on a linux/amd64 system where provenence/attestation is supported by buildx/buildkit (moby/buildkit:buildx-stable-1 => buildkitd github.com/moby/buildkit v0.11.2 944939944ca4cc58a11ace4af714083cfcd9a3c7). No --platform specified
if provenence is used the manifest becomes multi-arch, always including unknown/unknown for the build attestations and then other platforms that was included in the build. 0.2.5 manifest is

$ docker buildx imagetools inspect ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5
Name:      ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:9e5b39c09874bc1816c675ed11afca2c2ed6cee0c4ed2b3c1d5763c346c9ae3f
           
Manifests: 
  Name:        ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5@sha256:a2d924b0399f17432c3c8d6af2136903c5105bf71c726044a988427f849a8509
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64
               
  Name:        ghcr.io/cross-rs/x86_64-unknown-linux-gnu:0.2.5@sha256:a62738a88fd7403bf1fd5beb14d391013206ec8e7d3a4ba6522cf61a411b20a0
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.type:   attestation-manifest
    vnd.docker.reference.digest: sha256:a2d924b0399f17432c3c8d6af2136903c5105bf71c726044a988427f849a8509

docker run and docker pull, when a manifest is a single platform, will try to use that platform, if it's multi-arch, it'll use the same platform as the default, and if it doesn't exist panic.

now, there's one issue, we never enabled provenence. docker/build-push-action#771 suggests that setting --provence=false would fix this, but the docs for buildx 0.11 suggest that that is the default. I'm very confused...

edit: ah, https://docs.docker.com/engine/reference/commandline/buildx_build/#provenance

By default, a minimal provenance attestation will be created for the build result, which will only be attached for images pushed to registries.

as can be seen here
https://github.com/cross-rs/cross/actions/runs/4092128923/jobs/7057264550#step:10:5997

#21 exporting to docker image format
#21 exporting layers
#21 exporting layers 31.6s done
#21 exporting manifest sha256:3fa66b56f1735512fe1ce052aa69e94b019c6beb32e06114695ae48f266ee12b done
#21 exporting config sha256:061c8657a90b4c9588ac4159f6b35526c705e8d5e8d1dcded9e5bb35c2effc6d done
#21 sending tarball
#21 ...

Emilgardis · 2023-03-02T20:01:32Z

so there's three ways to solve this

specify --provenence=false, rebuild the images and publish
specify buildkit version = 0.10, rebuild images and publish
publish cross 0.2.6 with a fix including --platform=linux/amd64, this should probably also include Ensure ignored CLI arguments to cross are always passed at the end. #1142

ConradIrwin · 2023-03-02T23:08:56Z

Thanks for figuring this out! There's no rush on a fix from my point of view (I can use 0.2.4 successfully).

Emilgardis · 2023-03-02T23:44:33Z

If you missed it, you can also use CROSS_CONTAINER_OPTS="--platform linux/amd64" to tell docker that you actually want to use the amd64 image.

use CROSS_BUILD_OPTS="--platform linux/amd64" for pre-build

cross on main already passes --platform

ActuallyHappening · 2023-04-19T10:23:38Z

For anybody else, to save you spending days looking for a solution here is how I solved it:
cargo install cross --git https://github.com/cross-rs/cross
Yep, update and it worked fine for me.

CosminPerRam · 2023-12-29T17:48:54Z

Hey, sorry for my lack of knowledge, but what's exactly the problem that blocks cross of having arm native docker images?

Emilgardis · 2023-12-29T21:48:44Z

Hey, sorry for my lack of knowledge, but what's exactly the problem that blocks cross of having arm native docker images?

The blocker is implementation in #975 and particularly access to GHA runners on aarch64 architecture to not have to emulate the containers in CI

Emilgardis added A-container-engine Area: container engines A-macos-host Area: macos hosts labels Mar 2, 2023

Emilgardis added the bug label Mar 2, 2023

Emilgardis mentioned this issue Mar 18, 2023

Several docker images don't support m1 Mac builds #1224

Closed

11 tasks

mu88 mentioned this issue Mar 25, 2023

GitHub Action produces unknown architecture and OS docker/build-push-action#820

Closed

Emilgardis mentioned this issue Apr 4, 2023

Cross compiling on M1 Mac gives weird linker error #1241

Open

11 tasks

tisonkun mentioned this issue May 24, 2023

Deploy Java bindings for multiple platforms apache/opendal#2313

Closed

Emilgardis mentioned this issue Jul 17, 2023

Running cross on apple silicon with orbstack fails to download the containers. #1288

Closed

11 tasks

praveen-em mentioned this issue Apr 28, 2024

pact-core@14.3.4: Failed to load native module in Linux-x64 platform (redhat ubi8 minimal) pact-foundation/pact-js-core#502

Closed

5 tasks

Emilgardis mentioned this issue May 7, 2024

Cross does not have an ARM64 platform docker image for v0.2.5 and main #1487

Closed

Emilgardis mentioned this issue May 22, 2024

CROSS_CONTAINER_OPTS does not replace --platform linux/amd64 #1498

Open

11 tasks

Emilgardis mentioned this issue Jun 16, 2024

Basic command is not running from Readme #1515

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"no matching manifest" on cross v0.2.5 #1214

"no matching manifest" on cross v0.2.5 #1214

ConradIrwin commented Mar 2, 2023 •

edited

Loading

Emilgardis commented Mar 2, 2023

Emilgardis commented Mar 2, 2023

Emilgardis commented Mar 2, 2023

Emilgardis commented Mar 2, 2023 •

edited

Loading

Emilgardis commented Mar 2, 2023 •

edited

Loading

Emilgardis commented Mar 2, 2023 •

edited

Loading

Emilgardis commented Mar 2, 2023 •

edited

Loading

ConradIrwin commented Mar 2, 2023

Emilgardis commented Mar 2, 2023 •

edited

Loading

ActuallyHappening commented Apr 19, 2023

CosminPerRam commented Dec 29, 2023 •

edited

Loading

Emilgardis commented Dec 29, 2023

"no matching manifest" on cross v0.2.5 #1214

"no matching manifest" on cross v0.2.5 #1214

Comments

ConradIrwin commented Mar 2, 2023 • edited Loading

Checklist

Describe your issue

What target(s) are you cross-compiling for?

Which operating system is the host (e.g computer cross is on) running?

What architecture is the host?

What container engine is cross using?

cross version

Example

Additional information / notes

Emilgardis commented Mar 2, 2023

Emilgardis commented Mar 2, 2023

Emilgardis commented Mar 2, 2023

Emilgardis commented Mar 2, 2023 • edited Loading

Emilgardis commented Mar 2, 2023 • edited Loading

Emilgardis commented Mar 2, 2023 • edited Loading

Emilgardis commented Mar 2, 2023 • edited Loading

ConradIrwin commented Mar 2, 2023

Emilgardis commented Mar 2, 2023 • edited Loading

ActuallyHappening commented Apr 19, 2023

CosminPerRam commented Dec 29, 2023 • edited Loading

Emilgardis commented Dec 29, 2023

ConradIrwin commented Mar 2, 2023 •

edited

Loading

Emilgardis commented Mar 2, 2023 •

edited

Loading

Emilgardis commented Mar 2, 2023 •

edited

Loading

Emilgardis commented Mar 2, 2023 •

edited

Loading

Emilgardis commented Mar 2, 2023 •

edited

Loading

Emilgardis commented Mar 2, 2023 •

edited

Loading

CosminPerRam commented Dec 29, 2023 •

edited

Loading