Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[podman] kind 0.8.1 fails to download node image because podman is incompatible with image:tag@sha256 #1700

Closed
axelsimon opened this issue Jul 1, 2020 · 9 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. kind/external upstream bugs

Comments

@axelsimon
Copy link

axelsimon commented Jul 1, 2020

What happened:
Tried to run kind create cluster. Failed with:

ERROR: failed to create cluster: failed to pull image "kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f": command "docker pull kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f" failed 
with error: exit status 125 

What you expected to happen:
Create a cluster :)

How to reproduce it (as minimally and precisely as possible):
Run GO111MODULE="on" go get sigs.k8s.io/kind@v0.8.1 && kind create cluster

Anything else we need to know?:
This is using podman and not docker, not that it should make a difference.

The SHA256 hash that is pinned for kindest/node-v.1.18.2 in the pull command is "7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f" but on docker hub, the hash is "1ab458c61fbd408081c9d321ceec1a50adffc2d060e1013d823af3c61fe90ff2".
This makes the pull command fail. Manually pulling the image and removing the hash downloads the image fine.

Environment:

  • kind version: 0.8.1
  • Kubernetes version: (use kubectl version): none (yet?)
  • Docker version: (use docker info): podman version 1.9.3
  • OS (e.g. from /etc/os-release): Fedora 32 (Workstation Edition)
@axelsimon axelsimon added the kind/bug Categorizes issue or PR as related to a bug. label Jul 1, 2020
@aojea
Copy link
Contributor

aojea commented Jul 1, 2020

/assign

@aojea
Copy link
Contributor

aojea commented Jul 1, 2020

@axelsimon this is curious, I'm able to reproduce the issue but seems that podman does not like that image

docker pull kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f: Pulling from kindest/node
Digest: sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
Status: Image is up to date for kindest/node@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
docker.io/kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
 podman pull kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
Error: invalid image reference "kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f"

@BenTheElder
Copy link
Member

BenTheElder commented Jul 1, 2020

It's not the wrong hash.
The point of the hash is to pull the correct image with the tag being informational only.
This is a podman bug.

@BenTheElder
Copy link
Member

BenTheElder commented Jul 1, 2020

The current tag is a different version of that image, which is exactly why we use the hash.

@BenTheElder BenTheElder changed the title kind 0.8.1 fails to download node image because wrong SHA256 hash is pinned [podman] kind 0.8.1 fails to download node image because podman is incompatible with image:tag@sha256 Jul 1, 2020
@BenTheElder BenTheElder added the kind/external upstream bugs label Jul 1, 2020
@BenTheElder
Copy link
Member

BenTheElder commented Jul 1, 2020

The podman driver is supposed to mitigate this by dropping the hash tag.

Is podman in your path as a fake docker?

@BenTheElder
Copy link
Member

BenTheElder commented Jul 1, 2020

Setting KIND_EXPERIMENTAL_PROVIDER=podman will force the podman specific codepath, which you may need because podman fails to be docker compatible in other ways too.

In HEAD we have some trickery to detect when docker in PATH is not actually docker

@axelsimon
Copy link
Author

axelsimon commented Jul 1, 2020

The current tag is a different version of that image, which is exactly why we use the hash.

Thanks for clarifying this. That makes sense.

Is podman in your path as a fake docker?

which docker points to /usr/bin/docker, which is just a wrapper around podman. So yes, running docker will run podman.

I found the KIND_EXPERIMENTAL_PROVIDER=podman workaround later, but after i'd already downloaded the image manually and so thought that was why it was then clearing that step.

I'm running into another issue now, but it seems unrelated.

Thanks for jumping on this so fast, it's nice to see :)

@BenTheElder
Copy link
Member

BenTheElder commented Jul 3, 2020

[being bad at being on vacation => delayed responses]

gotcha yeah, in HEAD we have detection of "is this actually docker" => if not then employ podman specific code to deal with things like this.

we should also file a bug upstream about this particular problem, it shouldn't be terribly difficult, the expected behavior is that the sha256 is accepted and the :tag part is ignored / just informational for the humans.

closing as a known / already fixed issue.

@tao12345666333
Copy link
Member

tao12345666333 commented Jul 3, 2020

Upstream issue containers/podman#6721

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. kind/external upstream bugs
Projects
None yet
Development

No branches or pull requests

4 participants