kube play snubs save-loaded image

[edsantiago]

Take an image, podman save it, then podman load it, and it works fine for purposes of podman run. As in, instantaneous with no registry checks.

podman kube play, however, turns up its nose at it and wants to re-fetch it from the registry, causing unnecessary wear and tear on network electrons.

Too complicated to explain, so here's a reproducer. Sample output:

...
Podman run - does not re-pull
Tue Aug 29 18:53:46 UTC 2023


**** Podman kube play - why does it re-pull?
Trying to pull quay.io/libpod/testimage:20221018...
Getting image source signatures
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob 578f06cc66c5 skipped: already exists  
Copying config f5a99120db done  
Writing manifest to image destination

DO NOT RUN REPRODUCER ON A SYSTEM YOU CARE ABOUT!

It runs podman system reset because I wanted to eliminate variables. It'll probably work safely without the system reset but I haven't tested that.

[mheon]

@umohnani8 Is this deliberate to match Kubernetes more closely?

[edsantiago]

The podman save + podman load is a key part of the (what I consider) buggy behavior. Without save/load, kube play happily uses the existing image without pinging the registry.

(In other words: I'm no Kubernetes expert, but I have trouble seeing how that could be a compatibility feature)

[vrothberg]

That is somehow expected. save + load changes the digest of the image and kube play uses --pull=newer. With the changed digest, podman will consider the image on the registry to be newer and pull it.

[vrothberg]

The code in question is here: https://github.com/containers/podman/blob/main/pkg/domain/infra/abi/play.go#L995-L1000

I think this can be made smarter and only use --newer if the image has the latest tag or is on localhost/.

[vrothberg]

Another thing: the integration tests cache the images as tarballs. I'd expect caching them in the dir: transport to be 1) much faster and 2) not run into the digest issue.

[vrothberg]

Actually, we could use the oci:... transport instead. While testing, I noticed a bug and a short coming when pulling from oci:... which is now fixed in containers/common#1630.

[vrothberg]

Another thing: the integration tests cache the images as tarballs. I'd expect caching them in the dir: transport to be 1) much faster and 2) not run into the digest issue.

Tested it and found that 1) is not correct; pulling from docker-archive is faster than pulling from oci or dir.