weird: 'podman history' loses history if centos image is present

[edsantiago]

This is a surreal variant of #980. Please bear with me:

# podman run --name c1 fedora true; podman commit -q c1 i1;podman rm c1;podman history i1
3b72fee4c9a3099653f46614c1feae0b878ffccd86392a8874ffda3571cd9500
9268feff54297d8e7e97637152c90d420717ae628c09684a396d232af15c6017
ID             CREATED         CREATED BY                                      SIZE      COMMENT
3b72fee4c9a3   2 seconds ago   /bin/sh                                         2.56kB
289289d1a15b   7 days ago      /bin/sh -c #(nop) CMD ["/bin/bash"]             2.56kB
<missing>      7 days ago      /bin/sh -c #(nop) ADD file:d3cc5fb69dcca3e...   326.1MB
<missing>      7 days ago      /bin/sh -c #(nop) ENV DISTTAG=f30-updates-...   326.1MB
<missing>      5 months ago    /bin/sh -c #(nop) LABEL maintainer=Clement...   326.1MB

# podman run --name c2 i1;podman commit -q c2 i2;podman rm c2;podman history i2
da470886d9855dd0745d6412c531465ce170902ce6291128bccd2b47db650809
0d55e91ebd06535981a50cfeb47a5afb28d16cbfc99a7e1a16ced9d74f5fe45b
ID             CREATED          CREATED BY                                      SIZE      COMMENT
da470886d985   1 second ago     /bin/sh                                         1.024kB
3b72fee4c9a3   12 seconds ago   /bin/sh                                         2.56kB
289289d1a15b   7 days ago       /bin/sh -c #(nop) CMD ["/bin/bash"]             2.56kB
<missing>      7 days ago       /bin/sh -c #(nop) ADD file:d3cc5fb69dcca3e...   326.1MB
<missing>      7 days ago       /bin/sh -c #(nop) ENV DISTTAG=f30-updates-...   326.1MB
<missing>      5 months ago     /bin/sh -c #(nop) LABEL maintainer=Clement...   326.1MB

So far, so good. In particular, note the second (non-header) history line of the second command: it references 3b72fee4c9a3 which is i1, the image on which i2 is based.

Now remove those images, pull in centos, and repeat:

# podman rmi i1 i2
Untagged: localhost/i1:latest
3b72fee4c9a3099653f46614c1feae0b878ffccd86392a8874ffda3571cd9500
da470886d9855dd0745d6412c531465ce170902ce6291128bccd2b47db650809

# podman pull docker.io/stackbrew/centos:latest
Trying to pull docker.io/stackbrew/centos:latest...Getting image source signatures
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 done
Copying blob 3286cdf780ef done
Writing manifest to image destination
Storing signatures
61b442687d681ef80a7b1ae148ed5ce75a94d5fcca94315378839b3ad240f314

# podman run --name c1 fedora true; podman commit -q c1 i1;podman rm c1;podman history i1
ece3f40107c9a6442e3694f88fc0fb547e58fa322c21e6b4c800b256cd8b69db
5ca5150e8a3e6a96e665b82364c73d6f839959ae76b2406235c79deea1e1bb7d
ID             CREATED         CREATED BY                                      SIZE      COMMENT
ece3f40107c9   2 seconds ago   /bin/sh                                         2.56kB
289289d1a15b   7 days ago      /bin/sh -c #(nop) CMD ["/bin/bash"]             2.56kB
<missing>      7 days ago      /bin/sh -c #(nop) ADD file:d3cc5fb69dcca3e...   326.1MB
<missing>      7 days ago      /bin/sh -c #(nop) ENV DISTTAG=f30-updates-...   326.1MB
<missing>      5 months ago    /bin/sh -c #(nop) LABEL maintainer=Clement...   326.1MB

# podman run --name c2 i1;podman commit -q c2 i2;podman rm c2;podman history i2
63d24bb7e5bf5c604491e1a8db0263d8f20785af463a4127d568127902a90f5b
a376589461d608f27c9e0c7339c4671722a8235c3e1ae767525e23c38814178e
ID             CREATED         CREATED BY                                      SIZE      COMMENT
63d24bb7e5bf   1 second ago    /bin/sh                                         1.024kB
<missing>      7 seconds ago   /bin/sh                                         2.56kB
289289d1a15b   7 days ago      /bin/sh -c #(nop) CMD ["/bin/bash"]             2.56kB
<missing>      7 days ago      /bin/sh -c #(nop) ADD file:d3cc5fb69dcca3e...   326.1MB
<missing>      7 days ago      /bin/sh -c #(nop) ENV DISTTAG=f30-updates-...   326.1MB
<missing>      5 months ago    /bin/sh -c #(nop) LABEL maintainer=Clement...   326.1MB

Now look at the second (non-header) line of the second history command. I expected to see ece3..., the ID of i1. It now shows as <missing>.

podman-1.4.0-2.fc30.x86_64 and this time, 100% reproducible at least over about a dozen attempts.

[edsantiago]

@jwhonce this explains why we couldn't reproduce #980 on a clean system.

[mheon]

The two images in question, centos and fedora, don't share any layers... I'm not sure exactly what could be going on here

[edsantiago]

Fun addition:

# podman rmi centos
61b442687d681ef80a7b1ae148ed5ce75a94d5fcca94315378839b3ad240f314

# podman history i2
ID             CREATED             CREATED BY                                      SIZE      COMMENT
63d24bb7e5bf   About an hour ago   /bin/sh                                         1.024kB
ece3f40107c9   About an hour ago   /bin/sh                                         2.56kB
289289d1a15b   7 days ago          /bin/sh -c #(nop) CMD ["/bin/bash"]             2.56kB
<missing>      7 days ago          /bin/sh -c #(nop) ADD file:d3cc5fb69dcca3e...   326.1MB
<missing>      7 days ago          /bin/sh -c #(nop) ENV DISTTAG=f30-updates-...   326.1MB
<missing>      5 months ago        /bin/sh -c #(nop) LABEL maintainer=Clement...   326.1MB

(it's fixed!)

[edsantiago]

Oops - one confounding factor: it's not just centos. It's the combination of centos and rhel:

# podman images -a
REPOSITORY                              TAG      IMAGE ID       CREATED       SIZE
docker.io/library/fedora                latest   289289d1a15b   7 days ago    326 MB
registry.access.redhat.com/rhel7/rhel   latest   7e79b89d206c   2 weeks ago   214 MB
docker.io/stackbrew/centos              latest   61b442687d68   3 years ago   204 MB

Only when both are present does the history command show <missing>; if I rmi rhel or centos, history works as expected. I'm sorry for not noticing this earlier.

[mheon]

@mtrmac Any ideas here?

[mtrmac]

Lots of heuristics in libpod/image… One thing that quickly jumps at me is
https://github.com/containers/libpod/blob/3cabd81045c25172786a133c538fe97b5ab83c14/libpod/image/image.go#L734 is using an interestingImages index to index into images, but I haven’t read it all in detail nor tried to trace it.

(FWIW: the recorded history does not actually store image IDs; there is a record of the commands, and there is a sequence of layers. So, the history never records e.g. the ece3f40107c9 image ID; that ID is determined by locally finding an image that has the same sequence of layers - and there may be several of them, with different configs == different image IDs; or they can be no images matching the layer sequence.)

[rhatdan]

@edsantiago @mtrmac What should we do with this issue?

[edsantiago]

Issue still present in 1.5.0.

What should we do? I don't know. I don't especially like it; I think podman history should give accurate and consistent results. But it's been almost two months and nobody else is complaining or even monitoring this issue.

[github-actions]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[rhatdan]

Here we are nearly three months later and in the same state...

[baude]

@vrothberg PTAL