@sha256@sha256 strikes again! (on 0.11.1.1) #2086
Comments
openshift-ci-robot
added
the
kind/bug
|
The SHA256s seem to match here, where they did not in #877 - which seems to indicate we're pulling the right image, which is good. This could just be a display issue (or we could be setting a nonsensical name when we save the image) |
|
I seem to be able to reproduce this each time I try to boot a cluster, although I have no idea why/how. |
|
I don't see how this could happen in libpod code, I would guess it would be in containers/image perhaps? @mtrmac @vrothberg WDYT? |
|
I would think it actually comes from libpod ... maybe the image side of things ... but I need a replicator to trip it. There is actually one spot where we inject sha256 into the image name. But like i said, this would be vastly easier with a reproducer because it would allow us to figure out whats really going on and add a test to prevent regressions. |
…
^^^
There are actually very few This is not exactly #1139 , this is just a local detail of how |
That certainly looks like a likely culprit to me. @robszumski has a reliable reproducer (even if it's a bit complicated for a libpod unit test ;), is there anything he can check for to dig into this possibility? |
|
I suspect a $ podman pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8would recreate the situation (but I didn’t try it!). |
Not for me: $ podman pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777
Trying to pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777...Getting image source signatures
Copying blob sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17
71.68 MB / 71.68 MB [=====================================================] 21s
Copying blob sha256:47e4121c7dbd4743f868526085b5bb36f4dff5cec8c1a3f992a6b7f2bf06403c
9.57 MB / 9.57 MB [========================================================] 2s
Copying blob sha256:ca9c3b8314517310a56ef66741850ee0acce144bb241778cc13635b62fb990b0
10.34 MB / 10.34 MB [======================================================] 9s
Copying blob sha256:0f3ada1e64e347b6c2617705d16ba9cc1abc287d7b527b979a55c287ddbde2c9
140.15 KB / 140.15 KB [====================================================] 0s
Writing manifest to image destination
Storing signatures
e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8
Trying to pull quay.io/openshift-release-dev/ocp-release:4.0.0-8...Getting image source signatures
Skipping fetch of repeat blob sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17
Skipping fetch of repeat blob sha256:47e4121c7dbd4743f868526085b5bb36f4dff5cec8c1a3f992a6b7f2bf06403c
Skipping fetch of repeat blob sha256:ca9c3b8314517310a56ef66741850ee0acce144bb241778cc13635b62fb990b0
Skipping fetch of repeat blob sha256:0f3ada1e64e347b6c2617705d16ba9cc1abc287d7b527b979a55c287ddbde2c9
Writing manifest to image destination
Storing signatures
e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8
$ podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8
[
{
"Id": "e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8",
"Digest": "sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
"RepoTags": [
"quay.io/openshift-release-dev/ocp-release:4.0.0-8",
"quay.io/openshift-release-dev/ocp-release:none"
],
"RepoDigests": [
"quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
"quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
],
...
$ podman version
Version: 0.11.1.1
Go Version: go1.10.3
Git Commit: "594495db2637a6e485fffd8aa57c193244e16578"
Built: Fri Jan 4 16:26:59 2019
OS/Arch: linux/amd64Although there is an unnecessary duplicate in Should the line you linked earlier be using |
Ah, right. That’s the other bug. In that case, the digested
Sure; the reported
Yes (
Per #1139, instead of actually recording in long-term storage only a |
Avoid generating quay.io/openshift-release-dev/ocp-release@sha256@sha256:239... and similar when the image name is already digest-based [1]. It's not clear exactly how we get into this state, but as shown by the unit tests, the new code handles this case correctly (while the previous code does not). [1]: containers#2086
Avoid generating quay.io/openshift-release-dev/ocp-release@sha256@sha256:239... and similar when the image name is already digest-based [1]. It's not clear exactly how we get into this state, but as shown by the unit tests, the new code handles this case correctly (while the previous code does not). [1]: containers#2086 Signed-off-by: W. Trevor King <wking@tremily.us>
Filed as #2106. |
Avoid generating quay.io/openshift-release-dev/ocp-release@sha256@sha256:239... and similar when the image name is already digest-based [1]. It's not clear exactly how we get into this state, but as shown by the unit tests, the new code handles this case correctly (while the previous code does not). [1]: containers#2086 Signed-off-by: W. Trevor King <wking@tremily.us>
Avoid generating quay.io/openshift-release-dev/ocp-release@sha256@sha256:239... and similar when the image name is already digest-based [1]. It's not clear exactly how we get into this state, but as shown by the unit tests, the new code handles this case correctly (while the previous code does not). [1]: containers#2086 Signed-off-by: W. Trevor King <wking@tremily.us>
Avoid generating quay.io/openshift-release-dev/ocp-release@sha256@sha256:239... and similar when the image name is already digest-based [1]. It's not clear exactly how we get into this state, but as shown by the unit tests, the new code handles this case correctly (while the previous code does not). [1]: #2086 Signed-off-by: W. Trevor King <wking@tremily.us> Closes: #2106 Approved by: rhatdan
|
Ah, I expected the /close |
|
@wking: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
This turned up again with Podman 1.0.0 (which includes #2106). Details: $ sudo podman info
host:
BuildahVersion: 1.6-dev
Conmon:
package: podman-1.0.0-1.git82e8011.el7.x86_64
path: /usr/libexec/podman/conmon
version: 'conmon version 1.14.0-dev, commit: 961164166a18749fd1ec33baaa1b975729c9bf33-dirty'
Distribution:
distribution: '"rhcos"'
version: "4.0"
MemFree: 5856342016
MemTotal: 8368656384
OCIRuntime:
package: runc-1.0.0-57.dev.git2abd837.el7.x86_64
path: /usr/bin/runc
version: 'runc version spec: 1.0.0'
SwapFree: 0
SwapTotal: 0
arch: amd64
cpus: 2
hostname: ip-10-0-4-46
kernel: 3.10.0-957.1.3.el7.x86_64
os: linux
rootless: false
uptime: 24h 59m 45.06s (Approximately 1.00 days)
insecure registries:
registries: []
registries:
registries:
- registry.access.redhat.com
- docker.io
- registry.fedoraproject.org
- quay.io
- registry.centos.org
store:
ConfigFile: /etc/containers/storage.conf
ContainerStore:
number: 5
GraphDriverName: overlay
GraphOptions: null
GraphRoot: /var/lib/containers/storage
GraphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "true"
Supports d_type: "true"
ImageStore:
number: 13
RunRoot: /var/run/containers/storage
$ sudo podman inspect quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d
[
{
"Id": "0390f8fb5cc110a3de32bf4bfbd743d08b089a87ea23bb97bb16c6eded4f94b0",
"Digest": "sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
"RepoTags": [
"quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
"quay.io/openshift-release-dev/ocp-release:4.0.0-0.2"
],
"RepoDigests": [
"quay.io/openshift-release-dev/ocp-release@sha256@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
"quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d"
],
"Parent": "",
"Comment": "Release image for OpenShift",
"Created": "2019-01-27T17:08:50.2297719Z",
"Config": {
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=oci"
],
"Entrypoint": [
"/usr/bin/cluster-version-operator"
],
"Labels": {
"io.openshift.release": "4.0.0-0.2",
"io.openshift.release.base-image-digest": "sha256:6fab2d0267f836d79b558096dee6073125f8f0c6c682455ab3a4e36da5f496de"
}
},
"Version": "1.13.1",
"Author": "",
"Architecture": "amd64",
"Os": "linux",
"Size": 281134224,
"VirtualSize": 281134224,
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/var/lib/containers/storage/overlay/767ac365866b2bc0088dd3169755835702e02dd6d32b75d47c3bf2b81805b2ef/diff:/var/lib/containers/storage/overlay/0394cf35a8eab3f65e9705c89d5b217779cbc40cbbfa8bfdccd0d219b14a808b/diff:/var/lib/containers/storage/overlay/4b0cbf0d9d0ff230916734a790f47ab2adba69db44a79c8eac4c814ff4183c6d/diff:/var/lib/containers/storage/overlay/9197342671da8b555f200e47df101da5b7e38f6d9573b10bd3295ca9e5c0ae28/diff",
"MergedDir": "/var/lib/containers/storage/overlay/abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0/merged",
"UpperDir": "/var/lib/containers/storage/overlay/abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0/diff",
"WorkDir": "/var/lib/containers/storage/overlay/abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0/work"
}
},
"RootFS": {
"Type": "layers",
"Layers": [
"",
"",
"",
"",
""
]
},
"Labels": {
"io.openshift.release": "4.0.0-0.2",
"io.openshift.release.base-image-digest": "sha256:6fab2d0267f836d79b558096dee6073125f8f0c6c682455ab3a4e36da5f496de"
},
"Annotations": {},
"ManifestType": "application/vnd.docker.distribution.manifest.v1+prettyjws",
"User": "",
"History": [
{
"created": "2019-01-27T17:08:50.2297719Z",
"comment": "Release image for OpenShift"
},
{
"created": "2019-01-27T17:08:50.2297719Z"
},
{
"created": "2019-01-27T17:08:50.2297719Z"
},
{
"created": "2019-01-27T17:08:50.2297719Z"
},
{
"created": "2019-01-27T17:08:50.2297719Z"
}
]
}
]That looks exactly like this test, so I'm not sure what's going on. @mtrmac, anything I can collect to help debug? /reopen |
|
@wking: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Checking $ jq '.[] | select(.digest == "sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d")' /tmp/images.json
{
"id": "0390f8fb5cc110a3de32bf4bfbd743d08b089a87ea23bb97bb16c6eded4f94b0",
"digest": "sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
"names": [
"quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
"quay.io/openshift-release-dev/ocp-release:4.0.0-0.2"
],
"layer": "abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0",
"metadata": "{}",
"big-data-names": [
"manifest"
],
"big-data-sizes": {
"manifest": 3216
},
"big-data-digests": {
"manifest": "sha256:8a3bcd4ae028075d16c9bca814cec87a0b4dd4bfae9950c367489ca8cdbc8073"
},
"created": "2019-01-27T17:08:50.2297719Z"
}I don't see anything interesting in |
|
I can reproduce after injecting a digest name into a randomly-selected entry in my local |
|
Ok, #2251 should take care of this. |
github-actions
bot
added
the
locked - please file new issue/PR
[//]: # kind bug
And in case Prow notices:
/kind bug
Description
As seen back in #877, but now on 0.11.1.1.
Steps to reproduce the issue:
I haven't worked out a minimal reproducer yet, but we've had a number of folks using this script (openshift/installer#933) see:
Describe the results you expected:
I expected the
RepoDigestsentry to only contain a single@sha256.Additional information you deem important (e.g. issue happens only occasionally):
Some OpenShift Installer runs do not see this problem, but some (using the same
podmanand update payload image) do. Runs that see it once can reproduce it reliably. So there's some way to stumble into the broken state, and once you do you stay there. But there's also a way to avoid the broken state.Output of
podman version:@robszumski was kind enough to work through all of this, and he had:
Additional environment details (AWS, VirtualBox, physical, etc.):
AWS. Using:
as installed by the v0.8.0 installer. From the broken state:
Comparing that with a local:
yielded:
With the same
podman, pulling a fresh, similar image (-7) avoided the broken state:Also in this space is #1139, but I don't understand
RepoDigestsgeneration well enough to know if this is a dup of that issue or not.The text was updated successfully, but these errors were encountered: