Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

@sha256@sha256 strikes again! (on 0.11.1.1) #2086

Closed
wking opened this issue Jan 5, 2019 · 18 comments

Comments

Projects
None yet
8 participants
@wking
Copy link
Contributor

commented Jan 5, 2019

[//]: # kind bug

And in case Prow notices:

/kind bug

Description

As seen back in #877, but now on 0.11.1.1.

Steps to reproduce the issue:

I haven't worked out a minimal reproducer yet, but we've had a number of folks using this script (openshift/installer#933) see:

$ sudo podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8 -f '{{ index .RepoDigests 0 }}'
quay.io/openshift-release-dev/ocp-release@sha256@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777

Describe the results you expected:

I expected the RepoDigests entry to only contain a single @sha256.

Additional information you deem important (e.g. issue happens only occasionally):

Some OpenShift Installer runs do not see this problem, but some (using the same podman and update payload image) do. Runs that see it once can reproduce it reliably. So there's some way to stumble into the broken state, and once you do you stay there. But there's also a way to avoid the broken state.

Output of podman version:

@robszumski was kind enough to work through all of this, and he had:

$ podman version
Version:       0.11.1.1
Go Version:    go1.10.2
OS/Arch:       linux/amd64
$ podman info
host:
  BuildahVersion: 1.5-dev
  Conmon:
    package: podman-0.11.1.1-3.git594495d.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: ce5c8112ef76bda67e00829952505aa447b343c1-dirty'
  Distribution:
    distribution: '"rhcos"'
    version: "4.0"
  MemFree: 5730500608
  MemTotal: 8368660480
  OCIRuntime:
    package: runc-1.0.0-57.dev.git2abd837.el7.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  hostname: ip-10-0-6-138
  kernel: 3.10.0-957.1.3.el7.x86_64
  os: linux
  rootless: false
  uptime: 2h 8m 39.48s (Approximately 0.08 days)
insecure registries:
  registries: []
registries:
  registries:
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ContainerStore:
    number: 5
  GraphDriverName: overlay
  GraphOptions:
  - overlay.override_kernel_check=true
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 14
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):

AWS. Using:

$ cat /etc/os-release
NAME="Red Hat CoreOS"
VERSION="4.0"
ID="rhcos"
ID_LIKE="rhel fedora"
VERSION_ID="4.0"
PRETTY_NAME="Red Hat CoreOS 4.0"
ANSI_COLOR="0;31"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat 7"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.0"
REDHAT_SUPPORT_PRODUCT="Red Hat"
REDHAT_SUPPORT_PRODUCT_VERSION="4.0"
OSTREE_VERSION=47.235

as installed by the v0.8.0 installer. From the broken state:

$ sudo podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8
[
   {
       "Id": "e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8",
       "Digest": "sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
       "RepoTags": [
           "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
           "quay.io/openshift-release-dev/ocp-release:4.0.0-8"
       ],
       "RepoDigests": [
           "quay.io/openshift-release-dev/ocp-release@sha256@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
           "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
       ],
       "Parent": "",
       "Comment": "Release image for OpenShift",
       "Created": "2018-12-22T22:18:35.744591Z",
       "ContainerConfig": {
           "Env": [
               "OPENSHIFT_BUILD_NAME=cluster-version-operator",
               "OPENSHIFT_BUILD_NAMESPACE=ci-op-zlhfgrm2",
               "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
           ],
           "Entrypoint": [
               "/usr/bin/cluster-version-operator"
           ],
           "Labels": {
               "io.openshift.release": "4.0.0-8",
               "io.openshift.release.base-image-digest": "sha256:57ea7826cf3c73d2dd29dfff375f5c09da087cb0e6a72b998bb74a9d18adeaa7"
           }
       },
       "Version": "1.13.1",
       "Author": "",
       "Architecture": "amd64",
       "Os": "linux",
       "Size": 291664680,
       "VirtualSize": 291664680,
       "GraphDriver": {
           "Name": "overlay",
           "Data": {
               "LowerDir": "/var/lib/containers/storage/overlay/57266b1eabbfe64e1c3330b16f0534b77c76af1ba2e28d69ecbc59779785dd1d/diff:/var/lib/containers/storage/overlay/ccbac139cf89893ed2760507dc7f3e19210593e72a9a2d8f05c486e0bb77e7a6/diff:/var/lib/containers/storage/overlay/071d8bd765171080d01682844524be57ac9883e53079b6ac66707e192ea25956/diff",
               "MergedDir": "/var/lib/containers/storage/overlay/5c857d77af584de126b963367139d6546cbfe34208c92cbeee967e2ceaaaee10/merged",
               "UpperDir": "/var/lib/containers/storage/overlay/5c857d77af584de126b963367139d6546cbfe34208c92cbeee967e2ceaaaee10/diff",
               "WorkDir": "/var/lib/containers/storage/overlay/5c857d77af584de126b963367139d6546cbfe34208c92cbeee967e2ceaaaee10/work"
           }
       },
       "RootFS": {
           "Type": "layers",
           "Layers": [
               "",
               "",
               "",
               ""
           ]
       },
       "Labels": {
           "io.openshift.release": "4.0.0-8",
           "io.openshift.release.base-image-digest": "sha256:57ea7826cf3c73d2dd29dfff375f5c09da087cb0e6a72b998bb74a9d18adeaa7"
       },
       "Annotations": {},
       "ManifestType": "application/vnd.docker.distribution.manifest.v1+prettyjws",
       "User": ""
   }
]

Comparing that with a local:

$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8
$ podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8

yielded:

--- good
+++ broken
@@ -3,9 +3,11 @@
        "Id": "e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8",
        "Digest": "sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
        "RepoTags": [
+            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
            "quay.io/openshift-release-dev/ocp-release:4.0.0-8"
        ],
        "RepoDigests": [
+            "quay.io/openshift-release-dev/ocp-release@sha256@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
        ],
        "Parent": "",

With the same podman, pulling a fresh, similar image (-7) avoided the broken state:

$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-7
$ podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-7
[
    {
        "Id": "72d96fea0c4fbcc527f509963c11b7c9c9b0bf9ba5641cd18968eb9150f54464",
        "Digest": "sha256:3e70b2d2004c9335dabc24d6259c5b03739fd86eae435c191ee2b8592e86efec",
        "RepoTags": [
            "quay.io/openshift-release-dev/ocp-release:4.0.0-7"
        ],
        "RepoDigests": [
            "quay.io/openshift-release-dev/ocp-release@sha256:3e70b2d2004c9335dabc24d6259c5b03739fd86eae435c191ee2b8592e86efec"
        ],
        "Parent": "",
        "Comment": "Release image for OpenShift",
        "Created": "2018-12-15T06:41:33.392131088Z",
        "ContainerConfig": {
            "Env": [
                "OPENSHIFT_BUILD_NAME=cluster-version-operator",
                "OPENSHIFT_BUILD_NAMESPACE=ci-op-40fz390h",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Entrypoint": [
                "/usr/bin/cluster-version-operator"
            ],
            "Labels": {
                "io.openshift.release": "4.0.0-7",
                "io.openshift.release.base-image-digest": "sha256:f9585cce1889d934b6f9c8361bb1fcc532235ddc4b88ab8614be40545e6bf5d8"
            }
        },
        "Version": "1.13.1",
        "Author": "",
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 292396852,
        "VirtualSize": 292396852,
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/639b39d8f4d99083f7276b1915cc51b4c24db7e7e5cb9a0817e8b414843b105a/diff:/var/lib/containers/storage/overlay/eb0c639d1332b895a66623afad255f15d2ac507a51f4d54a8604d292660a7d45/diff:/var/lib/containers/storage/overlay/071d8bd765171080d01682844524be57ac9883e53079b6ac66707e192ea25956/diff",
                "MergedDir": "/var/lib/containers/storage/overlay/7ce4431b6f986e887ce043bab47c2e36838e460a3fa2ea7353ad5a30f85fe7c7/merged",
                "UpperDir": "/var/lib/containers/storage/overlay/7ce4431b6f986e887ce043bab47c2e36838e460a3fa2ea7353ad5a30f85fe7c7/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/7ce4431b6f986e887ce043bab47c2e36838e460a3fa2ea7353ad5a30f85fe7c7/work"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "",
                "",
                "",
                ""
            ]
        },
        "Labels": {
            "io.openshift.release": "4.0.0-7",
            "io.openshift.release.base-image-digest": "sha256:f9585cce1889d934b6f9c8361bb1fcc532235ddc4b88ab8614be40545e6bf5d8"
        },
        "Annotations": {},
        "ManifestType": "application/vnd.docker.distribution.manifest.v1+prettyjws",
        "User": ""
    }
]

Also in this space is #1139, but I don't understand RepoDigests generation well enough to know if this is a dup of that issue or not.

@mheon

This comment has been minimized.

Copy link
Collaborator

commented Jan 5, 2019

The SHA256s seem to match here, where they did not in #877 - which seems to indicate we're pulling the right image, which is good.

This could just be a display issue (or we could be setting a nonsensical name when we save the image)

@jwhonce jwhonce added this to the 1.0 milestone Jan 7, 2019

@robszumski

This comment has been minimized.

Copy link

commented Jan 7, 2019

I seem to be able to reproduce this each time I try to boot a cluster, although I have no idea why/how.

@rhatdan

This comment has been minimized.

Copy link
Member

commented Jan 8, 2019

I don't see how this could happen in libpod code, I would guess it would be in containers/image perhaps?

@mtrmac @vrothberg WDYT?

@baude

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2019

I would think it actually comes from libpod ... maybe the image side of things ... but I need a replicator to trip it. There is actually one spot where we inject sha256 into the image name. But like i said, this would be vastly easier with a reproducer because it would allow us to figure out whats really going on and add a test to prevent regressions.

@mtrmac

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2019

$ sudo podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8

   "RepoTags": [
       "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",

^^^

       "quay.io/openshift-release-dev/ocp-release:4.0.0-8"
   ],
   "RepoDigests": [
       "quay.io/openshift-release-dev/ocp-release@sha256@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
       "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
   ],

There are actually very few RepoDigests references throughout podman, pretty sure this is

repoDigests = append(repoDigests, strings.SplitN(name, ":", 2)[0]+"@"+i.Digest().String())
when Image.Names contains a name@digest value.

This is not exactly #1139 , this is just a local detail of how Names+Digest are combined into `RepoDigests; #1139 is related in the sense the code combining the two IMHO should not exist at all, because it is often enough correct but wrong in general.

@wking

This comment has been minimized.

Copy link
Contributor Author

commented Jan 8, 2019

... when Image.Names contains a name@digest value.

That certainly looks like a likely culprit to me. @robszumski has a reliable reproducer (even if it's a bit complicated for a libpod unit test ;), is there anything he can check for to dig into this possibility?

@mtrmac

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2019

I suspect a

$ podman pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8

would recreate the situation (but I didn’t try it!).

@wking

This comment has been minimized.

Copy link
Contributor Author

commented Jan 8, 2019

I suspect a

$ podman pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8

would recreate the situation (but I didn’t try it!).

Not for me:

$ podman pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777
Trying to pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777...Getting image source signatures
Copying blob sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17
 71.68 MB / 71.68 MB [=====================================================] 21s
Copying blob sha256:47e4121c7dbd4743f868526085b5bb36f4dff5cec8c1a3f992a6b7f2bf06403c
 9.57 MB / 9.57 MB [========================================================] 2s
Copying blob sha256:ca9c3b8314517310a56ef66741850ee0acce144bb241778cc13635b62fb990b0
 10.34 MB / 10.34 MB [======================================================] 9s
Copying blob sha256:0f3ada1e64e347b6c2617705d16ba9cc1abc287d7b527b979a55c287ddbde2c9
 140.15 KB / 140.15 KB [====================================================] 0s
Writing manifest to image destination
Storing signatures
e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8
Trying to pull quay.io/openshift-release-dev/ocp-release:4.0.0-8...Getting image source signatures
Skipping fetch of repeat blob sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17
Skipping fetch of repeat blob sha256:47e4121c7dbd4743f868526085b5bb36f4dff5cec8c1a3f992a6b7f2bf06403c
Skipping fetch of repeat blob sha256:ca9c3b8314517310a56ef66741850ee0acce144bb241778cc13635b62fb990b0
Skipping fetch of repeat blob sha256:0f3ada1e64e347b6c2617705d16ba9cc1abc287d7b527b979a55c287ddbde2c9
Writing manifest to image destination
Storing signatures
e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8
$ podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8
[
    {
        "Id": "e3cd6917002af26debf8052e7f1d95ef7ed912111593248cff8de9ca076569e8",
        "Digest": "sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
        "RepoTags": [
            "quay.io/openshift-release-dev/ocp-release:4.0.0-8",
            "quay.io/openshift-release-dev/ocp-release:none"
        ],
        "RepoDigests": [
            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
        ],
...
$ podman version
Version:       0.11.1.1
Go Version:    go1.10.3
Git Commit:    "594495db2637a6e485fffd8aa57c193244e16578"
Built:         Fri Jan  4 16:26:59 2019
OS/Arch:       linux/amd64

Although there is an unnecessary duplicate in RepoDigests.

Should the line you linked earlier be using reference.Parse or similar instead of its current colon splitting? Or were you saying that it should somehow be dropped altogether (although I'm not sure where that leaves RepoDigests)?

@mtrmac

This comment has been minimized.

Copy link
Collaborator

commented Jan 9, 2019

I suspect a

$ podman pull quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777
$ podman pull quay.io/openshift-release-dev/ocp-release:4.0.0-8

would recreate the situation (but I didn’t try it!).

Not for me:

$ podman inspect quay.io/openshift-release-dev/ocp-release:4.0.0-8
        "RepoTags": [
            "quay.io/openshift-release-dev/ocp-release:4.0.0-8",
            "quay.io/openshift-release-dev/ocp-release:none"
        ],
        "RepoDigests": [
            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777",
            "quay.io/openshift-release-dev/ocp-release@sha256:2390e665f4b3a2a12c2e676e7a75af6ff74f66e57242ab2f2e58b9d8e6837777"
        ],

Ah, right. That’s the other bug. In that case, the digested RepoTags value could have been created by CRI-O pulling an image using a digest reference (they do now share the image DB, don’t they?).

Although there is an unnecessary duplicate in RepoDigests.

Sure; the reported RepoDigests array, in the current implementation, has always exactly as many entries as RepoTags, all with the same digest value.

Should the line you linked earlier be using reference.Parse or similar instead of its current colon splitting?

Yes (reference.ParseNormalizedNamed to be consistent with c/image), that would help not mess up the syntax in the short term.

Or were you saying that it should somehow be dropped altogether (although I'm not sure where that leaves RepoDigests)?

Per #1139, instead of actually recording in long-term storage only a Names==RepoTags array, and a single Digest value, I think we should be recording a Names array and a RepoDigests array (actually recording the repo@digest values for every pull, whether pulled by tag or digest) — or maybe embed the repo@digest values into Names==RepoTags directly; then deprecate Digest as ambiguous. At that point the code combining Names+Digest would go away entirely.

wking added a commit to wking/libpod that referenced this issue Jan 9, 2019

libpod/image: Use ParseNormalizedNamed in RepoDigests
Avoid generating
quay.io/openshift-release-dev/ocp-release@sha256@sha256:239... and
similar when the image name is already digest-based [1].  It's not
clear exactly how we get into this state, but as shown by the unit
tests, the new code handles this case correctly (while the previous
code does not).

[1]: containers#2086

wking added a commit to wking/libpod that referenced this issue Jan 9, 2019

libpod/image: Use ParseNormalizedNamed in RepoDigests
Avoid generating
quay.io/openshift-release-dev/ocp-release@sha256@sha256:239... and
similar when the image name is already digest-based [1].  It's not
clear exactly how we get into this state, but as shown by the unit
tests, the new code handles this case correctly (while the previous
code does not).

[1]: containers#2086

Signed-off-by: W. Trevor King <wking@tremily.us>
@wking

This comment has been minimized.

Copy link
Contributor Author

commented Jan 9, 2019

Yes (reference.ParseNormalizedNamed to be consistent with c/image), that would help not mess up the syntax in the short term.

Filed as #2106.

wking added a commit to wking/libpod that referenced this issue Jan 9, 2019

libpod/image: Use ParseNormalizedNamed in RepoDigests
Avoid generating
quay.io/openshift-release-dev/ocp-release@sha256@sha256:239... and
similar when the image name is already digest-based [1].  It's not
clear exactly how we get into this state, but as shown by the unit
tests, the new code handles this case correctly (while the previous
code does not).

[1]: containers#2086

Signed-off-by: W. Trevor King <wking@tremily.us>

wking added a commit to wking/libpod that referenced this issue Jan 9, 2019

libpod/image: Use ParseNormalizedNamed in RepoDigests
Avoid generating
quay.io/openshift-release-dev/ocp-release@sha256@sha256:239... and
similar when the image name is already digest-based [1].  It's not
clear exactly how we get into this state, but as shown by the unit
tests, the new code handles this case correctly (while the previous
code does not).

[1]: containers#2086

Signed-off-by: W. Trevor King <wking@tremily.us>

rh-atomic-bot added a commit that referenced this issue Jan 9, 2019

libpod/image: Use ParseNormalizedNamed in RepoDigests
Avoid generating
quay.io/openshift-release-dev/ocp-release@sha256@sha256:239... and
similar when the image name is already digest-based [1].  It's not
clear exactly how we get into this state, but as shown by the unit
tests, the new code handles this case correctly (while the previous
code does not).

[1]: #2086

Signed-off-by: W. Trevor King <wking@tremily.us>

Closes: #2106
Approved by: rhatdan
@mheon

This comment has been minimized.

Copy link
Collaborator

commented Jan 10, 2019

@wking Can you confirm that #2106 resolved this?

@wking

This comment has been minimized.

Copy link
Contributor Author

commented Jan 10, 2019

Ah, I expected the Fixes #2086 in my topic post to auto-close this. And yeah, it should be fixed.

/close

@openshift-ci-robot

This comment has been minimized.

Copy link
Collaborator

commented Jan 10, 2019

@wking: Closing this issue.

In response to this:

Ah, I expected the Fixes #2086 in my topic post to auto-close this. And yeah, it should be fixed.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wking

This comment has been minimized.

Copy link
Contributor Author

commented Jan 31, 2019

This turned up again with Podman 1.0.0 (which includes #2106). Details:

$ sudo podman info
host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-1.0.0-1.git82e8011.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.14.0-dev, commit: 961164166a18749fd1ec33baaa1b975729c9bf33-dirty'
  Distribution:
    distribution: '"rhcos"'
    version: "4.0"
  MemFree: 5856342016
  MemTotal: 8368656384
  OCIRuntime:
    package: runc-1.0.0-57.dev.git2abd837.el7.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  hostname: ip-10-0-4-46
  kernel: 3.10.0-957.1.3.el7.x86_64
  os: linux
  rootless: false
  uptime: 24h 59m 45.06s (Approximately 1.00 days)
insecure registries:
  registries: []
registries:
  registries:
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 5
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 13
  RunRoot: /var/run/containers/storage

$ sudo podman inspect quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d
[
    {
        "Id": "0390f8fb5cc110a3de32bf4bfbd743d08b089a87ea23bb97bb16c6eded4f94b0",
        "Digest": "sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
        "RepoTags": [
            "quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
            "quay.io/openshift-release-dev/ocp-release:4.0.0-0.2"
        ],
        "RepoDigests": [
            "quay.io/openshift-release-dev/ocp-release@sha256@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
            "quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d"
        ],
        "Parent": "",
        "Comment": "Release image for OpenShift",
        "Created": "2019-01-27T17:08:50.2297719Z",
        "Config": {
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "container=oci"
            ],
            "Entrypoint": [
                "/usr/bin/cluster-version-operator"
            ],
            "Labels": {
                "io.openshift.release": "4.0.0-0.2",
                "io.openshift.release.base-image-digest": "sha256:6fab2d0267f836d79b558096dee6073125f8f0c6c682455ab3a4e36da5f496de"
            }
        },
        "Version": "1.13.1",
        "Author": "",
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 281134224,
        "VirtualSize": 281134224,
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/767ac365866b2bc0088dd3169755835702e02dd6d32b75d47c3bf2b81805b2ef/diff:/var/lib/containers/storage/overlay/0394cf35a8eab3f65e9705c89d5b217779cbc40cbbfa8bfdccd0d219b14a808b/diff:/var/lib/containers/storage/overlay/4b0cbf0d9d0ff230916734a790f47ab2adba69db44a79c8eac4c814ff4183c6d/diff:/var/lib/containers/storage/overlay/9197342671da8b555f200e47df101da5b7e38f6d9573b10bd3295ca9e5c0ae28/diff",
                "MergedDir": "/var/lib/containers/storage/overlay/abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0/merged",
                "UpperDir": "/var/lib/containers/storage/overlay/abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0/work"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "",
                "",
                "",
                "",
                ""
            ]
        },
        "Labels": {
            "io.openshift.release": "4.0.0-0.2",
            "io.openshift.release.base-image-digest": "sha256:6fab2d0267f836d79b558096dee6073125f8f0c6c682455ab3a4e36da5f496de"
        },
        "Annotations": {},
        "ManifestType": "application/vnd.docker.distribution.manifest.v1+prettyjws",
        "User": "",
        "History": [
            {
                "created": "2019-01-27T17:08:50.2297719Z",
                "comment": "Release image for OpenShift"
            },
            {
                "created": "2019-01-27T17:08:50.2297719Z"
            },
            {
                "created": "2019-01-27T17:08:50.2297719Z"
            },
            {
                "created": "2019-01-27T17:08:50.2297719Z"
            },
            {
                "created": "2019-01-27T17:08:50.2297719Z"
            }
        ]
    }
]

That looks exactly like this test, so I'm not sure what's going on. @mtrmac, anything I can collect to help debug?

/reopen

@openshift-ci-robot

This comment has been minimized.

Copy link
Collaborator

commented Jan 31, 2019

@wking: Reopened this issue.

In response to this:

This turned up again with Podman 1.0.0 (which includes #2106). Details:

$ sudo podman info
host:
 BuildahVersion: 1.6-dev
 Conmon:
   package: podman-1.0.0-1.git82e8011.el7.x86_64
   path: /usr/libexec/podman/conmon
   version: 'conmon version 1.14.0-dev, commit: 961164166a18749fd1ec33baaa1b975729c9bf33-dirty'
 Distribution:
   distribution: '"rhcos"'
   version: "4.0"
 MemFree: 5856342016
 MemTotal: 8368656384
 OCIRuntime:
   package: runc-1.0.0-57.dev.git2abd837.el7.x86_64
   path: /usr/bin/runc
   version: 'runc version spec: 1.0.0'
 SwapFree: 0
 SwapTotal: 0
 arch: amd64
 cpus: 2
 hostname: ip-10-0-4-46
 kernel: 3.10.0-957.1.3.el7.x86_64
 os: linux
 rootless: false
 uptime: 24h 59m 45.06s (Approximately 1.00 days)
insecure registries:
 registries: []
registries:
 registries:
 - registry.access.redhat.com
 - docker.io
 - registry.fedoraproject.org
 - quay.io
 - registry.centos.org
store:
 ConfigFile: /etc/containers/storage.conf
 ContainerStore:
   number: 5
 GraphDriverName: overlay
 GraphOptions: null
 GraphRoot: /var/lib/containers/storage
 GraphStatus:
   Backing Filesystem: xfs
   Native Overlay Diff: "true"
   Supports d_type: "true"
 ImageStore:
   number: 13
 RunRoot: /var/run/containers/storage

$ sudo podman inspect quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d
[
   {
       "Id": "0390f8fb5cc110a3de32bf4bfbd743d08b089a87ea23bb97bb16c6eded4f94b0",
       "Digest": "sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
       "RepoTags": [
           "quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
           "quay.io/openshift-release-dev/ocp-release:4.0.0-0.2"
       ],
       "RepoDigests": [
           "quay.io/openshift-release-dev/ocp-release@sha256@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
           "quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d"
       ],
       "Parent": "",
       "Comment": "Release image for OpenShift",
       "Created": "2019-01-27T17:08:50.2297719Z",
       "Config": {
           "Env": [
               "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
               "container=oci"
           ],
           "Entrypoint": [
               "/usr/bin/cluster-version-operator"
           ],
           "Labels": {
               "io.openshift.release": "4.0.0-0.2",
               "io.openshift.release.base-image-digest": "sha256:6fab2d0267f836d79b558096dee6073125f8f0c6c682455ab3a4e36da5f496de"
           }
       },
       "Version": "1.13.1",
       "Author": "",
       "Architecture": "amd64",
       "Os": "linux",
       "Size": 281134224,
       "VirtualSize": 281134224,
       "GraphDriver": {
           "Name": "overlay",
           "Data": {
               "LowerDir": "/var/lib/containers/storage/overlay/767ac365866b2bc0088dd3169755835702e02dd6d32b75d47c3bf2b81805b2ef/diff:/var/lib/containers/storage/overlay/0394cf35a8eab3f65e9705c89d5b217779cbc40cbbfa8bfdccd0d219b14a808b/diff:/var/lib/containers/storage/overlay/4b0cbf0d9d0ff230916734a790f47ab2adba69db44a79c8eac4c814ff4183c6d/diff:/var/lib/containers/storage/overlay/9197342671da8b555f200e47df101da5b7e38f6d9573b10bd3295ca9e5c0ae28/diff",
               "MergedDir": "/var/lib/containers/storage/overlay/abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0/merged",
               "UpperDir": "/var/lib/containers/storage/overlay/abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0/diff",
               "WorkDir": "/var/lib/containers/storage/overlay/abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0/work"
           }
       },
       "RootFS": {
           "Type": "layers",
           "Layers": [
               "",
               "",
               "",
               "",
               ""
           ]
       },
       "Labels": {
           "io.openshift.release": "4.0.0-0.2",
           "io.openshift.release.base-image-digest": "sha256:6fab2d0267f836d79b558096dee6073125f8f0c6c682455ab3a4e36da5f496de"
       },
       "Annotations": {},
       "ManifestType": "application/vnd.docker.distribution.manifest.v1+prettyjws",
       "User": "",
       "History": [
           {
               "created": "2019-01-27T17:08:50.2297719Z",
               "comment": "Release image for OpenShift"
           },
           {
               "created": "2019-01-27T17:08:50.2297719Z"
           },
           {
               "created": "2019-01-27T17:08:50.2297719Z"
           },
           {
               "created": "2019-01-27T17:08:50.2297719Z"
           },
           {
               "created": "2019-01-27T17:08:50.2297719Z"
           }
       ]
   }
]

That looks exactly like this test, so I'm not sure what's going on. @mtrmac, anything I can collect to help debug?

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wking

This comment has been minimized.

Copy link
Contributor Author

commented Jan 31, 2019

Checking /var/lib/containers/storage/overlay-images/images.json (after copying it off to a host with jq):

$ jq '.[] | select(.digest == "sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d")' /tmp/images.json
{
  "id": "0390f8fb5cc110a3de32bf4bfbd743d08b089a87ea23bb97bb16c6eded4f94b0",
  "digest": "sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
  "names": [
    "quay.io/openshift-release-dev/ocp-release@sha256:8580a118ce951dd241e4a4b73a0e5f4cda3b56088b6c1ab56ccadbf8e270fb1d",
    "quay.io/openshift-release-dev/ocp-release:4.0.0-0.2"
  ],
  "layer": "abbb738b4caef9404bbfe4bcb60d056fcce54b38549793d1e5514de9c25a12a0",
  "metadata": "{}",
  "big-data-names": [
    "manifest"
  ],
  "big-data-sizes": {
    "manifest": 3216
  },
  "big-data-digests": {
    "manifest": "sha256:8a3bcd4ae028075d16c9bca814cec87a0b4dd4bfae9950c367489ca8cdbc8073"
  },
  "created": "2019-01-27T17:08:50.2297719Z"
}

I don't see anything interesting in /var/lib/containers/storage/overlay-images/0390f8fb5cc110a3de32bf4bfbd743d08b089a87ea23bb97bb16c6eded4f94b0/manifest, but I can attach that too if it would help.

@wking

This comment has been minimized.

Copy link
Contributor Author

commented Jan 31, 2019

I can reproduce after injecting a digest name into a randomly-selected entry in my local /var/lib/containers/storage/overlay-images/images.json. Continuing to dig...

@wking

This comment has been minimized.

Copy link
Contributor Author

commented Jan 31, 2019

Ok, #2251 should take care of this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.