Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman inspect returns Error: unexpected end of JSON input #22655

Open
swogat opened this issue May 9, 2024 · 10 comments
Open

podman inspect returns Error: unexpected end of JSON input #22655

swogat opened this issue May 9, 2024 · 10 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@swogat
Copy link

swogat commented May 9, 2024

Issue Description

[root@overcloud-cephstorage-0 ~]# podman version
Version: 3.4.1-dev
API Version: 3.4.1-dev
Go Version: go1.16.7
Built: Tue Oct 19 16:11:42 2021
OS/Arch: linux/amd64

So i have a image registry hosted in a server having ip 172.25.201.68 on port 8787, i had pulled 2 images earlier and everything was working fine, then i ran the pull again and now when inspecting one of the images i get Error: unexpected end of JSON input, but for others the podman inspect is working fine. I do not see this error in some of the machines for the same image, but i see this error on few machines with the same image.

[root@overcloud-cephstorage-0 ~]# podman pull 172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo Trying to pull 172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo... Getting image source signatures Copying blob dc0deb7c6c28 skipped: already exists Copying blob 1615f362c3b9 skipped: already exists Copying blob a3ed95caeb02 done Copying blob a3ed95caeb02 done Copying blob a3ed95caeb02 done Copying blob a3ed95caeb02 done Copying blob a3ed95caeb02 done Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob 788f94ed5eb6 skipped: already exists Copying blob e2cbf3fc8258 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Writing manifest to image destination Storing signatures 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c [root@overcloud-cephstorage-0 ~]# [root@overcloud-cephstorage-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c Error: unexpected end of JSON input

Whereas the same image is working on a different node:
[root@overcloud-controller-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c [ { "Id": "9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c", "Digest": "sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf", "RepoTags": [ "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo" ], "RepoDigests": [ "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:9f5c53d6342200114593f63fe4e1321596bb9b45861abcdcb454fd3fbaae3a6f", "172.25.201.68:8787/tripleomaster/openstack-cron@sha256:de2f5e94491ebf5702cf7e1cefbce4643e0f42005cc210d817c0521c2b3effaf" ], "Parent": "", "Comment": "", "Created": "2022-02-09T05:23:45.932319128Z", ...

Steps to reproduce the issue

Steps to reproduce the issue

  1. pull the image
  2. try to inspect it

Describe the results you received

i got Error: unexpected end of JSON input when i tried to inspect the image

Describe the results you expected

podman inspect should work without any errors

podman info output

[root@overcloud-cephstorage-0 ~]# podman info
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.0-1.module_el8.7.0+1106+45480ee0.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: 235fd1687e002bf8eaa7ea2f999104be869858b4'
  cpus: 176
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: journald
  hostname: overcloud-cephstorage-0
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-365.el8.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 129442824192
  memTotal: 3246167511040
  ociRuntime:
    name: runc
    package: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.5.2
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.8-2.module_el8.7.0+1106+45480ee0.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 17185h 35m 36.45s (Approximately 716.04 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  172.25.201.68:8787:
    Blocked: false
    Insecure: true
    Location: 172.25.201.68:8787
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: 172.25.201.68:8787
  search:
  - registry.redhat.io
  - registry.access.redhat.com
  - registry.fedoraproject.org
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 26
    paused: 0
    running: 24
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 6
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.1-dev
  Built: 1634659902
  BuiltTime: Tue Oct 19 16:11:42 2021
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.4.1-dev

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

this is a tripleo openstack setup

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting
@swogat swogat added the kind/bug Categorizes issue or PR as related to a bug. label May 9, 2024
@mheon
Copy link
Member

mheon commented May 9, 2024

That's a very old Podman version. Upstream can only really support the latest version, so I can only recommend that you upgrade. If you have commercial support, you may want to open a case there.

@swogat
Copy link
Author

swogat commented May 9, 2024 •
edited

Hi,
I ran yum update podman and still facing the similar issue, please find the below outputs:
`[root@overcloud-cephstorage-0 ~]# podman pull 172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo
Trying to pull 172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo...
Getting image source signatures
Copying blob 1615f362c3b9 skipped: already exists
Copying blob a3ed95caeb02 . 0.0 b/s
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 . 0.0 b/s
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 . 0.0 b/s
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 . 0.0 b/s
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 . 0.0 b/s
Copying blob dc0deb7c6c28 skipped: already exists
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 . 0.0 b/s
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob 788f94ed5eb6 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob e2cbf3fc8258 skipped: already exists
Copying blob 1615f362c3b9 skipped: already exists
Copying blob a3ed95caeb02 .. 0.0 b/s
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 .. 0.0 b/s
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 .. 0.0 b/s
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 .. 0.0 b/s
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 .. 0.0 b/s
Copying blob dc0deb7c6c28 skipped: already exists
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 .. 0.0 b/s
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob 788f94ed5eb6 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob e2cbf3fc8258 skipped: already exists
Copying blob 1615f362c3b9 skipped: already exists
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 done
Copying blob dc0deb7c6c28 skipped: already exists
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 done
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob 788f94ed5eb6 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob e2cbf3fc8258 skipped: already exists
Writing manifest to image destination
9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c

[root@overcloud-cephstorage-0 ~]# podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c
Error: readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

[root@overcloud-cephstorage-0 ~]# podman info
host:
arch: amd64
buildahVersion: 1.33.5
cgroupControllers:

  • cpuset
  • cpu
  • cpuacct
  • blkio
  • memory
  • devices
  • freezer
  • net_cls
  • perf_event
  • net_prio
  • hugetlb
  • pids
  • rdma
    cgroupManager: systemd
    cgroupVersion: v1
    conmon:
    package: conmon-2.1.0-1.module_el8.7.0+1106+45480ee0.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: 235fd1687e002bf8eaa7ea2f999104be869858b4'
    cpuUtilization:
    idlePercent: 99.8
    systemPercent: 0.11
    userPercent: 0.09
    cpus: 176
    databaseBackend: boltdb
    distribution:
    distribution: centos
    version: "8"
    eventLogger: journald
    freeLocks: 2022
    hostname: overcloud-cephstorage-0
    idMappings:
    gidmap: null
    uidmap: null
    kernel: 4.18.0-365.el8.x86_64
    linkmode: dynamic
    logDriver: k8s-file
    memFree: 132547715072
    memTotal: 3246167511040
    networkBackend: cni
    networkBackendInfo:
    backend: cni
    dns:
    package: podman-plugins-4.9.4-0.1.module_el8+971+3d3df00d.x86_64
    path: /usr/libexec/cni/dnsname
    version: |-
    CNI dnsname plugin
    version: 1.4.0-dev
    commit: unknown
    CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0
    package: containernetworking-plugins-1.0.1-1.module_el8.6.0+944+d413f95e.x86_64
    path: /usr/libexec/cni
    ociRuntime:
    name: runc
    package: runc-1.0.2-1.module_el8.6.0+926+8bef8ae7.x86_64
    path: /usr/bin/runc
    version: |-
    runc version 1.0.2
    spec: 1.0.2-dev
    go: go1.16.7
    libseccomp: 2.5.2
    os: linux
    pasta:
    executable: ""
    package: ""
    version: ""
    remoteSocket:
    exists: true
    path: /run/podman/podman.sock
    security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
    serviceIsRemote: false
    slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.8-2.module_el8.7.0+1106+45480ee0.x86_64
    version: |-
    slirp4netns version 1.1.8
    commit: d361001f495417b880f20329121e3aa431a8f90f
    libslirp: 4.4.0
    SLIRP_CONFIG_VERSION_MAX: 3
    libseccomp: 2.5.2
    swapFree: 0
    swapTotal: 0
    uptime: 17186h 48m 58.00s (Approximately 716.08 days)
    variant: ""
    plugins:
    authorization: null
    log:
  • k8s-file
  • none
  • passthrough
  • journald
    network:
  • bridge
  • macvlan
  • ipvlan
    volume:
  • local
    registries:
    172.25.201.68:8787:
    Blocked: false
    Insecure: true
    Location: 172.25.201.68:8787
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: 172.25.201.68:8787
    PullFromMirror: ""
    search:
  • registry.redhat.io
  • registry.access.redhat.com
  • registry.fedoraproject.org
  • registry.centos.org
  • docker.io
    store:
    configFile: /etc/containers/storage.conf
    containerStore:
    number: 26
    paused: 0
    running: 24
    stopped: 2
    graphDriverName: overlay
    graphOptions:
    overlay.mountopt: nodev,metacopy=on
    graphRoot: /var/lib/containers/storage
    graphRootAllocated: 598776721408
    graphRootUsed: 28039868416
    graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
    imageCopyTmpDir: /var/tmp
    imageStore:
    number: 6
    runRoot: /run/containers/storage
    transientStore: false
    volumePath: /var/lib/containers/storage/volumes
    version:
    APIVersion: 4.9.4-dev
    Built: 1708535009
    BuiltTime: Wed Feb 21 17:03:29 2024
    GitCommit: ""
    GoVersion: go1.21.7 (Red Hat 1.21.7-1.module_el8+960+4060efbe)
    Os: linux
    OsArch: linux/amd64
    Version: 4.9.4-dev

[root@overcloud-cephstorage-0 ~]#
`

[root@overcloud-cephstorage-0 ~]# podman version
Client: Podman Engine
Version: 4.9.4-dev
API Version: 4.9.4-dev
Go Version: go1.21.7 (Red Hat 1.21.7-1.module_el8+960+4060efbe)
Built: Wed Feb 21 17:03:29 2024
OS/Arch: linux/amd64

@mheon
Copy link
Member

mheon commented May 9, 2024

What distribution are you using that is shipping -dev versions of Podman?

@mheon
Copy link
Member

mheon commented May 9, 2024

Also, we need details on the container that cannot be inspected. How was it created? Can it be deleted? Does this happen again if you delete + recreate?

@swogat
Copy link
Author

swogat commented May 9, 2024

[root@overcloud-cephstorage-0 ~]# cat /etc/os-release
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
[root@overcloud-cephstorage-0 ~]# yum repolist
repo id repo name
appstream CentOS Stream 8 - AppStream
baseos CentOS Stream 8 - BaseOS
extras CentOS Stream 8 - Extras
zabbix Zabbix Official Repository - x86_64
zabbix-non-supported Zabbix Official Repository non-supported - x86_64
[root@overcloud-cephstorage-0 ~]#

It is a openstack setup, so i have a director node which acts as a local registry, it pulls image from quay.io/tripleomaster and stores it locally, which is then pulled by this particular and other systems from the director node. I have containers running already using this image so deleting the image would affect the running container which will then lead to outage in my infra.

@swogat
Copy link
Author

swogat commented May 9, 2024

`[root@overcloud-cephstorage-0 ~]# yum provides podman
Last metadata expiration check: 1:06:19 ago on Thu 09 May 2024 02:57:02 PM UTC.
podman-3.1.0-0.13.module_el8.5.0+733+9bb5dffa.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3.1.0-0.13.module_el8.5.0+733+9bb5dffa

podman-3.3.0-0.15.module_el8.5.0+870+f792de72.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3.3.0-0.15.module_el8.5.0+870+f792de72

podman-3.3.0-0.17.module_el8.5.0+874+6db8bee3.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3.3.0-0.17.module_el8.5.0+874+6db8bee3

podman-3.3.0-2.module_el8.5.0+877+1c30e0c9.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3.3.0-2.module_el8.5.0+877+1c30e0c9

podman-3.3.0-4.module_el8.5.0+890+6b136101.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3.3.0-4.module_el8.5.0+890+6b136101

podman-3.3.0-8.module_el8.5.0+901+79ce9cba.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3.3.0-8.module_el8.5.0+901+79ce9cba

podman-3.3.1-3.module_el8.5.0+911+f19012f9.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3.3.1-3.module_el8.5.0+911+f19012f9

podman-3.3.1-7.module_el8.5.0+914+45625a54.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3.3.1-7.module_el8.5.0+914+45625a54

podman-3.3.1-9.module_el8.6.0+938+04eb0c17.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3.3.1-9.module_el8.6.0+938+04eb0c17

podman-4.0.0-0.10.module_el8.6.0+944+d413f95e.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 4.0.0-0.10.module_el8.6.0+944+d413f95e

podman-1:3.4.1-3.module_el8.6.0+954+963caf36.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 1:3.4.1-3.module_el8.6.0+954+963caf36

podman-2:4.0.2-1.module_el8.7.0+1106+45480ee0.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 2:4.0.2-1.module_el8.7.0+1106+45480ee0

podman-2:4.2.0-1.module_el8.7.0+1196+721f4eb0.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 2:4.2.0-1.module_el8.7.0+1196+721f4eb0

podman-2:4.2.0-1.module_el8.7.0+1216+b022c01d.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 2:4.2.0-1.module_el8.7.0+1216+b022c01d

podman-3:4.3.1-2.module_el8.8.0+1254+78119b6e.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3:4.3.1-2.module_el8.8.0+1254+78119b6e

podman-3:4.5.1-5.module_el8+461+0c6b54ea.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3:4.5.1-5.module_el8+461+0c6b54ea

podman-3:4.6.0-1.module_el8+567+36ff2357.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3:4.6.0-1.module_el8+567+36ff2357

podman-3:4.6.1-1.module_el8+643+8db347f4.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3:4.6.1-1.module_el8+643+8db347f4

podman-3:4.6.1-2.module_el8+653+feef7bfe.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3:4.6.1-2.module_el8+653+feef7bfe

podman-3:4.6.1-4.module_el8+664+4072b3ae.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3:4.6.1-4.module_el8+664+4072b3ae

podman-3:4.6.1-5.module_el8+712+4cd1bd69.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3:4.6.1-5.module_el8+712+4cd1bd69

podman-3:4.8.3-1.module_el8+804+f131391c.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3:4.8.3-1.module_el8+804+f131391c

podman-3:4.9.2-1.module_el8+899+3d7192e3.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3:4.9.2-1.module_el8+899+3d7192e3

podman-3:4.9.4-0.1.module_el8+971+3d3df00d.x86_64 : Manage Pods, Containers and Container Images
Repo : @System
Matched from:
Provide : podman = 3:4.9.4-0.1.module_el8+971+3d3df00d

podman-3:4.9.4-0.1.module_el8+971+3d3df00d.x86_64 : Manage Pods, Containers and Container Images
Repo : appstream
Matched from:
Provide : podman = 3:4.9.4-0.1.module_el8+971+3d3df00d

`

@swogat
Copy link
Author

swogat commented May 9, 2024

These are the errors i am getting during deployment:

2024-05-10 01:39:55.049902 | 48d539a1-1679-a34d-cfef-00000000ef41 | FATAL | Pre-fetch all the containers | overcloud-cephstorage-0 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: Error: readObjectStart: expect { or n, but found \u0000, error found in #0 byte of ...||..., bigger context ...||...\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-10 01:39:55.052316 | 48d539a1-1679-a34d-cfef-00000000ef41 | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:16:14.628960 | 1.60s 2024-05-10 01:39:55.056783 | 48d539a1-1679-a34d-cfef-00000000ef41 | TIMING | Pre-fetch all the containers | overcloud-cephstorage-0 | 0:16:14.633414 | 1.61s 2024-05-10 01:39:55.059263 | 48d539a1-1679-a34d-cfef-00000000eca7 | SKIPPED | Fail if container image is undefined | overcloud-controller-no-ceph-3 2024-05-10 01:39:55.060612 | 48d539a1-1679-a34d-cfef-00000000eca7 | TIMING | tripleo_container_tag : Fail if container image is undefined | overcloud-controller-no-ceph-3 | 0:16:14.637259 | 0.28s 2024-05-10 01:39:55.062934 | 48d539a1-1679-a34d-cfef-00000000ef41 | FATAL | Pre-fetch all the containers | overcloud-cephstorage-1 | item=172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to run ['/bin/podman', 'inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json'] ['inspect', '9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c', '--format', 'json']: time=\"2024-05-09T17:39:54Z\" level=warning msg=\"Failed to decode the keys [\\\"network.network_backend\\\"] from \\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-09T17:39:54Z\" level=warning msg=\"Failed to decode the keys [\\\"network.network_backend\\\"] from \\\"/usr/share/containers/containers.conf\\\".\"\ntime=\"2024-05-09T17:39:54Z\" level=warning msg=\"Failed to decode the keys [\\\"network.network_backend\\\"] from \\\"/usr/share/containers/containers.conf\\\".\"\nError: unexpected end of JSON input\n", "prefetch_image": "172.25.201.68:8787/tripleomaster/openstack-cron:current-tripleo"} 2024-05-10 01:39:55.065031 | 48d539a1-1679-a34d-cfef-00000000ef41 | TIMING | Pre-fetch all the containers | overcloud-cephstorage-1 | 0:16:14.641

@mheon
Copy link
Member

mheon commented May 9, 2024

Without the Podman command used to create the container, our ability to help is strictly limited.

@swogat
Copy link
Author

swogat commented May 9, 2024

But, we are not creating the container in this case. The container using this image is already running. What we are trying to do:
After we pull the image using podman pull, we get a signature for that image as below:
Copying blob a3ed95caeb02 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob 788f94ed5eb6 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob e2cbf3fc8258 skipped: already exists Writing manifest to image destination **9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c**

So we take the signature and run an podman inspect 9a0ed16c31a654d6130fa0d146d5f14e3529a0eeb1eab1d28c8594f2b2cf0f5c, which gives us the error we are getting.
The same exact command with the exact signature is working on other nodes.

@Luap99
Copy link
Member

Luap99 commented May 14, 2024

sounds like you storage is corrupted, can you remove the image? Do other images work?
If you don't care about the data try podman system reset to remove everything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mheon @swogat @Luap99