buildah push triggers "layer not known" corruption for concurrent buildah commit #966

zmedico · 2018-08-29T01:28:09Z

Description

When buildah commit is used to create an image while buildah push is running for an unrelated image, the image produced by buildah commit has "layer not known" corruption after the push completes for the other image. I have verified that the patch from containers/storage#210 corrects the problem.

Steps to reproduce the issue:

Start a buildah push command for a large image.
Use buildah from, buildah commit, and buildah rm to create an unrelated image.
When the buildah push command from step 1 completes, the unrelated image in step 2 has "layer not known corruption".

Describe the results you received:

The image size shows as "-1 B" in the buildah images display, and buildah inspect produces an error like this:

error reading build object "1689c0c7aed8": error reading image: error importing build settings from image "1689c0c7aed8": error instantiating image: layer not known

Describe the results you expected:

The "layer not known" error should not occur.

Output of rpm -q buildah or apt list buildah:

I'm using buildah v1.3 built from source on Gentoo Linux.

# emerge -pv buildah --nodeps

These are the packages that would be merged, in order:

[ebuild   R   ~] app-emulation/buildah-1.3::gentoo  USE="-ostree (-selinux)" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB

Output of buildah version:

Version:         1.3
Go Version:      go1.9.4
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.3.1
libcni Version:  v0.6.0
Git Commit:      4888163
Built:           Tue Aug  7 01:10:01 2018
OS/Arch:         linux/amd64

Output of cat /etc/*release:

Gentoo Base System release 2.4.1
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo/Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"

Output of uname -a:

Linux releng3.localdomain 4.14.24-vanilla-docker-1 #1 SMP Wed Mar 7 18:29:49 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E7- 8837 @ 2.67GHz GenuineIntel GNU/Linux

Output of cat /etc/containers/storage.conf:

cat: /etc/containers/storage.conf: No such file or directory

The text was updated successfully, but these errors were encountered:

Add patch for "layer not known" corruption issue 966. See: containers/buildah#966 Package-Manager: Portage-2.3.48, Repoman-2.3.10

TomSweeneyRedHat · 2018-08-29T13:19:12Z

Thanks for the heads up @zmedico , and better yet, for providing the fix! I've just vendored container/storage in #969, that should hopefully fix things up.

TomSweeneyRedHat · 2019-04-08T20:25:07Z

Reopening for @EmilienM

EmilienM · 2019-04-08T20:27:00Z

Still hitting that bug:

error instantiating image for "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]192.168.24.1:8787/tripleomaster/centos-binary-heat-api-cfn:current-tripleo": layer not known

+ podman ps --all --size
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES  SIZE


+ podman images
REPOSITORY                                                              TAG                                      IMAGE ID       CREATED         SIZE
192.168.24.1:8787/tripleomaster/centos-binary-cinder-api                current-tripleo-updated-20190406050831   1f285e4101cf   6 minutes ago   1.33 GB
192.168.24.1:8787/tripleomaster/centos-binary-ceilometer-notification   current-tripleo-updated-20190406050831   b7d495017497   6 minutes ago   832 MB
192.168.24.1:8787/tripleomaster/centos-binary-ceilometer-compute        current-tripleo-updated-20190406050831   504844b95877   6 minutes ago   859 MB
192.168.24.1:8787/tripleomaster/centos-binary-ceilometer-central        current-tripleo-updated-20190406050831   5635f6042275   6 minutes ago   859 MB
192.168.24.1:8787/tripleomaster/centos-binary-aodh-notifier             current-tripleo-updated-20190406050831   e629dd5554f7   7 minutes ago   857 MB
192.168.24.1:8787/tripleomaster/centos-binary-aodh-listener             current-tripleo-updated-20190406050831   1db46c2456f1   7 minutes ago   857 MB
192.168.24.1:8787/tripleomaster/centos-binary-aodh-evaluator            current-tripleo-updated-20190406050831   c76361f31ec8   7 minutes ago   857 MB
192.168.24.1:8787/tripleomaster/centos-binary-aodh-api                  current-tripleo-updated-20190406050831   08e98ea12420   7 minutes ago   856 MB
192.168.24.1:8787/tripleomaster/centos-binary-cinder-volume             current-tripleo                          7bd5d92e2340   30 hours ago    1.31 GB
192.168.24.1:8787/tripleomaster/centos-binary-cinder-backup             current-tripleo                          8bba34655437   30 hours ago    1.3 GB
192.168.24.1:8787/tripleomaster/centos-binary-cinder-api                current-tripleo                          50d5a34f1bee   31 hours ago    1.3 GB
192.168.24.1:8787/tripleomaster/centos-binary-cinder-scheduler          current-tripleo                          3ab6c873698d   31 hours ago    1.2 GB
192.168.24.1:8787/tripleomaster/centos-binary-glance-api                current-tripleo                          5a950f2d209c   31 hours ago    unable to determine size
192.168.24.1:8787/tripleomaster/centos-binary-ceilometer-compute        current-tripleo                          13ccea7931e9   31 hours ago    828 MB
192.168.24.1:8787/tripleomaster/centos-binary-heat-api-cfn              current-tripleo                          cf18ca8b05a7   31 hours ago    unable to determine size


+ podman stats --all --no-stream
ID   NAME   CPU %   MEM USAGE / LIMIT   MEM %   NET IO   BLOCK IO   PIDS


+ podman version
Version:            1.2.0
RemoteAPI Version:  1
Go Version:         go1.10.2
OS/Arch:            linux/amd64


+ podman info
host:
  BuildahVersion: 1.7.2
  Conmon:
    package: podman-1.2-2.git3bd528e.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.14.0-dev, commit: 345710c5d359e8d5b126906e24615d6a3e28c131-dirty'
  Distribution:
    distribution: '"centos"'
    version: "7"
  MemFree: 586137600
  MemTotal: 8369274880
  OCIRuntime:
    package: runc-1.0.0-60.dev.git2abd837.el7.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 8569221120
  SwapTotal: 8589930496
  arch: amd64
  cpus: 8
  hostname: standalone.localdomain
  kernel: 3.10.0-957.10.1.el7.x86_64
  os: linux
  rootless: false
  uptime: 1h 0m 8.06s (Approximately 0.04 days)
insecure registries:
  registries:
  - 192.168.24.1:8787
registries:
  registries:
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 17
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 36
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

EmilienM · 2019-04-08T20:27:30Z

ping @vrothberg @nalind for information ^

EmilienM · 2019-04-08T22:03:32Z

Note that if I run "buildah pull" instead of podman, it works fine. I wonder if it's a vendoring issue on the podman side.

vrothberg · 2019-04-09T08:21:43Z

I have a hard time understanding the issue.

How is pulling related to the initial bug description? How can we reproduce it?

EmilienM · 2019-04-09T13:42:36Z

sorry @vrothberg for the poor details I put into my comments. From what I can tell now, is that everything works fine when I manipulate the images with buildah instead of podman. I now think the vendoring isn't up to date enough in podman and something was fixed in container/storage that isn't in podman vendoring?

TomSweeneyRedHat · 2019-04-09T13:46:29Z

The other off the cuff thought is #1319 and #1361 that @rhatdan and @mtrmac did in the past month or two might also be needed in Podman?

TomSweeneyRedHat · 2019-04-09T13:46:38Z

@mheon FYI

rhatdan · 2019-04-10T13:36:39Z

@EmilienM Are you saying you are having problems with podman push?

EmilienM · 2019-04-10T13:37:48Z

@EmilienM <https://github.com/EmilienM> Are you saying you are having problems with podman push?

No, we had issues with "podman pull".

rhatdan · 2019-04-11T16:36:55Z

I guess I meant pull.

rhatdan · 2019-06-08T18:28:27Z

@EmilienM @vrothberg What is the scoop on this one? It is getting rather old, is it still an issue?

vrothberg · 2019-06-09T09:46:18Z

We don't have a reproducer, so I can't really tell if it's still an issue. One thing we need to tackle at some point is consolidating the push/pull-related code of Buildah and Podman. I planned looking into this in the context of CRI-O's port on libpod.

TomSweeneyRedHat · 2019-07-22T21:44:07Z

This one has been hanging around for a while without a reproducer and I've not heard other similar issues. I'm going to close this now. Please reopen or create a new PR as pertinent if the error arises again.

gentoo-bot pushed a commit to gentoo/gentoo that referenced this issue Aug 29, 2018

app-emulation/buildah: revbump to 1.3-r1 for issue 966

9bfb166

Add patch for "layer not known" corruption issue 966. See: containers/buildah#966 Package-Manager: Portage-2.3.48, Repoman-2.3.10

TomSweeneyRedHat mentioned this issue Aug 29, 2018

Update container/storage for lock fix #969

Closed

rhatdan closed this as completed Nov 27, 2018

TomSweeneyRedHat reopened this Apr 8, 2019

TomSweeneyRedHat closed this as completed Jul 22, 2019

github-actions bot added the locked - please file new issue/PR label Sep 15, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildah push triggers "layer not known" corruption for concurrent buildah commit #966

buildah push triggers "layer not known" corruption for concurrent buildah commit #966

zmedico commented Aug 29, 2018

TomSweeneyRedHat commented Aug 29, 2018

TomSweeneyRedHat commented Apr 8, 2019

EmilienM commented Apr 8, 2019

EmilienM commented Apr 8, 2019

EmilienM commented Apr 8, 2019

vrothberg commented Apr 9, 2019

EmilienM commented Apr 9, 2019

TomSweeneyRedHat commented Apr 9, 2019

TomSweeneyRedHat commented Apr 9, 2019

rhatdan commented Apr 10, 2019

EmilienM commented Apr 10, 2019 via email

rhatdan commented Apr 11, 2019

rhatdan commented Jun 8, 2019

vrothberg commented Jun 9, 2019

TomSweeneyRedHat commented Jul 22, 2019

buildah push triggers "layer not known" corruption for concurrent buildah commit #966

buildah push triggers "layer not known" corruption for concurrent buildah commit #966

Comments

zmedico commented Aug 29, 2018

TomSweeneyRedHat commented Aug 29, 2018

TomSweeneyRedHat commented Apr 8, 2019

EmilienM commented Apr 8, 2019

EmilienM commented Apr 8, 2019

EmilienM commented Apr 8, 2019

vrothberg commented Apr 9, 2019

EmilienM commented Apr 9, 2019

TomSweeneyRedHat commented Apr 9, 2019

TomSweeneyRedHat commented Apr 9, 2019

rhatdan commented Apr 10, 2019

EmilienM commented Apr 10, 2019 via email

rhatdan commented Apr 11, 2019

rhatdan commented Jun 8, 2019

vrothberg commented Jun 9, 2019

TomSweeneyRedHat commented Jul 22, 2019