Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

overlayfs fails to run container with a strange file checksum error #10180

Closed
tatsuya6502 opened this issue Jan 19, 2015 · 76 comments
Closed

overlayfs fails to run container with a strange file checksum error #10180

tatsuya6502 opened this issue Jan 19, 2015 · 76 comments

Comments

@tatsuya6502
Copy link

@tatsuya6502 tatsuya6502 commented Jan 19, 2015

I started to use "overlay" driver today and seeing odd behaviors in my containers (CentOS 6).

  1. yum install <pkg> in Dockerfile always fails, but running it again in the same container always succeeds.
  2. tail -f <file> doesn't show appended contents

For 1, this simple Dockerfile always fails at second yum install (for perl)

FROM centos:centos6

RUN yum install -y sudo
RUN yum install -y perl

with this error:

Step 2 : RUN yum install -y perl
 ---> Running in 21669e152088
...
Rpmdb checksum is invalid: dCDPT(pkg checksums): perl-version.x86_64 3:0.77-136.el6_6.1 - u

INFO[0018] The command [/bin/sh -c yum install -y perl] returned a non-zero code: 1

However, running the same command twice in the same container always succeeds.

FROM centos:centos6

RUN yum install -y sudo 
RUN (yum install -y perl || yum install -y perl)
Step 2 : RUN (yum install -y perl || yum install -y perl)
 ---> Running in 5bf1177b4437
...

Rpmdb checksum is invalid: dCDPT(pkg checksums): perl-version.x86_64 3:0.77-136.el6_6.1 - u
  Installing : 4:perl-5.10.1-136.el6_6.1.x86_64                             6/6
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirror.vastspace.net
 * extras: mirror.vastspace.net
 * updates: mirror.vastspace.net
Package 4:perl-5.10.1-136.el6_6.1.x86_64 already installed and latest version
Nothing to do
 ---> 2671034175e9
Removing intermediate container 5bf1177b4437
Successfully built 2671034175e9

Even though I change perl to something else, I get the same behavior.

For 2, I noticed that doing tail -f <application.log> first shows last 10 lines of the log file, then it doesn't show any appended contents. If I press Ctrl+C and redo, it will show the most recent contents.

I have only tried the official centos:centos6 image so far, and the above behavior is 100% reproducible. I didn't try other images.

Docker Host:

CoreOS latest alpha with ext4 root file system and "overlay" driver. I'm running it on EC2, and recreated it once but the problem persists.

core@core-01 ~ $ cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=561.0.0
VERSION_ID=561.0.0
BUILD_ID=
PRETTY_NAME="CoreOS 561.0.0"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Docker Version and Info:

core@core-01 ~ $ docker --version
Docker version 1.4.1, build 5bc2ff8-dirty

core@core-01 ~ $ docker info
Containers: 1
Images: 4
Storage Driver: overlay
Execution Driver: native-0.2
Kernel Version: 3.18.2
Operating System: CoreOS 561.0.0
CPUs: 1
Total Memory: 3.68 GiB
Name: core-01
ID: TM2L:NIZI:CUGT:EUKX:ACP3:EOFX:YQVR:CTL7:R7L2:ILQF:6E3H:HAHQ

Kernel Config:

core@core-01 ~ $ wget https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh
...
2015-01-19 11:09:00 (133 MB/s) - 'check-config.sh' saved [4476/4476]

core@core-01 ~ $ chmod a+x check-config.sh
core@core-01 ~ $ ./check-config.sh
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_MACVLAN: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_CGROUP_PERF: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
    - CONFIG_EXT4_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
@jessfraz
Copy link
Contributor

@jessfraz jessfraz commented Jan 19, 2015

I was under the impression coreos used btrfs as the backing filesystem.

On Mon, Jan 19, 2015 at 3:16 AM, Tatsuya Kawano notifications@github.com
wrote:

I started to use "overlay" driver today and seeing odd behaviors in my
containers (CentOS 6).

  1. yum install in Dockerfile always fails, but running it again in the
    same container always succeeds.
  2. tail -f doesn't show appended contents

For 1, this simple Dockerfile always fails at second yum install (for
perl)

FROM centos:centos6

RUN yum install -y sudo
RUN yum install -y perl

with this error:

Step 2 : RUN yum install -y perl
---> Running in 21669e152088
...
Rpmdb checksum is invalid: dCDPT(pkg checksums): perl-version.x86_64 3:0.77-136.el6_6.1 - u

INFO[0018] The command [/bin/sh -c yum install -y perl] returned a non-zero code: 1

However, running the same command twice in the same container always
succeeds.

FROM centos:centos6

RUN yum install -y sudo
RUN (yum install -y perl || yum install -y perl)

Step 2 : RUN (yum install -y perl || yum install -y perl)
---> Running in 5bf1177b4437
...

Rpmdb checksum is invalid: dCDPT(pkg checksums): perl-version.x86_64 3:0.77-136.el6_6.1 - u
Installing : 4:perl-5.10.1-136.el6_6.1.x86_64 6/6
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile

  • base: mirror.vastspace.net
  • extras: mirror.vastspace.net
  • updates: mirror.vastspace.net
    Package 4:perl-5.10.1-136.el6_6.1.x86_64 already installed and latest version
    Nothing to do
    ---> 2671034175e9
    Removing intermediate container 5bf1177b4437
    Successfully built 2671034175e9

Even though I change perl to something else, I get the same behavior.

For 2, I noticed that doing tail -f <application.log> first shows last 10
lines of the log file, then it doesn't show any appended contents. If I
press Ctrl+C and redo, it will show the most recent contents.

I have only tried the official centos:centos6 image so far, and the above
behavior is 100% reproducible. I didn't try other images.

Docker Host:

CoreOS latest alpha with ext4 root file system and "overlay" driver. I'm
running it on EC2, and recreated it once but the problem persists.

core@core-01 ~ $ cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=561.0.0
VERSION_ID=561.0.0
BUILD_ID=
PRETTY_NAME="CoreOS 561.0.0"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Docker Version and Info:

core@core-01 ~ $ docker --version
Docker version 1.4.1, build 5bc2ff8-dirty

core@core-01 ~ $ docker info
Containers: 1
Images: 4
Storage Driver: overlay
Execution Driver: native-0.2
Kernel Version: 3.18.2
Operating System: CoreOS 561.0.0
CPUs: 1
Total Memory: 3.68 GiB
Name: core-01
ID: TM2L:NIZI:CUGT:EUKX:ACP3:EOFX:YQVR:CTL7:R7L2:ILQF:6E3H:HAHQ

Kernel Config:

core@core-01 ~ $ wget https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh
...
2015-01-19 11:09:00 (133 MB/s) - 'check-config.sh' saved [4476/4476]

core@core-01 ~ $ chmod a+x check-config.sh
core@core-01 ~ $ ./check-config.sh
info: reading kernel config from /proc/config.gz ...

Generally Necessary:

  • cgroup hierarchy: properly mounted [/sys/fs/cgroup]
  • CONFIG_NAMESPACES: enabled
  • CONFIG_NET_NS: enabled
  • CONFIG_PID_NS: enabled
  • CONFIG_IPC_NS: enabled
  • CONFIG_UTS_NS: enabled
  • CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
  • CONFIG_CGROUPS: enabled
  • CONFIG_CGROUP_CPUACCT: enabled
  • CONFIG_CGROUP_DEVICE: enabled
  • CONFIG_CGROUP_FREEZER: enabled
  • CONFIG_CGROUP_SCHED: enabled
  • CONFIG_MACVLAN: enabled
  • CONFIG_VETH: enabled
  • CONFIG_BRIDGE: enabled
  • CONFIG_NF_NAT_IPV4: enabled
  • CONFIG_IP_NF_FILTER: enabled
  • CONFIG_IP_NF_TARGET_MASQUERADE: enabled
  • CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
  • CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
  • CONFIG_NF_NAT: enabled
  • CONFIG_NF_NAT_NEEDED: enabled
  • CONFIG_POSIX_MQUEUE: enabled

Optional Features:

  • CONFIG_MEMCG_SWAP: enabled
  • CONFIG_RESOURCE_COUNTERS: enabled
  • CONFIG_CGROUP_PERF: enabled
  • Storage Drivers:
    • "aufs":
      • CONFIG_AUFS_FS: missing
      • CONFIG_EXT4_FS_POSIX_ACL: enabled
      • CONFIG_EXT4_FS_SECURITY: enabled
    • "btrfs":
      • CONFIG_BTRFS_FS: enabled
    • "devicemapper":
      • CONFIG_BLK_DEV_DM: enabled
      • CONFIG_DM_THIN_PROVISIONING: enabled
      • CONFIG_EXT4_FS: enabled
      • CONFIG_EXT4_FS_POSIX_ACL: enabled
      • CONFIG_EXT4_FS_SECURITY: enabled
    • "overlay":
      • CONFIG_OVERLAY_FS: enabled


Reply to this email directly or view it on GitHub
#10180.

@unclejack
Copy link
Contributor

@unclejack unclejack commented Jan 19, 2015

Overlay is currently experimental and not officially supported.

CentOS 6.x has user space tooling and configuration that's only supported with the provided kernel, 2.6.32. We can't provide support for custom kernels on this distribution - nor on RHEL or any other RHEL/CentOS derived distribution.

Please take a look at these comments for more information:
comment from me on the matter: #9696 (comment)
official comment from jperrin who works on CentOS: #9696 (comment)

Overlay has some known problems and I wouldn't be surprised to track down these problems to the following overlay related problems:
#9572
#9874
#10169

I'm going to close this issue now because 1) we're tracking overlay issues in multiple other issues already 2) this overlay problem is reported on an unsupported setup - CentOS 6 with custom kernel

Debian jessie with kernel 3.18.x, Ubuntu 14.04/14.10 with kernel 3.18.x and Fedora with kernel 3.18.x would be better setups for testing overlay.

@unclejack unclejack closed this Jan 19, 2015
@tatsuya6502
Copy link
Author

@tatsuya6502 tatsuya6502 commented Jan 19, 2015

I was under the impression coreos used btrfs as the backing filesystem.

Here is the release notes of the CoreOS release I'm using.

Release notes: https://coreos.com/releases/#561.0.0

  • Linux 3.18.2
  • Switch to ext4 as the default root filesystem. Existing systems using btrfs are unaffected.

More info will be found here.

CoreOS looks to move from Btrfs to overlayfs

@unclejack
Copy link
Contributor

@unclejack unclejack commented Jan 19, 2015

@tatsuya6502 I apologize, I was under the impression you're on CentOS.

@unclejack unclejack reopened this Jan 19, 2015
@jessfraz jessfraz changed the title CentOS 6 on docker + overlayfs fails to run yum install with a strange file checksum error docker + overlayfs fails to run container with a strange file checksum error Jan 19, 2015
@jessfraz jessfraz changed the title docker + overlayfs fails to run container with a strange file checksum error overlayfs fails to run container with a strange file checksum error Jan 19, 2015
@tatsuya6502
Copy link
Author

@tatsuya6502 tatsuya6502 commented Jan 19, 2015

@unclejack That's ok. I think the issue title I put was confusing.

@jessfraz
Copy link
Contributor

@jessfraz jessfraz commented Jan 19, 2015

So overlay is currently an experimental graphdriver, there are some known issues with tests failing we are working on see #9874.
I wonder if this has anything to do with those.

@tatsuya6502
Copy link
Author

@tatsuya6502 tatsuya6502 commented Jan 19, 2015

@unclejack btw, thanks for the pointers for these issues. I'll take a look.

@tatsuya6502
Copy link
Author

@tatsuya6502 tatsuya6502 commented Jan 19, 2015

@jfrazelle Thanks for the info. I think this issue may be related to the following test case as my problem only happens after switching containers. (Run a container, modify the image, stop the container, run a new container with the image, try to modify the image -> error)

#9874

--- FAIL: TestSaveAndLoadRepoFlags (0.47s)
    docker_cli_save_load_test.go:266: inspect is not the same after a save / load
@porjo
Copy link
Contributor

@porjo porjo commented Feb 27, 2015

I use CoreOS and have just switched from Btrfs to Ext4 and I having the same issue. I was also seeing errors like this:

error: rpmdb: BDB0689 Packages page 122 is on free list with type 7
error: rpmdb: BDB0061 PANIC: Invalid argument
error: db5 error(-30973) from dbcursor->c_put: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: error(-30973) adding header #163 record
error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery
error: db5 error(-30973) from dbcursor->c_close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: rpmdb: BDB0060 PANIC: fatal region error detected; run recovery
error: db5 error(-30973) from db->sync: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

I found that inserting RUN rpm --rebuilddb prior to yum install -y <pkg> resolved both errors, however that should not be necessary.

@marineam
Copy link

@marineam marineam commented Feb 27, 2015

iirc Linux 3.19 or 4.0 adds support for multiple read-only layers in overlayfs which could allow using overlayfs to handle stacking layers instead of re-copying the whole filesystem tree in every layer.

@oszi
Copy link

@oszi oszi commented Mar 1, 2015

I also have this issue with CoreOS 598 and Fedora Rawhide containers.

@wuxxin
Copy link

@wuxxin wuxxin commented Mar 8, 2015

same issue with: bare metal ubuntu trusty with:
kernel 3.18.8 and 3.19.1 both with overlay fs and ext4
docker 1.5 and 1.5.0-dev, build a78ce5c
container is fedora 21

@vaijab
Copy link
Contributor

@vaijab vaijab commented Mar 10, 2015

Same issue here as well on CoreOS with 3.18.4 kernel.

@t0mk
Copy link

@t0mk t0mk commented Mar 12, 2015

same issue, coreos 607, kernel 3.18.6, docker 1.5.0, trying to build image from centos:6

Rpmdb checksum is invalid: dCDPT(pkg checksums): php-pdo.x86_64 0:5.3.3-40.el6_6 - u

Reverting to btrfs CoreOS used to help. "Used to" because I noticed that the 607 is stable now, so no non-overlay CoreOS in the offical channels. I guess that means I can't build the centos:6 image on CoreOS now.

@alanfranz
Copy link

@alanfranz alanfranz commented Dec 10, 2016

@xiaoshengaimm post your Dockerfile. Also, please be advised that plugins=1 and yum-plugin-ovl must be employed together, not alternatively. The first command enables plugins, the second installs the plugin to support overlay.

centos 6.6 is an old centos release (two years old, I think?) and most probably the fix is not baked in that image. Use the latest centos:6 release and most probably you won't experience any issue out of the box, as long as plugins=1 is in yum.conf.

@jamshid
Copy link
Contributor

@jamshid jamshid commented Dec 23, 2016

@jperrin did something change recently with the centos builds? I swear the "yum" checksum problem with overlay2 was fixed before I made a recent docker pull of centos.

Maybe yum-utils used to include yum-plugin-ovl but it no longer does?

This is necessary for overlayfs to work, please add yum-plugin-ovl back in. I guess I should file another bug please let me know what project.

# docker info
Containers: 48
 Running: 38
 Paused: 0
 Stopped: 10
Images: 81
Server Version: 1.12.4
Storage Driver: overlay2
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-53-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 31.31 GiB
Name: 63
ID: AUQE:OS5D:5RNW:SBMA:ZMSP:M5PG:7URM:5XTE:CSN2:4JVM:5C6E:JPYG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 provider=generic
Insecure Registries:
 127.0.0.0/8
# cat | docker build -
FROM centos:7.2.1511
RUN yum update -y && yum install -y curl which initscripts net-tools bind-utils
RUN yum install -y openssh-server

Sending build context to Docker daemon 2.048 kB
Step 1 : FROM centos:7.2.1511
7.2.1511: Pulling from library/centos
a2392627bec4: Pull complete 
Digest: sha256:2632572665fc957592ae2332662c11a380fdb869eec172eaa0fa25d673a82ac6
Status: Downloaded newer image for centos:7.2.1511
 ---> feac5e0dfdb2
Step 2 : RUN yum update -y && yum install -y curl which initscripts net-tools bind-utils
...
Step 3 : RUN yum install -y openssh-server
Running transaction
  Installing : fipscheck-lib-1.4.1-5.el7.x86_64                             1/5 
  Installing : fipscheck-1.4.1-5.el7.x86_64                                 2/5 
  Installing : openssh-6.6.1p1-31.el7.x86_64                                3/5 
  Installing : tcp_wrappers-libs-7.6-77.el7.x86_64                          4/5 
  Installing : openssh-server-6.6.1p1-31.el7.x86_64                         5/5

Rpmdb checksum is invalid: dCDPT(pkg checksums): openssh.x86_64 0:6.6.1p1-31.el7 - u
 
The command '/bin/sh -c yum install -y openssh-server' returned a non-zero code: 1
@MattGat
Copy link

@MattGat MattGat commented Apr 25, 2017

I have the same issue with Docker version 17.03.1-ce, build c6d412e and CENTOS 7

RUN rpm --rebuilddb && yum install -y XXX => This solution has resolved this issue

lukaszx0 added a commit to lukaszx0/protobuf that referenced this issue May 11, 2017
This is a workaround (moby/moby#10180 (comment)) the docker issue (moby/moby#10180) which breaks protoc-artifacts build process with following error

```Rpmdb checksum is invalid: dCDPT(pkg checksums): devtoolset-1.1-elfutils.x86_64 0:0.154-6.el6 - u

The command '/bin/sh -c yum clean all && yum install -y devtoolset-1.1                    devtoolset-1.1-libstdc++-devel                    devtoolset-1.1-libstdc++-devel.i686' returned a non-zero code: 1```

moby/moby#10180 (comment)
dklawren added a commit to mozilla-conduit/docker-bmo that referenced this issue Aug 24, 2017
nehaljwani added a commit to conda/conda-docker that referenced this issue Aug 24, 2017
kinogmt added a commit to kinogmt/centos-ssh that referenced this issue Feb 14, 2018
@orodbhen
Copy link

@orodbhen orodbhen commented Apr 2, 2018

If you're unable to install the yum plugin because of dependency conflicts (e.g. an older base image), You can work around this with touch /var/lib/rpm/*. See the bug report for details.

lae added a commit to lae/ansible-role-travis-lxc that referenced this issue May 1, 2018
…n OverlayFS

Yum reports rpmdb checksum failures on OverlayFS. This plugin fixes that.
See moby/moby#10180 for more information.
robnagler added a commit to radiasoft/containers that referenced this issue May 21, 2019
nichubbard pushed a commit to nichubbard/ucesb that referenced this issue Jun 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet