[Overlay] Value too large for defined data type

[stainboy]

After upgrade my coreos to 618, docker has changed the underlying filesystem from btrfs to extfs + overlay. Which eventually causes it fails to write any files large than 2.5GB in the containers.

Any error code 27 and message "Value too large for defined data type" will be thrown if the above operation happens.

A quick and dirty workaround is to copy the large file from readonly layer to writable layer and make write operation against the new file.

Below is my os environment:

$ docker info
Containers: 1
Images: 46
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Kernel Version: 3.19.0
Operating System: CoreOS 618.0.0
CPUs: 24
Total Memory: 126 GiB
Name: cnpvg50820393.pvgl.sap.corp
ID: CGDX:RF6J:OKCI:32FL:I3TS:AXQA:GVEQ:7IVJ:HDV7:DAU6:BBXH:XHX6

[unclejack]

@stainboy Please keep in mind that Docker didn't decide on its own to use overlay, it was told to use overlay. How much free disk space have you got? Can you provide an exact sequence of steps to follow to reproduce this, please?

[stainboy]

@unclejack Thanks for the quick responding.

1. can you guide me how to change the underlying fs back to btrfs?

2. here is the repro steps,

   a. vi Dockerfile

   FROM debian
   RUN dd if=/dev/zero of=/home/dummy.bin bs=1 count=0 seek=3G
   CMD echo 'foo' >> /home/dummy.bin

   b. docker build .
   c. docker run the newly created image

RESULT

docker build .
Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon
Step 0 : FROM debian
---> c9fa20ecce88
Step 1 : RUN dd if=/dev/zero of=/home/dummy.bin bs=1 count=0 seek=3G
---> Running in 63e65b5f648f
0+0 records in
0+0 records out
0 bytes (0 B) copied, 1.898e-05 s, 0.0 kB/s
---> be6d49e2ecf7
Removing intermediate container 63e65b5f648f
Step 2 : CMD echo 'foo' >> /home/dummy.bin
---> Running in 8fc172955c0d
---> d90471f6dabb
Removing intermediate container 8fc172955c0d
Successfully built d90471f6dabb
docker run d90471f6dabb
/bin/sh: 1: cannot create /home/dummy.bin: Value too large for defined data type

[unclejack]

@stainboy Thanks for providing the steps to reproduce the problem. No, I can't help you switch it back because that's how CoreOS set things up. You should ask your distribution vendor for help with a temporary workaround.

The potential overlay bug affecting Docker will be investigated by the Docker project.

[stainboy]

@unclejack Thanks for the information. I will temporary reinstall the old coreos distribution and keep an eye on this issue.

[mischief]

i think that this is not a coreos or overlayfs problem, but a problem with the /bin/sh in debian, which is 'dash'.

see http://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Value-too-large-for-defined-data-type

[mischief]

it seems that this does work on coreos w/ btrfs driver for docker, but not on coreos w/ overlayfs driver. quite strange :/

[stainboy]

True, it works fine w/ btrfs. However, unlike what you said in the previous session, this is a generic issue that happens on all kinds of images.

Actually, in my case, I have created a kvm base image and put a 4GB size of win8.img into it. Every time I create a new container, kvm crash against overlay fs.

[xiaods]

Upgrade to kernel 4.2, Docker 1.8.2 then testing again.

[stainboy]

The issue still exists on the latest coreos 815.0.0 (kernel 4.2, docker 1.8.2)

Repro steps:

# show current path
pwd
# create dockerfile
cat > Dockerfile << EOF 
FROM debian:jessie
RUN dd if=/dev/zero of=/tmp/dummy bs=1 count=0 seek=3G
CMD echo 'foo' >> /tmp/dummy
EOF
# build
docker build -t issue:11700 .
# test
docker run --rm issue:11700
# show version
uname -r
cat /etc/os-release
docker version

Output:

core@yy ~/repo/issue-11700 $ # show current path
core@yy ~/repo/issue-11700 $ pwd
/home/core/repo/issue-11700
core@yy ~/repo/issue-11700 $ # create dockerfile
core@yy ~/repo/issue-11700 $ cat > Dockerfile << EOF
> FROM debian:jessie
> RUN dd if=/dev/zero of=/tmp/dummy bs=1 count=0 seek=3G
> CMD echo 'foo' >> /tmp/dummy
> EOF
core@yy ~/repo/issue-11700 $ # build
core@yy ~/repo/issue-11700 $ docker build -t issue:11700 .
# test
docker run --rm issue:11700
Sending build context to Docker daemon 2.048 kB
Step 0 : FROM debian:jessie
 ---> 8c00acfb0175
Step 1 : RUN dd if=/dev/zero of=/tmp/dummy bs=1 count=0 seek=3G
 ---> Running in 3c95fbf7fc28
0+0 records in
0+0 records out
0 bytes (0 B) copied, 2.9604e-05 s, 0.0 kB/s
 ---> 95133dbfe46c
Removing intermediate container 3c95fbf7fc28
Step 2 : CMD echo 'foo' >> /tmp/dummy
 ---> Running in d7d264a44a19
 ---> 9ea7daa1326b
Removing intermediate container d7d264a44a19
Successfully built 9ea7daa1326b
core@yy ~/repo/issue-11700 $ # test
core@yy ~/repo/issue-11700 $ docker run --rm issue:11700
/bin/sh: 1: cannot create /tmp/dummy: Value too large for defined data type
core@yy ~/repo/issue-11700 $ uname -r
4.2.0-coreos-r1
core@yy ~/repo/issue-11700 $ cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=815.0.0
VERSION_ID=815.0.0
BUILD_ID=
PRETTY_NAME="CoreOS 815.0.0"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"
core@yy ~/repo/issue-11700 $ docker version
Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   fae4436-dirty
 Built:        Thu Sep 24 08:01:15 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   fae4436-dirty
 Built:        Thu Sep 24 08:01:15 UTC 2015
 OS/Arch:      linux/amd64

[LK4D4]

This works on kernel 4.3.0

[LK4D4]

I tried both debian:jessie and busybox base images with 4.3.0 kernel. Going to close for now. Feel free to reopen if you still see it with newer kernels.