docker fails with overlay storage #10169

simonthum · 2015-01-18T21:02:50Z

Hi,

I'm on gentoo docker 1.4.1 (bare if old intel metal), and when loading the official php repos (see below) docker fails so miserably the daemon process stops. It does work when using the standard (dm) backend.

Relevant portion of docker.log:

time="2015-01-18T20:23:09+01:00" level="info" msg="+job trust_key_check(/library/php)" 
time="2015-01-18T20:23:09+01:00" level="info" msg="-job trust_key_check(/library/php) = OK (0)" 
time="2015-01-18T20:23:10+01:00" level="warning" msg="image layer checksum mismatch: computed \"tarsum.dev+sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\", expected \"tarsum.dev+sha256:324d4cf44ee7daa46266c1df830c61a7df615c0632176a339e7310e34723d67a\"" 
time="2015-01-18T20:23:57+01:00" level="error" msg="Error from V2 registry: operation not supported" 
Error pulling image (5.6-cli) from php, operation not supported
time="2015-01-18T20:24:00+01:00" level="info" msg="-job pull(php, 5.6-cli) = ERR (1)" 
Error pulling image (5.6-cli) from php, operation not supported
time="2015-01-18T20:24:00+01:00" level="info" msg="-job build() = ERR (1)" 
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x6f9d9c]

goroutine 67 [running]:
runtime.panic(0xa9c2e0, 0x10dec53)
        /usr/lib/go/src/pkg/runtime/panic.c:279 +0xf5
bufio.(*Writer).flush(0xc2081b6ac0, 0x0, 0x0)
        /usr/lib/go/src/pkg/bufio/bufio.go:530 +0xdc
bufio.(*Writer).Flush(0xc2081b6ac0, 0x0, 0x0)
        /usr/lib/go/src/pkg/bufio/bufio.go:519 +0x39
net/http.(*response).Flush(0xc2081d28c0)
        /usr/lib/go/src/pkg/net/http/server.go:1023 +0x4c
github.com/docker/docker/utils.(*WriteFlusher).Write(0xc208111830, 0xc2086169a0, 0xd7, 0x154, 0xd7, 0x0, 0x0)
        /var/tmp/portage/app-emulation/docker-1.4.1/work/docker-1.4.1/.gopath/src/github.com/docker/docker/utils/utils.go:265 +0x142
github.com/docker/docker/engine.(*Output).Write(0xc2081b7140, 0xc2086169a0, 0xd7, 0x154, 0x0, 0x0, 0x0)
        /var/tmp/portage/app-emulation/docker-1.4.1/work/docker-1.4.1/.gopath/src/github.com/docker/docker/engine/streams.go:90 +0x183
github.com/docker/docker/engine.(*Output).Write(0xc20834e180, 0xc2086169a0, 0xd7, 0x154, 0x0, 0x0, 0x0)
        /var/tmp/portage/app-emulation/docker-1.4.1/work/docker-1.4.1/.gopath/src/github.com/docker/docker/engine/streams.go:90 +0x183

On the command line it looks like this:

Sending build context to Docker daemon 16.38 MB
Sending build context to Docker daemon 
Step 0 : FROM php:5.6-cli
php:5.6-cli: The image you are pulling has been verified

511136ea3c5a: Pull complete 
511136ea3c5a: Download complete 
16386e29a1f4: Error downloading dependent layers 
96957ad89738: Download complete 
00bd6c82bd11: Download complete 
1c45f38e55c2: Download complete 
f11ad85ffa51: Download complete 
14d29ce05bbb: Download complete 
2ce0403d50fa: Download complete 
2706a4a38771: Downloading [==>                                                ] 8.076 MB/148.4 MB 13m23s
53da24420fb4: Download complete 
afdec00ced84: Download complete 
afdec00ced84: Error pulling image (5.6-cli) from php, operation not supported 
INFO[0053] Error pulling image (5.6-cli) from php, operation not supported

docker -D info (after restart)

Containers: 0
Images: 1
Storage Driver: overlay
Execution Driver: native-0.2
Kernel Version: 3.18.1
Operating System: Gentoo/Linux
CPUs: 2
Total Memory: 3.885 GiB
Name: redemption
ID: ZL4Y:XPIN:OJLL:MEYO:NYHG:EDRC:KE7Z:X372:4NJU:TGSX:54NG:P2O3
Debug mode (server): false
Debug mode (client): true
Fds: 9
Goroutines: 10
EventsListeners: 0
Init SHA1: 0c681e739b02c3625ceaaa3f0ad3c5027d5cd0db
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker

docker version

Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

The text was updated successfully, but these errors were encountered:

jessfraz · 2015-01-19T01:28:35Z

Can you try curling and running this script:
https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh
it
checks your kernel config.

On Sunday, January 18, 2015, simonthum notifications@github.com wrote:

Hi,

I'm on gentoo docker 1.4.1 (bare if old intel metal), and when loading the
official php repos (see below) docker fails so miserably the daemon process
stops. It does work when using the standard (dm) backend.

Relevant portion of docker.log:

time="2015-01-18T20:23:09+01:00" level="info" msg="+job trust_key_check(/library/php)"
time="2015-01-18T20:23:09+01:00" level="info" msg="-job trust_key_check(/library/php) = OK (0)"
time="2015-01-18T20:23:10+01:00" level="warning" msg="image layer checksum mismatch: computed "tarsum.dev+sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", expected "tarsum.dev+sha256:324d4cf44ee7daa46266c1df830c61a7df615c0632176a339e7310e34723d67a""
time="2015-01-18T20:23:57+01:00" level="error" msg="Error from V2 registry: operation not supported"
Error pulling image (5.6-cli) from php, operation not supported
time="2015-01-18T20:24:00+01:00" level="info" msg="-job pull(php, 5.6-cli) = ERR (1)"
Error pulling image (5.6-cli) from php, operation not supported
time="2015-01-18T20:24:00+01:00" level="info" msg="-job build() = ERR (1)"
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x6f9d9c]

goroutine 67 [running]:
runtime.panic(0xa9c2e0, 0x10dec53)
/usr/lib/go/src/pkg/runtime/panic.c:279 +0xf5
bufio.(_Writer).flush(0xc2081b6ac0, 0x0, 0x0)
/usr/lib/go/src/pkg/bufio/bufio.go:530 +0xdc
bufio.(_Writer).Flush(0xc2081b6ac0, 0x0, 0x0)
/usr/lib/go/src/pkg/bufio/bufio.go:519 +0x39
net/http.(_response).Flush(0xc2081d28c0)
/usr/lib/go/src/pkg/net/http/server.go:1023 +0x4cgithub.com/docker/docker/utils.(_WriteFlusher).Write(0xc208111830, 0xc2086169a0, 0xd7, 0x154, 0xd7, 0x0, 0x0)
/var/tmp/portage/app-emulation/docker-1.4.1/work/docker-1.4.1/.gopath/src/github.com/docker/docker/utils/utils.go:265 +0x142github.com/docker/docker/engine.(_Output).Write(0xc2081b7140, 0xc2086169a0, 0xd7, 0x154, 0x0, 0x0, 0x0)
/var/tmp/portage/app-emulation/docker-1.4.1/work/docker-1.4.1/.gopath/src/github.com/docker/docker/engine/streams.go:90 +0x183github.com/docker/docker/engine.(_Output).Write(0xc20834e180, 0xc2086169a0, 0xd7, 0x154, 0x0, 0x0, 0x0)
/var/tmp/portage/app-emulation/docker-1.4.1/work/docker-1.4.1/.gopath/src/github.com/docker/docker/engine/streams.go:90 +0x183

On the command line it looks like this:

Sending build context to Docker daemon 16.38 MB
Sending build context to Docker daemon
Step 0 : FROM php:5.6-cli
php:5.6-cli: The image you are pulling has been verified

511136ea3c5a: Pull complete
511136ea3c5a: Download complete
16386e29a1f4: Error downloading dependent layers
96957ad89738: Download complete
00bd6c82bd11: Download complete
1c45f38e55c2: Download complete
f11ad85ffa51: Download complete
14d29ce05bbb: Download complete
2ce0403d50fa: Download complete
2706a4a38771: Downloading [==> ] 8.076 MB/148.4 MB 13m23s
53da24420fb4: Download complete
afdec00ced84: Download complete
afdec00ced84: Error pulling image (5.6-cli) from php, operation not supported
INFO[0053] Error pulling image (5.6-cli) from php, operation not supported

docker -D info (after restart)

Containers: 0
Images: 1
Storage Driver: overlay
Execution Driver: native-0.2
Kernel Version: 3.18.1
Operating System: Gentoo/Linux
CPUs: 2
Total Memory: 3.885 GiB
Name: redemption
ID: ZL4Y:XPIN:OJLL:MEYO:NYHG:EDRC:KE7Z:X372:4NJU:TGSX:54NG:P2O3
Debug mode (server): false
Debug mode (client): true
Fds: 9
Goroutines: 10
EventsListeners: 0
Init SHA1: 0c681e739b02c3625ceaaa3f0ad3c5027d5cd0db
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker

docker version

Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

—
Reply to this email directly or view it on GitHub
#10169.

addhewarman · 2015-01-19T07:05:18Z

you need kernel 3.18 overlay runs well on top of that.

simonthum · 2015-01-19T09:07:54Z

As it says

Kernel Version: 3.18.1

but I will re-check the config when I'm back home.

simonthum · 2015-01-19T19:21:31Z

Ok, my check-config isn't perfect but that may not be the cause; see below.

info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_MACVLAN: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_CGROUP_PERF: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
    - CONFIG_EXT4_FS_POSIX_ACL: missing
    - CONFIG_EXT4_FS_SECURITY: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
    - CONFIG_EXT4_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: missing
    - CONFIG_EXT4_FS_SECURITY: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled

My / is ext4; I will try with the remaining flags although I don't think they're likely candidates.

jessfraz · 2015-01-19T23:39:20Z

Overlayfs is currently experimental.

You can find a few open issues here, if you think any relate to yours:
#9572
#9874
#10180

addhewarman · 2015-01-20T03:11:42Z

@simonthum you need to edit your /etc/default/docker to something like below, but please save or export your container first. in some cases it's might delete or remove all your current docker images / container

#Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -s overlay"

@jfrazelle yes it's experimental so far it's best although it's not perfect yet with aufs i am not able to use all my disk IOPS and my container limit to 10G but after i am using overlay container can use all my IOPS and disk although again it's not a best practice to be done.

simonthum · 2015-01-20T08:58:49Z

Thanks, I'll try that when I get to it.

simonthum · 2015-01-20T20:10:31Z

The dns settings didn't change anything. I assume that the exception in the log, while obviously a followup, could be handled better. Any then it seems the common them in those bug is the computation of hashes that don't match, is there anything I can do to examine hash computation in more detail?

LK4D4 · 2015-01-22T18:36:26Z

closed by #10244
If you still experiencing this problem pls open new issue

orzel · 2015-03-03T00:29:29Z

I have the same problem here on two servers. It works well on two others.. can't find anything different between those. The common setup is : gentoo, kernel 3.18.8, overlay storage backend (over ext4 on dedicated partition), go 1.4.1 and docker 1.5.0

check-config reports everything ok/'enabled'.

I tried adding "--dns 8.8.8.8 --dns 8.8.4.4" but this doesn't change anything.

addhewarman · 2015-03-03T03:02:54Z

@orzel #Use DOCKER_OPTS to modify the daemon startup options, that's dns things is only optional.
DOCKER_OPTS="-s overlay"

orzel · 2015-03-03T12:15:33Z

@addhewarman you misunderstood. I 'm using the overlay storage backend since the very beginning, and i know how to configure docker. The ticket is about docker failing when using overlay. You suggested adding those dns stuff, and i was just reporting that it would not change the bug.

simonthum · 2015-03-04T16:22:04Z

@orzel I also still have the problem with 1.5.1, but as @LK4D4 said you should make a new bug, perhaps as this one is associated with the grace fix to hash failure. I am on a self-compiled kernel, nothing special but it could be me so I didn't open a new one yet.

orzel · 2015-03-04T23:35:36Z

@simonthum yes, you probably right.

unclejack mentioned this issue Jan 19, 2015

overlayfs fails to run container with a strange file checksum error #10180

Closed

jessfraz added the area/storage/overlay label Jan 19, 2015

stevvooe mentioned this issue Jan 20, 2015

Panic when flushing http response in engine #10228

Closed

LK4D4 closed this as completed Jan 22, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker fails with overlay storage #10169

docker fails with overlay storage #10169

simonthum commented Jan 18, 2015

jessfraz commented Jan 19, 2015

addhewarman commented Jan 19, 2015

simonthum commented Jan 19, 2015

simonthum commented Jan 19, 2015

jessfraz commented Jan 19, 2015

addhewarman commented Jan 20, 2015

simonthum commented Jan 20, 2015

simonthum commented Jan 20, 2015

LK4D4 commented Jan 22, 2015

orzel commented Mar 3, 2015

addhewarman commented Mar 3, 2015

orzel commented Mar 3, 2015

simonthum commented Mar 4, 2015

orzel commented Mar 4, 2015

docker fails with overlay storage #10169

docker fails with overlay storage #10169

Comments

simonthum commented Jan 18, 2015

jessfraz commented Jan 19, 2015

addhewarman commented Jan 19, 2015

simonthum commented Jan 19, 2015

simonthum commented Jan 19, 2015

jessfraz commented Jan 19, 2015

addhewarman commented Jan 20, 2015

simonthum commented Jan 20, 2015

simonthum commented Jan 20, 2015

LK4D4 commented Jan 22, 2015

orzel commented Mar 3, 2015

addhewarman commented Mar 3, 2015

orzel commented Mar 3, 2015

simonthum commented Mar 4, 2015

orzel commented Mar 4, 2015