Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to create new file after few days in my container #33497

Closed
goldyfruit opened this issue Jun 2, 2017 · 14 comments
Closed

Unable to create new file after few days in my container #33497

goldyfruit opened this issue Jun 2, 2017 · 14 comments

Comments

@goldyfruit
Copy link

goldyfruit commented Jun 2, 2017

Hi guys.

After few days, some of my containers are not able to write in some directories anymore.

Example:

root@controller002:/# touch /toto
root@controller002:/# touch /tmp/toto
touch: cannot touch '/tmp/toto': No such file or directory
root@controller002:/# touch /usr/toto
root@controller002:/# touch /usr/local/share/toto
touch: cannot touch '/usr/local/share/toto': No such file or directory
root@controller002:/# apt install strace
E: Could not open lock file /var/lib/dpkg/lock - open (2: No such file or directory)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?
root@controller002:/# touch /var/lib/dpkg/lock
touch: cannot touch '/var/lib/dpkg/lock': No such file or directory

The application needs to create sockets but it can't and it fails with a trace like that:

  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 514, in handle_one_response
    write(b'')
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 453, in write
    wfile.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe

If I restart the container, it's works. Make me think something related to overlayfs and some layers issue.

Docker version:

Containers: 51
 Running: 50
 Paused: 0
 Stopped: 1
Images: 53
Server Version: 17.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-77-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 251.7 GiB
Name: controller002
ID: PZFT:D4TX:YUBW:7EVF:UWJW:P4WC:ZE5U:6FDI:SJBU:MXPY:WQ3O:Z4G3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 171
 Goroutines: 175
 System Time: 2017-06-07T14:03:43.360752503Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
Experimental: false
Insecure Registries:
 10.129.9.165:4000
 127.0.0.0/8
Live Restore Enabled: true
# docker inspect neutron_vpnaas_agent 
[
    {
        "Id": "94728fd92408205b742ad24758e9e67eccd0f5ccf26fb4785a7ad1716080f616",
        "Created": "2017-05-04T21:06:45.722631859Z",
        "Path": "kolla_start",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 5636,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-05-12T01:50:38.499248394Z",
            "FinishedAt": "2017-05-12T01:50:28.207985293Z"
        },
        "Image": "sha256:e4f25095819760b9fa8ef819080c5a600be8f76c340244159d646660274f075e",
        "ResolvConfPath": "/var/lib/docker/containers/94728fd92408205b742ad24758e9e67eccd0f5ccf26fb4785a7ad1716080f616/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/94728fd92408205b742ad24758e9e67eccd0f5ccf26fb4785a7ad1716080f616/hostname",
        "HostsPath": "/var/lib/docker/containers/94728fd92408205b742ad24758e9e67eccd0f5ccf26fb4785a7ad1716080f616/hosts",
        "LogPath": "/var/lib/docker/containers/94728fd92408205b742ad24758e9e67eccd0f5ccf26fb4785a7ad1716080f616/94728fd92408205b742ad24758e9e67eccd0f5ccf26fb4785a7ad1716080f616-json.log",
        "Name": "/neutron_vpnaas_agent",
        "RestartCount": 0,
        "Driver": "overlay2",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/etc/localtime:/etc/localtime:ro",
                "/lib/modules:/lib/modules:ro",
                "kolla_logs:/var/log/kolla/:rw",
                "/etc/kolla/neutron-vpnaas-agent/:/var/lib/kolla/config_files/:ro",
                "/run:/run:shared",
                "neutron_metadata_socket:/var/lib/neutron/kolla/:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label=disable"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "overlay2",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf-init/diff:/var/lib/docker/overlay2/4b348e76b377fd75e0431ce39f578a50f7c522ee1ac68eaf671210d37f6377e2/diff:/var/lib/docker/overlay2/e2b345d15345c32af04f7231dc3b01f8874d8c2e726149d0d7c85763a1ec8853/diff:/var/lib/docker/overlay2/287f82d5f5cd1d07fcdf2f29f6608205f20b0eacc27351b339823ee93c8162b3/diff:/var/lib/docker/overlay2/b0a741a85499e2311c0563e1ffced8b360356221046a6d075fecd8c5c3bc2735/diff:/var/lib/docker/overlay2/9d83cd446a61e1c6351392e601a37e9da33355006acc8d1d96152c9e86726d37/diff:/var/lib/docker/overlay2/660a8d51391730ad5e78da1643854093797b1dc294538e7e214b6589d9eb08b9/diff:/var/lib/docker/overlay2/ce3609612187162dcf68eb6f2d54551df4b01ac0eb1fcb96e8252df0a19fdec2/diff:/var/lib/docker/overlay2/34e7b70a75d8c6956f382660892ee1b2d9e9c129a0f400d48c52c7a4d712f9f9/diff:/var/lib/docker/overlay2/4307b68b8e43ed8703419652682700741a17b7ded5bc437d1e1ced0f971a3695/diff:/var/lib/docker/overlay2/04c80fb15cf13302a61f3ef52b3084279f6e2753993bd015f9cc9c209bf31d7f/diff:/var/lib/docker/overlay2/68a978ef28affe37dc2600b66ba82c4af77f0350e3054088753c219464214cf3/diff:/var/lib/docker/overlay2/2d3a0edc4350db44dde8f46dee9a041cdabcc59cfbd7ae8eb3a376e41b438220/diff:/var/lib/docker/overlay2/2e697fd0456059d75ceaec4593bdd2a3aac64ece4133a9a447021cef23ca2781/diff:/var/lib/docker/overlay2/10e19da47e0b7b02b9ea33023114c8036bbaa1958682afa7ddbfd74f5c458849/diff:/var/lib/docker/overlay2/98bd44fac6d8325961977e172c438941e687ce16bf56c936ba4ac1da6f6e532e/diff:/var/lib/docker/overlay2/604f5d0d4269bb89ce0d7a133754ec80c4950a732f491487e2628862af7777c9/diff:/var/lib/docker/overlay2/84ffbfd6fa1affd67e2bf9594cf69f5baa976a5f18e46a3a9737a3d032d6586a/diff:/var/lib/docker/overlay2/8a96cadc50daa85c413b0fb7064c2daaaf8f6a7ef22093f444db0993fcfbaae5/diff:/var/lib/docker/overlay2/27a174260e9a89b5028610c1583331a88157355810522cc362b094f2386ac6f8/diff:/var/lib/docker/overlay2/eb1071474699f9f9a8e0ff427deb1693f720d48eacbc33cbf16d359aa364f993/diff:/var/lib/docker/overlay2/6a6ff5ee4c353a7675ab9d36ea384aa36532331d8ff49f2ead8267a4aacd758c/diff:/var/lib/docker/overlay2/68a7f6056ae83c16e56b58f6f494f8a61b0db3d4fba2391a3c3ed3b84a8af86e/diff:/var/lib/docker/overlay2/d71fe834cba44dc724bd5d53746ed9342b9b8d1b9520ebb3da5f54cf1dc52c5b/diff:/var/lib/docker/overlay2/f13341125648bebc74a4f3c3d8a240612e66d5d7e6e9bce843acd8fc41b7a07a/diff:/var/lib/docker/overlay2/dade9d79b14e436c7c422ca9502634d00ee1131dfa0e11b5353c2b4181a685db/diff:/var/lib/docker/overlay2/6a686c78db2071da81a2966be581b9ca3ffeae67b652ba9230985863dfe0fd61/diff:/var/lib/docker/overlay2/bd8c57b02af95d2ec6311b107148e056f12749f1d2634dba676b5aa12f2a4fcd/diff:/var/lib/docker/overlay2/c834793d55f39829ab5c9d4d833ddbfb328e04eefc2dda2f458b7d82e38089b1/diff",
                "MergedDir": "/var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf/merged",
                "UpperDir": "/var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf/diff",
                "WorkDir": "/var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf/work"
            }
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "kolla_logs",
                "Source": "/var/lib/docker/volumes/kolla_logs/_data",
                "Destination": "/var/log/kolla",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/etc/localtime",
                "Destination": "/etc/localtime",
                "Mode": "ro",
                "RW": false,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/lib/modules",
                "Destination": "/lib/modules",
                "Mode": "ro",
                "RW": false,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/run",
                "Destination": "/run",
                "Mode": "shared",
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/etc/kolla/neutron-vpnaas-agent",
                "Destination": "/var/lib/kolla/config_files",
                "Mode": "ro",
                "RW": false,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "neutron_metadata_socket",
                "Source": "/var/lib/docker/volumes/neutron_metadata_socket/_data",
                "Destination": "/var/lib/neutron/kolla",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "controller002",
            "Domainname": "",
            "User": "neutron",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "KOLLA_SERVICE_NAME=neutron-vpnaas-agent",
                "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
                "PATH=/var/lib/kolla/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "KOLLA_BASE_DISTRO=ubuntu",
                "KOLLA_INSTALL_TYPE=source",
                "KOLLA_INSTALL_METATYPE=mixed",
                "DEBIAN_FRONTEND=noninteractive"
            ],
            "Cmd": [
                "kolla_start"
            ],
            "ArgsEscaped": true,
            "Image": "10.129.9.165:4000/kollaglue/ubuntu-source-neutron-vpnaas-agent:2.0.0",
            "Volumes": {
                "/etc/localtime": {},
                "/lib/modules": {},
                "/run": {},
                "/var/lib/kolla/config_files/": {},
                "/var/lib/neutron/kolla/": {},
                "/var/log/kolla/": {}
            },
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "kolla_version": "2.0.0"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "59a746826d8a6d314c82c1008589ce5e7f3edf650ade86a98beca31637c7f89f",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "2d78daf4598697de392944e0de234e4355ace1b3861d843c642930e58457a9b1",
                    "EndpointID": "e11ae8ba091fba6266f1e6b8953794021d71a47d443b85fe3f82b7b64f4267d5",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]

Please find a paste of # strace -ff docker exec neutron_vpnaas_agent touch /tmp/toto : https://paste.api-zulu.com/sehegemaye.vbs

@goldyfruit
Copy link
Author

goldyfruit commented Jun 7, 2017

Do you need more information ? I'll have to restart the container :/
I would like to provide you the most as I can !

EDIT: I've added some information.

@goldyfruit
Copy link
Author

Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.250727610Z" level=debug msg="Calling GET /_ping"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.251452165Z" level=debug msg="Calling POST /v1.27/containers/neutron_vpnaas_agent/exec"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.251567034Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"touch\",\"/tmp/toto\"],\"Detach\":false,\"DetachKeys\":\"\",\"Env\":null,\"Privileged\":false,\"Tty\":false,\"User\":\"\"}"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.252101140Z" level=debug msg="Calling POST /v1.27/exec/f47fcf854c8e5a1429b7b3867f12db59d0f40f43c4b24e5d2d6a2af0c987395a/start"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.252200744Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"touch\",\"/tmp/toto\"],\"Detach\":false,\"DetachKeys\":\"\",\"Env\":null,\"Privileged\":false,\"Tty\":false,\"User\":\"\"}"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.252287361Z" level=debug msg="starting exec command f47fcf854c8e5a1429b7b3867f12db59d0f40f43c4b24e5d2d6a2af0c987395a in container 94728fd92408205b742ad24758e9e67eccd0f5ccf26fb4785a7ad1716080f616"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.252558898Z" level=debug msg="attach: stdout: begin"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.252569586Z" level=debug msg="attach: stderr: begin"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.304161376Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"start-process\", Id:\"94728fd92408205b742ad24758e9e67eccd0f5ccf26fb4785a7ad1716080f616\", Status:0x0, Pid:\"f47fcf854c8e5a1429b7b3867f12db59d0f40f43c4b24e5d2d6a2af0c987395a\", Timestamp:(*timestamp.Timestamp)(0xc422412660)}"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.304233823Z" level=debug msg="libcontainerd: event unhandled: type:\"start-process\" id:\"94728fd92408205b742ad24758e9e67eccd0f5ccf26fb4785a7ad1716080f616\" pid:\"f47fcf854c8e5a1429b7b3867f12db59d0f40f43c4b24e5d2d6a2af0c987395a\" timestamp:<seconds:1496844098 nanos:303935087 > "
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.304304124Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"exit\", Id:\"94728fd92408205b742ad24758e9e67eccd0f5ccf26fb4785a7ad1716080f616\", Status:0x1, Pid:\"f47fcf854c8e5a1429b7b3867f12db59d0f40f43c4b24e5d2d6a2af0c987395a\", Timestamp:(*timestamp.Timestamp)(0xc422412870)}"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.304407371Z" level=debug msg="attach: stderr: end"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.304434051Z" level=debug msg="attach: stdout: end"
Jun 07 14:01:38 controller002 dockerd[3544]: time="2017-06-07T14:01:38.304752432Z" level=debug msg="Calling GET /v1.27/exec/f47fcf854c8e5a1429b7b3867f12db59d0f40f43c4b24e5d2d6a2af0c987395a/json"

@AkihiroSuda
Copy link
Member

cc @dmcgowan @stevvooe Is this known issue?

@goldyfruit
Copy link
Author

Maybe it's something fixed in kernel 4.8 or 4.10 ?

@dmcgowan
Copy link
Member

dmcgowan commented Jun 9, 2017

I don't know of any issues which would cause this. After you restart are you able to reproduce getting the container into this state? Also can you list the directory contents for each of the files which complained about the directory not existing? Is there any process in the container that could be responsible for deleting files? Also look at the mount and check whether all the lower directories still exist on the host.

@goldyfruit
Copy link
Author

goldyfruit commented Jun 9, 2017

@dmcgowan yes, I'm able to reproduce the state (sometimes it's few days, sometime it's a week).

The directories are not empty (except for /tmp), there is nothing deleting files (this issues appends in different containers on different hosts).

All directories still exist on the host.

# ls -ld /var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf/merged
drwx------ 2 root root 4096 May  4 21:06 /var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf/merged

# ls -ld /var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf/diff
drwxr-xr-x 8 root root 4096 Jun  2 19:04 /var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf/diff

# ls -ld /var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf/work
drwx------ 3 root root 4096 Jun  7 14:52 /var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf/work

# for i in $(cat /tmp/docker-bug.txt | tr ':' '\n'); do echo $i; ls -ld $i; done
/var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf-init/diff
drwxr-xr-x 4 root root 4096 May  4 21:06 /var/lib/docker/overlay2/8cd48bda0c7673046b098a11e0e4f123a2a788eab39fd6109454b75564df68cf-init/diff
/var/lib/docker/overlay2/4b348e76b377fd75e0431ce39f578a50f7c522ee1ac68eaf671210d37f6377e2/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/4b348e76b377fd75e0431ce39f578a50f7c522ee1ac68eaf671210d37f6377e2/diff
/var/lib/docker/overlay2/e2b345d15345c32af04f7231dc3b01f8874d8c2e726149d0d7c85763a1ec8853/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/e2b345d15345c32af04f7231dc3b01f8874d8c2e726149d0d7c85763a1ec8853/diff
/var/lib/docker/overlay2/287f82d5f5cd1d07fcdf2f29f6608205f20b0eacc27351b339823ee93c8162b3/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/287f82d5f5cd1d07fcdf2f29f6608205f20b0eacc27351b339823ee93c8162b3/diff
/var/lib/docker/overlay2/b0a741a85499e2311c0563e1ffced8b360356221046a6d075fecd8c5c3bc2735/diff
drwxr-xr-x 5 root root 4096 May  4 21:06 /var/lib/docker/overlay2/b0a741a85499e2311c0563e1ffced8b360356221046a6d075fecd8c5c3bc2735/diff
/var/lib/docker/overlay2/9d83cd446a61e1c6351392e601a37e9da33355006acc8d1d96152c9e86726d37/diff
drwxr-xr-x 7 root root 4096 May  4 21:06 /var/lib/docker/overlay2/9d83cd446a61e1c6351392e601a37e9da33355006acc8d1d96152c9e86726d37/diff
/var/lib/docker/overlay2/660a8d51391730ad5e78da1643854093797b1dc294538e7e214b6589d9eb08b9/diff
drwxr-xr-x 4 root root 4096 May  4 21:06 /var/lib/docker/overlay2/660a8d51391730ad5e78da1643854093797b1dc294538e7e214b6589d9eb08b9/diff
/var/lib/docker/overlay2/ce3609612187162dcf68eb6f2d54551df4b01ac0eb1fcb96e8252df0a19fdec2/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/ce3609612187162dcf68eb6f2d54551df4b01ac0eb1fcb96e8252df0a19fdec2/diff
/var/lib/docker/overlay2/34e7b70a75d8c6956f382660892ee1b2d9e9c129a0f400d48c52c7a4d712f9f9/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/34e7b70a75d8c6956f382660892ee1b2d9e9c129a0f400d48c52c7a4d712f9f9/diff
/var/lib/docker/overlay2/4307b68b8e43ed8703419652682700741a17b7ded5bc437d1e1ced0f971a3695/diff
drwxr-xr-x 9 root root 4096 May  4 21:06 /var/lib/docker/overlay2/4307b68b8e43ed8703419652682700741a17b7ded5bc437d1e1ced0f971a3695/diff
/var/lib/docker/overlay2/04c80fb15cf13302a61f3ef52b3084279f6e2753993bd015f9cc9c209bf31d7f/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/04c80fb15cf13302a61f3ef52b3084279f6e2753993bd015f9cc9c209bf31d7f/diff
/var/lib/docker/overlay2/68a978ef28affe37dc2600b66ba82c4af77f0350e3054088753c219464214cf3/diff
drwxr-xr-x 9 root root 4096 May  4 21:06 /var/lib/docker/overlay2/68a978ef28affe37dc2600b66ba82c4af77f0350e3054088753c219464214cf3/diff
/var/lib/docker/overlay2/2d3a0edc4350db44dde8f46dee9a041cdabcc59cfbd7ae8eb3a376e41b438220/diff
drwxr-xr-x 6 root root 4096 May  4 21:06 /var/lib/docker/overlay2/2d3a0edc4350db44dde8f46dee9a041cdabcc59cfbd7ae8eb3a376e41b438220/diff
/var/lib/docker/overlay2/2e697fd0456059d75ceaec4593bdd2a3aac64ece4133a9a447021cef23ca2781/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/2e697fd0456059d75ceaec4593bdd2a3aac64ece4133a9a447021cef23ca2781/diff
/var/lib/docker/overlay2/10e19da47e0b7b02b9ea33023114c8036bbaa1958682afa7ddbfd74f5c458849/diff
drwxr-xr-x 7 root root 4096 May  4 21:06 /var/lib/docker/overlay2/10e19da47e0b7b02b9ea33023114c8036bbaa1958682afa7ddbfd74f5c458849/diff
/var/lib/docker/overlay2/98bd44fac6d8325961977e172c438941e687ce16bf56c936ba4ac1da6f6e532e/diff
drwxr-xr-x 4 root root 4096 May  4 21:06 /var/lib/docker/overlay2/98bd44fac6d8325961977e172c438941e687ce16bf56c936ba4ac1da6f6e532e/diff
/var/lib/docker/overlay2/604f5d0d4269bb89ce0d7a133754ec80c4950a732f491487e2628862af7777c9/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/604f5d0d4269bb89ce0d7a133754ec80c4950a732f491487e2628862af7777c9/diff
/var/lib/docker/overlay2/84ffbfd6fa1affd67e2bf9594cf69f5baa976a5f18e46a3a9737a3d032d6586a/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/84ffbfd6fa1affd67e2bf9594cf69f5baa976a5f18e46a3a9737a3d032d6586a/diff
/var/lib/docker/overlay2/8a96cadc50daa85c413b0fb7064c2daaaf8f6a7ef22093f444db0993fcfbaae5/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/8a96cadc50daa85c413b0fb7064c2daaaf8f6a7ef22093f444db0993fcfbaae5/diff
/var/lib/docker/overlay2/27a174260e9a89b5028610c1583331a88157355810522cc362b094f2386ac6f8/diff
drwxr-xr-x 9 root root 4096 May  4 21:06 /var/lib/docker/overlay2/27a174260e9a89b5028610c1583331a88157355810522cc362b094f2386ac6f8/diff
/var/lib/docker/overlay2/eb1071474699f9f9a8e0ff427deb1693f720d48eacbc33cbf16d359aa364f993/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/eb1071474699f9f9a8e0ff427deb1693f720d48eacbc33cbf16d359aa364f993/diff
/var/lib/docker/overlay2/6a6ff5ee4c353a7675ab9d36ea384aa36532331d8ff49f2ead8267a4aacd758c/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/6a6ff5ee4c353a7675ab9d36ea384aa36532331d8ff49f2ead8267a4aacd758c/diff
/var/lib/docker/overlay2/68a7f6056ae83c16e56b58f6f494f8a61b0db3d4fba2391a3c3ed3b84a8af86e/diff
drwxr-xr-x 3 root root 4096 May  4 21:06 /var/lib/docker/overlay2/68a7f6056ae83c16e56b58f6f494f8a61b0db3d4fba2391a3c3ed3b84a8af86e/diff
/var/lib/docker/overlay2/d71fe834cba44dc724bd5d53746ed9342b9b8d1b9520ebb3da5f54cf1dc52c5b/diff
drwxr-xr-x 7 root root 4096 May  4 21:06 /var/lib/docker/overlay2/d71fe834cba44dc724bd5d53746ed9342b9b8d1b9520ebb3da5f54cf1dc52c5b/diff
/var/lib/docker/overlay2/f13341125648bebc74a4f3c3d8a240612e66d5d7e6e9bce843acd8fc41b7a07a/diff
drwxr-xr-x 3 root root 4096 May  4 20:24 /var/lib/docker/overlay2/f13341125648bebc74a4f3c3d8a240612e66d5d7e6e9bce843acd8fc41b7a07a/diff
/var/lib/docker/overlay2/dade9d79b14e436c7c422ca9502634d00ee1131dfa0e11b5353c2b4181a685db/diff
drwxr-xr-x 3 root root 4096 May  4 20:24 /var/lib/docker/overlay2/dade9d79b14e436c7c422ca9502634d00ee1131dfa0e11b5353c2b4181a685db/diff
/var/lib/docker/overlay2/6a686c78db2071da81a2966be581b9ca3ffeae67b652ba9230985863dfe0fd61/diff
drwxr-xr-x 3 root root 4096 May  4 20:24 /var/lib/docker/overlay2/6a686c78db2071da81a2966be581b9ca3ffeae67b652ba9230985863dfe0fd61/diff
/var/lib/docker/overlay2/bd8c57b02af95d2ec6311b107148e056f12749f1d2634dba676b5aa12f2a4fcd/diff
drwxr-xr-x 6 root root 4096 May  4 20:24 /var/lib/docker/overlay2/bd8c57b02af95d2ec6311b107148e056f12749f1d2634dba676b5aa12f2a4fcd/diff
/var/lib/docker/overlay2/c834793d55f39829ab5c9d4d833ddbfb328e04eefc2dda2f458b7d82e38089b1/diff
drwxr-xr-x 21 root root 4096 May  4 20:24 /var/lib/docker/overlay2/c834793d55f39829ab5c9d4d833ddbfb328e04eefc2dda2f458b7d82e38089b1/diff

@thaJeztah
Copy link
Member

I do see Live Restore Enabled: true, wonder if there's anything related to that that may cause the issue (filesystem being re-mounted after daemon is restarted e.g.?) - sorry just thinking out loud.

@goldyfruit Do you know if this occurs after the daemon was restarted?

@goldyfruit
Copy link
Author

goldyfruit commented Jun 9, 2017

@thaJeztah I don't remember but I just tested to restart the docker deamon on an another host and I don't have any issue.

You suggestion is not a bad one, how can I confirm that the filesystem are not remounted ?
The live-restore option is not recommended ?

Thanks for your time guys !

@goldyfruit
Copy link
Author

@thaJeztah I'll turned off the live-restore until we have a clear view the impact.
I'm not 100% sure that it's not the issue.

Any other thoughts guys ?

@thaJeztah
Copy link
Member

No other thoughts at this point, but keep us posted if disabling live-restore makes a difference

@goldyfruit
Copy link
Author

I turned off the live-restore option, keep you posted. 👍

@goldyfruit
Copy link
Author

So, sadly live-restore is not the problem, I still having the same issue as before.

@argiris-mat
Copy link

We just experience this on an Azure box and its driving us crazy. @goldyfruit where you able to fix this?

@goldyfruit
Copy link
Author

@argiris-mat Don't have this issue anymore since the Docker 18.0X versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants