Overlay volumes are out of control #928
Comments
|
I can't think of a reason that RancherOS would behave differently than any other distro here. If you do see different results on Ubuntu though, we'll definitely look into this! |
|
I have tested it on Ubuntu, and the issue is not present. My other thought would be Storage Driver differences, since rancherOS is |
|
Currently having the same issue. Not sure if it is related to |
|
I have also seen huge lists making df unusable, I've not done any testing towards what causes it though |
|
Is there a workaround for this? I awoke this morning to many downed services (that have been working fine for months) and discovered all hosts were out of disk space. There are thousands of these entries in /var/lib/docker/overlay. I'd like to figure out how to fix it quickly vs, re-deploying new hosts and updating DNS. |
|
@JacieChao |
|
We could set overlay2 for both system-docker and user docker as storage driver. According to my verification,this issue can be mitigated by using overlay2. The verify steps are as follows:
As the result, using overlay2 as storage driver can save almost 10% storage, and the storage growth was not obvious when upgrading the container. In my test, When running |
|
Close it. |
|
What's the proper way to fix this? |
When running
sudo dflogged in asranchervia ssh, I get 2000+ entities such as this:They're all the same folder, but once in a while the size changes.
Tracking it down, its a container than I start on a schedule using rancher-cron (something I am working on as a simple cron solution) once a minute. The container itself is set to
io.rancher.container.start_once=trueand maintained byrancher.Output of
docker inspect:[ { "Id": "ab407a3abe6fe864c2251a1b9269c7544b9e6c0c9121fc604e3548db86f6626d", "Created": "2016-05-07T00:27:33.477429259Z", "Path": "docker/startup", "Args": [ "redacted", ], "State": { "Status": "exited", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 0, "Error": "", "StartedAt": "2016-05-09T19:58:20.768540168Z", "FinishedAt": "2016-05-09T19:58:29.30269597Z" }, "Image": "redacted", "ResolvConfPath": "/var/lib/docker/containers/ab407a3abe6fe864c2251a1b9269c7544b9e6c0c9121fc604e3548db86f6626d/resolv.conf", "HostnamePath": "/var/lib/docker/containers/ab407a3abe6fe864c2251a1b9269c7544b9e6c0c9121fc604e3548db86f6626d/hostname", "HostsPath": "/var/lib/docker/containers/ab407a3abe6fe864c2251a1b9269c7544b9e6c0c9121fc604e3548db86f6626d/hosts", "LogPath": "/var/lib/docker/containers/ab407a3abe6fe864c2251a1b9269c7544b9e6c0c9121fc604e3548db86f6626d/ab407a3abe6fe864c2251a1b9269c7544b9e6c0c9121fc604e3548db86f6626d-json.log", "Name": "redacted", "RestartCount": 0, "Driver": "overlay", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": null, "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": { "max-file": "2", "max-size": "25m" } }, "NetworkMode": "default", "PortBindings": null, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Dns": [ "169.254.169.250" ], "DnsOptions": null, "DnsSearch": [ "redacted", "rancher.internal" ], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "ShmSize": 67108864, "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "KernelMemory": 0, "Memory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": -1, "OomKillDisable": false, "PidsLimit": 0, "Ulimits": null }, "GraphDriver": { "Name": "overlay", "Data": { "LowerDir": "/var/lib/docker/overlay/827fa1ac20bdb1e95033f8765ff100e87b741dc6d6b1961462976daf22770bf5/root", "MergedDir": "/var/lib/docker/overlay/b276a4fc6a0ba41534242436f94469cf19395f9ee2cdfaf25c71c64d68eca5af/merged", "UpperDir": "/var/lib/docker/overlay/b276a4fc6a0ba41534242436f94469cf19395f9ee2cdfaf25c71c64d68eca5af/upper", "WorkDir": "/var/lib/docker/overlay/b276a4fc6a0ba41534242436f94469cf19395f9ee2cdfaf25c71c64d68eca5af/work" } }, "Mounts": [ { "Name": "4e508d7e2944e34ed79514880209fd26605fd88e3750ba30ecb2262fbd9d692e", "Source": "/var/lib/docker/volumes/4e508d7e2944e34ed79514880209fd26605fd88e3750ba30ecb2262fbd9d692e/_data", "Destination": "/app/storage", "Driver": "local", "Mode": "", "RW": true, "Propagation": "" } ], "Config": { "Hostname": "ab407a3abe6f", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "ExposedPorts": { "80/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "redacted" ], "Cmd": [ "php", "artisan", "schedule:run" ], "Image": "[redacted]", "Volumes": { "/app/storage": {} }, "WorkingDir": "/app", "Entrypoint": [ "docker/startup" ], "MacAddress": "02:9e:59:c2:f9:7b", "OnBuild": null, "Labels": { "io.rancher.container.ip": "10.42.217.199/16", "io.rancher.container.name": "[redacted]", "io.rancher.container.pull_image": "always", "io.rancher.container.start_once": "true", "io.rancher.container.uuid": "16822e5b-d62b-4256-a18d-0de8457c87bb", "io.rancher.cron.schedule": "0 * * * * * ", "io.rancher.project.name": "[redacted]", "io.rancher.project_service.name": "[redacted]", "io.rancher.scheduler.affinity:host_label": "type=node", "io.rancher.service.deployment.unit": "[redacted]", "io.rancher.service.launch.config": "io.rancher.service.primary.launch.config", "io.rancher.stack.name": "[redacted]", "io.rancher.stack_service.name": "[redacted]" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": null, "SandboxKey": "[redacted, because key could mean private]", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "bridge": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "73031b501fba177f5e59753ceb8dcd0e1c00e8285d7b145b430a672f735fdcd8", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "" } } } } ]output of
docker info:This also spun
t2.smallinstance out of control (not entirely sure as there were other services on that machine), leading to 30+ load, with very minimal actual usage, filled with lines like:As soon as the box had containers removed from it, it stopped having load problems.
I am now re-running everything using an Ubuntu 14.04 instance to see if the issue is with RancherOS or not.
Unfortunately, before I figured that its a good idea to open this issue, I have terminated the box, so I cannot pull any additional from the instance.
RancherOS Version: 0.4.4
Where are you running RancherOS? AWS
The text was updated successfully, but these errors were encountered: