New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Device-mapper does not release free space from removed images #3182

Closed
AaronFriel opened this Issue Dec 12, 2013 · 202 comments

Comments

Projects
None yet
@AaronFriel

AaronFriel commented Dec 12, 2013

Docker claims, via docker info to have freed space after an image is deleted, but the data file retains its former size and the sparse file allocated for the device-mapper storage backend file will continue to grow without bound as more extents are allocated.

I am using lxc-docker on Ubuntu 13.10:

Linux ergodev-zed 3.11.0-14-generic #21-Ubuntu SMP Tue Nov 12 17:04:55 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

This sequence of commands reveals the problem:

Doing a docker pull stackbrew/ubuntu:13.10 increased space usage reported docker info, before:

Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

And after docker pull stackbrew/ubuntu:13.10:

Containers: 0
Images: 3
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 413.1 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.8 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

And after docker rmi 8f71d74c8cfc, it returns:

Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

Only problem is, the data file has expanded to 414MiB (849016 512-byte sector blocks) per stat. Some of that space is properly reused after an image has been deleted, but the data file never shrinks. And under some mysterious condition (not yet able to reproduce) I have 291.5 MiB allocated that can't even be reused.

My dmsetup ls looks like this when there are 0 images installed:

# dmsetup ls
docker-252:0-131308-pool        (252:2)
ergodev--zed--vg-root   (252:0)
cryptswap       (252:1)

And a du of the data file shows this:

# du /var/lib/docker/devicemapper/devicemapper/data -h
656M    /var/lib/docker/devicemapper/devicemapper/data

How can I have docker reclaim space, and why doesn't docker automatically do this when images are removed?

@vreon

This comment has been minimized.

Contributor

vreon commented Dec 12, 2013

+1, I'm very interested in hearing some discussion on this subject. My strategy so far has been

  • be careful what you build/pull
  • be prepared to blow away your /var/lib/docker 😐

@AaronFriel, which version of Docker are you on? 0.7.1?

@kiorky

This comment has been minimized.

Contributor

kiorky commented Dec 12, 2013

/cc @regilero (link also in to #2276)

@alexlarsson

This comment has been minimized.

Contributor

alexlarsson commented Dec 12, 2013

Starting from a fresh /var/lib/docker:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
292M -rw-------. 1 root root 100G Dec 12 17:29 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root   89 Dec 12 17:29 /var/lib/docker/devicemapper/devicemapper/json
732K -rw-------. 1 root root 2.0G Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

Then after docker pull busybox it grew a bit:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
297M -rw-------. 1 root root 100G Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root  181 Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/json
756K -rw-------. 1 root root 2.0G Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 1
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 296.6 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

docker rmi busybox does not make the file larger, but makes the space free in the devicemapper pool:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
298M -rw-------. 1 root root 100G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root   89 Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/json
772K -rw-------. 1 root root 2.0G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

The loopback file doesn't grow if we download the image again:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
298M -rw-------. 1 root root 100G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root  181 Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/json
772K -rw-------. 1 root root 2.0G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 1
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 296.6 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

So, it seems like we fail to re-sparsify the loopback file when the thinp device discards a block.

@alexlarsson

This comment has been minimized.

Contributor

alexlarsson commented Dec 12, 2013

However, if i create a file inside the container fs image it does reclaim the space in the loopback file.
I.e. I did this in the busybox image:

 cd lib
 cat libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 lib
c.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so
.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 > a_file.bin
/lib # ls -l a_file.bin
-rw-r--r--    1 root     root      47090160 Dec 12 16:41 a_file.bin

This grew the data file from 299M to 344M. But when i removed a_file.bin (and waited a bit) it got back to 299M.

So, this seems to me like it is a devicemapper bug. I.e. it forwards discards from the thinp device to the underlying device, but it doesn't discard when removing thinp devices from the pool.

@alexlarsson

This comment has been minimized.

Contributor

alexlarsson commented Dec 16, 2013

This seem to be a kernel issue, I was looking at working around it bu using BLKDISCARD, but i failed. See this bugs for some details: https://bugzilla.redhat.com/show_bug.cgi?id=1043527

@alexlarsson

This comment has been minimized.

Contributor

alexlarsson commented Dec 17, 2013

I put my workaround in https://github.com/alexlarsson/docker/tree/blkdiscard, but we're still researching if we can do better than this.

@alexlarsson

This comment has been minimized.

Contributor

alexlarsson commented Dec 17, 2013

@blalor

This comment has been minimized.

blalor commented Dec 29, 2013

Having this problem on CentOS (2.6.32-358.23.2.el6.x86_64) with Docker 0.7.0, as well. Old, but the problem's not isolated to Ubuntu.

@skakri

This comment has been minimized.

skakri commented Jan 3, 2014

Same issue on Arch GNU/Linux 3.12.6-1-ARCH, Docker version 0.7.2.

@blalor

This comment has been minimized.

blalor commented Jan 7, 2014

Still exists on 0.7.0 on CentOS.

@nmz787

This comment has been minimized.

nmz787 commented Jan 9, 2014

Still exists in 0.7.2 on ubuntu 12.04.3 LTS.

A lot of the space is in docker/devicemapper/devicemapper/data and metadata, but also in docker/devicemapper/mnt

It's neat that I learned you can see the container file systems in docker/devicemapper/mnt/SOME_KIND_OF_ID/rootfs

but it's not neat that my hard disk is almost completely eaten up and only fixable by rmdir -r docker.

@logicminds

This comment has been minimized.

logicminds commented Jan 20, 2014

I am having a similar issue while writing docker support for rspec-system. My test VM (docker host) has a 8GB drive and after repeatedly creating images without deleting them my drive fills up. But after removing all images and containers the drive is still 100% full. I figured it was an ID-10T error but just gave up and destroyed the VM all together.

@mengzechao

This comment has been minimized.

mengzechao commented Jan 21, 2014

Still exist in 0.7.5 on ubuntu 13.04.

@unclejack

This comment has been minimized.

Contributor

unclejack commented Jan 21, 2014

This issue has been fixed by PR #3256 which was recently merged. This fix will be included in a future release.

I'm closing this issue now because the fix has been merged to master.

@unclejack unclejack closed this Jan 21, 2014

@alexlarsson

This comment has been minimized.

Contributor

alexlarsson commented Jan 21, 2014

Note: Its not fully fixed until you also run a kernel with http://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=6d03f6ac888f2cfc9c840db0b965436d32f1816d in it. Without that the docker fix is only partial.

@logicminds

This comment has been minimized.

logicminds commented Jan 21, 2014

What is the work around to remove space. I am using rhel 6.5 and it might be a while to get the new kernel.

Sent from my iPhone

On Jan 21, 2014, at 6:18 AM, Alexander Larsson notifications@github.com wrote:

Note: Its not fully fixed until you also run a kernel with http://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=6d03f6ac888f2cfc9c840db0b965436d32f1816d in it. Without that the docker fix is only partial.


Reply to this email directly or view it on GitHub.

@alexlarsson

This comment has been minimized.

Contributor

alexlarsson commented Jan 22, 2014

@logicminds There is no super easy way to recover the space atm. Technically it should be possible to manually re-sparsify the loopback files. But that would require all the non-used blocks to be zeroed or something for easily detection of the sparse areas, which is not done on thinp device removal.

@logicminds

This comment has been minimized.

logicminds commented Jan 22, 2014

@alexlarsson Does this also affect OEL 6.5? The OEL 6.5 actually uses the uek 3.8 linux kernel and since I have the option between switching from 2.6 to 3.8 kernel this might be a simple switch for me.

@alexlarsson

This comment has been minimized.

Contributor

alexlarsson commented Jan 22, 2014

@logicminds I don't even know if that commit is in the upstream kernel yet. That link is from the device-mapper tree. Its definitely not in 3.8.

@alexlarsson

This comment has been minimized.

Contributor

alexlarsson commented Jan 22, 2014

I'm looking at creating a tool like fstrim that can be used to get back the space.

alexlarsson added a commit to alexlarsson/docker that referenced this issue Feb 20, 2014

devicemapper: Add trim-pool driver command
This command suspends the pool, extracts all metadata from the metadata pool and
then manually discards all regions not in use on the data device. This will
re-sparsify the underlying loopback file and regain space on the host operating system.

This is required in some cases because the discards we do when deleting images and
containers isn't enought to fully free all space unless you have a very new kernel.
See: moby#3182 (comment)

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)

alexlarsson added a commit to alexlarsson/docker that referenced this issue Mar 17, 2014

devicemapper: Add trim-pool driver command
This command suspends the pool, extracts all metadata from the metadata pool and
then manually discards all regions not in use on the data device. This will
re-sparsify the underlying loopback file and regain space on the host operating system.

This is required in some cases because the discards we do when deleting images and
containers isn't enought to fully free all space unless you have a very new kernel.
See: moby#3182 (comment)

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)

alexlarsson added a commit to alexlarsson/docker that referenced this issue Mar 20, 2014

devicemapper: Add trim-pool driver command
This command suspends the pool, extracts all metadata from the metadata pool and
then manually discards all regions not in use on the data device. This will
re-sparsify the underlying loopback file and regain space on the host operating system.

This is required in some cases because the discards we do when deleting images and
containers isn't enought to fully free all space unless you have a very new kernel.
See: moby#3182 (comment)

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)

alexlarsson added a commit to alexlarsson/docker that referenced this issue Mar 28, 2014

devicemapper: Add trim-pool driver command
This command suspends the pool, extracts all metadata from the metadata pool and
then manually discards all regions not in use on the data device. This will
re-sparsify the underlying loopback file and regain space on the host operating system.

This is required in some cases because the discards we do when deleting images and
containers isn't enought to fully free all space unless you have a very new kernel.
See: moby#3182 (comment)

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)

alexlarsson added a commit to alexlarsson/docker that referenced this issue Apr 1, 2014

devicemapper: Add trim-pool driver command
This command suspends the pool, extracts all metadata from the metadata pool and
then manually discards all regions not in use on the data device. This will
re-sparsify the underlying loopback file and regain space on the host operating system.

This is required in some cases because the discards we do when deleting images and
containers isn't enought to fully free all space unless you have a very new kernel.
See: moby#3182 (comment)

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
@ir-fuel

This comment has been minimized.

ir-fuel commented May 22, 2016

I'm using service docker start

@mercuriete

This comment has been minimized.

mercuriete commented May 26, 2016

@ir-fuel thanks, it works fine. 👍

@asarkar

This comment has been minimized.

asarkar commented Jun 22, 2016

Reinstalling docker to release disk space is the most ridiculous answer that I came across while looking for a solution for this issue. Not only that's a waste of time, it's not even allowed in most environments. It's a good way to get paid if you're an hourly worker.

@ir-fuel

This comment has been minimized.

ir-fuel commented Jun 23, 2016

I completely agree. It's amazing that a product like Docker just keeps on eating away disk space with nothing you can do about it except for uninstall/reinstall.

@ror6ax

This comment has been minimized.

ror6ax commented Jun 23, 2016

Checking in to this issue yet another time to see nothing has changed. +1

@awolfe-silversky

This comment has been minimized.

awolfe-silversky commented Jun 23, 2016

This issue is marked closed. We need a resolution. No workaround, no reconfiguration. What is the real status, and what are the configuration settings that are implicated? Dropping and recreating a production Docker node is not acceptable.

@kpande

This comment has been minimized.

kpande commented Jun 23, 2016

workaround is to avoid using the docker device-mapper driver, unfortunately.

@ir-fuel

This comment has been minimized.

ir-fuel commented Jun 23, 2016

And what is the alternative? How do we go about avoiding this?

@ror6ax

This comment has been minimized.

ror6ax commented Jun 23, 2016

If this is not recommended feature to use - why is it silent and default
one? If you do not care for people using devicemapper - I might be even ok
with this. But do inform the user about it! Do you realize the amount of
headache people have due to this amazing 'workaround' you settled on??
On 23 Jun 2016 4:32 p.m., "kpande" notifications@github.com wrote:

workaround is to avoid using the docker device-mapper driver,
unfortunately.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#3182 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/ACRlBbL5BDn3qMzUW_UiaALB32anUER4ks5qOpkJgaJpZM4BTlBd
.

@kpande

This comment has been minimized.

kpande commented Jun 23, 2016

no one forced you to use Docker. there is a lot of learning that should be done before you configure your production server, it's a valid question, why did these issues make it through your test environment into prod?

@blalor

This comment has been minimized.

blalor commented Jun 23, 2016

There's always rkt

@ripienaar

This comment has been minimized.

ripienaar commented Jun 23, 2016

the general bitchy unhelpful snark is no doubt why no-one from upstream cares to give you proper answers.

@asarkar

This comment has been minimized.

asarkar commented Jun 24, 2016

no one forced you to use Docker.

That's like Oracle telling a Java developer to use PHP due to a JVM bug. That's also not consistent with the elevator pitch here

Three years ago, Docker made an esoteric Linux kernel technology called containerization simple and accessible to everyone.

I'm sure a lot of people are grateful that Docker took off like it did and that couldn't have happened without volunteering from the community. However, it shouldn't be this hard to admit that it has it's problems too without implicitly dropping the "I'm a upstream contributor so shut up and listen" line whenever someone brings up an unlikable point.

@ror6ax

This comment has been minimized.

ror6ax commented Jun 24, 2016

Wait. I did report an issue, provided the details of my machine and setup,
which I'm not obliged to. None of devteam responded to my and others bug
reports in half a year period. Now I stated this fact, you call my behavior
bitchy? Do you even open-source? I'm looking for Go project to work on, and
it will not be Docker, I give you that. Is this your goal?
On 23 Jun 2016 16:45, "gregory grey" ror6ax@gmail.com wrote:

If this is not recommended feature to use - why is it silent and default
one? If you do not care for people using devicemapper - I might be even ok
with this. But do inform the user about it! Do you realize the amount of
headache people have due to this amazing 'workaround' you settled on??
On 23 Jun 2016 4:32 p.m., "kpande" notifications@github.com wrote:

workaround is to avoid using the docker device-mapper driver,
unfortunately.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#3182 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ACRlBbL5BDn3qMzUW_UiaALB32anUER4ks5qOpkJgaJpZM4BTlBd
.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jun 24, 2016

First of all, if you still have this problem, please open a new issue;

Wait. I did report an issue

You replied on a 3 year old, closed issue; following the discussion above, the original issue was resolved. Your issue may be the same, but needs more research to be sure; the errors you're reporing indicate that it may actually be something else.

I really recommend to open a new issue, not commenting on a closed issue

provided the details of my machine and setup, which I'm not obliged to.

You're not obliged to, but without any information to go on, it's unlikely to be resolved. So, when reporting a bug, please include the information that's asked for in the template:
https://raw.githubusercontent.com/docker/docker/master/.github/ISSUE_TEMPLATE.md

None of devteam responded to my and others bug reports in half a year period.

If you mean "one of the maintainers", please keep in mind that there's almost 24000 issues and PRs, and less than 20 maintainers, many of whom doing that besides their daily job. Not every comment will be noticed especially if it's on a closed issue.

If this is not recommended feature to use - why is it silent and default one?

It's the default if aufs, btrfs, and zfs are not supported, you can find the priority that's used when selecting drivers; see daemon/graphdriver/driver_linux.go. It's still above overlay, because unfortunately there's some remaining issues with that driver that some people may be affected by.

Automatically selecting a graphdriver is just to "get things running"; the best driver for your situation depends on your use-case. Docker cannot make that decision automatically, so this is up to the user to configure.

If you do not care for people using devicemapper - I might be even ok with this.

Reading back the discussion above, I see that the upstream devicemapper maintainers have looked into this multple times, trying to assist users reporting these issues, and resolving the issue. The issue was resolved for those that reported it, or in some cases, depended on distros updating devicemapper versions. I don't think that can be considered "not caring".

Also, why is default installation uses 'strongly discouraged' storage option?

Running on loop devices is fine for getting docker running, and currently the only way to set up devicemapper automatically. For production, and to get a better performance overall, use direct-lvm, as explained in the devicemapper section in the storage driver user guide.

Why wasn't I told so at installation?

That's out of scope for the installation, really. If you're going to use some software in production, it should be reasonable to assume that you get yourself familiar with that software, and know what's needed to set it up for your use case. Some maintainers even argued if the warning should be output at all. Linux is not a "holding hands" OS (does your distro show a warning that data loss can occur if you're using RAID-0? If you have ports opened in your Firewall?)

@shadowmint

This comment has been minimized.

shadowmint commented Oct 27, 2016

Deeply reluctant as I am, to once again resurrect this ancient thread, there is still no meaningful advice in it about how to work around this issue on an existing machine encountering this issue.

This is my best effort at a tldr; for the entire thread; I hope it helps others who find this thread.

Issue encountered

Your volume has a significant (and growing) amount of space which is in /var/lib/docker and you're using ext3.

Resolution

You're out of luck. Upgrade your file system or see blowing docker away at the bottom.

Issue encountered

Your volume has a significant (and growing) amount of space which is in /var/lib/docker and you're not using ext3 (eg. system currently using xfs or ext4)

Resolution

You may be able to reclaim space on your device using standard docker commands.

Read http://blog.yohanliyanage.com/2015/05/docker-clean-up-after-yourself/

Run these commands:

docker volume ls
docker ps
docker images

If you have nothing listed in any of these, see blowing docker away at the bottom.

If you see old stale images, unused containers, etc. you can perform manual cleanup with:

# Delete 'exited' containers
docker rm -v $(docker ps -a -q -f status=exited)

# Delete 'dangling' images
docker rmi $(docker images -f "dangling=true" -q)

# Delete 'dangling' volumes
docker volume rm $(docker volume ls -qf dangling=true)

This should reclaim much of the hidden container space in the devicemapper.

Blowing docker away

Didn't work? You're out of luck.

Your best bet at this point is:

service docker stop
rm -rf /var/lib/docker
service docker start

This will destroy all your docker images. Make sure to export ones you want to keep before doing this.

Ultimately, please read https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production; but I hope this will assist others who find this thread.

If you have problems with using advice above open a new ticket that specifically addresses the issue you encounter and link to this issue; do not post it here.

@gg7

This comment has been minimized.

gg7 commented Nov 4, 2016

rm -rf /var/lib/docker

You can also use nuke-graph-directory.sh.

openstack-gerrit pushed a commit to openstack/fuel-qa that referenced this issue Nov 22, 2016

[8.0] Reboot procedure fixed
Added the workaround to avoid the thin pool exhaustion probably caused
by the issue moby/moby#3182

Change-Id: I2edc74189872754383bf9a1f3215490d46e38b27
@gdring2

This comment has been minimized.

gdring2 commented Mar 2, 2017

After removing the files as stated above, I can't start Docker any more:
Mar 02 04:53:40 pmdc001b dockerd[29522]: Error starting daemon: error initializing graphdriver: open /var/lib/docker/devicemapper/devicemapper/data: no such file or directory

@pauvos

This comment has been minimized.

pauvos commented Mar 21, 2017

Just ran into this issue on CentOS 7.3 and didn't want to debug devmapper issues that exist for more than 3 years, so I followed this DC/OS guide, purged the original packages, switched over to overlayfs and everything seems to work fine now: https://dcos.io/docs/1.7/administration/installing/custom/system-requirements/install-docker-centos/ (had to modify the ExecStart command for docker version 17.03 though -> "dockerd --storage-driver=overlay")

Server Version: 17.03.0-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
...
Operating System: CentOS Linux 7 (Core)

(purging volumes, images and containers didn't help. Deleting stuff in /var/lib/docker led to the problem described by @gdring2 )

@invisibleroads

This comment has been minimized.

invisibleroads commented Oct 28, 2017

Running docker system prune freed a lot of space on my machines.

https://docs.docker.com/engine/reference/commandline/system_prune/

@Noah-Huppert

This comment has been minimized.

Noah-Huppert commented Apr 5, 2018

Well, this is kind of... lame.

In my case I found this issue after I uninstalled Docker and deleted the the /var/lib/docker directory, so I couldn't run the equivalent of service docker stop ... service docker start.

I found that my system was not reporting the space from deleting /var/lib/docker as freed (I had ~14 GB sitting in what seemed like limbo).

The fix to this is to simply reload your file system, in my case I just rebooted and the space was reclaimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment