Docker.qcow2 never shrinks - disk space usage leak in docker for mac #371

Open
yakticus opened this Issue Aug 19, 2016 · 78 comments

Projects

None yet
@yakticus
yakticus commented Aug 19, 2016 edited

Expected behavior

Amount of disk space used under ~/Library/Containers/com.docker.docker/ should shrink when deleting images and/or containers

Actual behavior

Amount of disk space used under ~/Library/Containers/com.docker.docker/ doesn't shrink after deleting ~50 containers and ~141 images (out of a total of 194). My Docker.qcow2 was 41GB both before and after doing these deletions.

On the Diagnose & Feedback screen I see an error that looks relevant:

Docker for Mac: version: mac-v1.12.0.1
OS X: version 10.11.6 (build: 15G31)
logs: /tmp/5F7A165B-83AC-4514-AFC4-DB53430D2E9D/20160819-141130.tar.gz
[OK]     docker-cli
[OK]     app
[OK]     moby-syslog
[ERROR]  disk
         disk check failed with: Failure("exec: /Applications/Docker.app/Contents/Resources/bin/../../../Contents/MacOS/qemu-img check /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 > /tmp/5F7A165B-83AC-4514-AFC4-DB53430D2E9D/20160819-141130/qemu-img-check.stdout 2> /tmp/5F7A165B-83AC-4514-AFC4-DB53430D2E9D/20160819-141130/qemu-img-check.stderr: exit 2")
[OK]     virtualization
[OK]     system
[OK]     menubar
[OK]     osxfs
[OK]     db
[OK]     slirp
[OK]     moby-console
[OK]     logs
[OK]     vmnetd
[OK]     env
[OK]     moby
[OK]     driver.amd64-linux

Information

For some history, see docker/docker#23437

  • Diagnostic ID from "Diagnose & Feedback" in the menu.
    I might not have this anymore since I had to do a factory reset in order to reclaim the disk space. After the factory reset, it is: 5F7A165B-83AC-4514-AFC4-DB53430D2E9D
  • a reproducible case if this is a bug, Dockerfiles FTW

Easy to reproduce by pulling some images, observing the size of Docker.qcow2, deleting the images and seeing that the size of Docker.qcow2 does not change. Let me know if more info is needed

  • page URL if this is a docs issue or the name of a man page
  • host distribution and version (OSX 10.10.x, OSX 10.11.x)

OSX 10.11.6

Steps to reproduce the behavior

  1. pull several images
  2. run a couple of containers
  3. observe size of ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
  4. stop and delete all containers
  5. delete all images
  6. repeat step 3. Size has not shrunk despite there being no images or containers

Transcript:

$ docker pull frolvlad/alpine-scala
Using default tag: latest
latest: Pulling from frolvlad/alpine-scala
e110a4a17941: Pull complete
052edde983dc: Pull complete
44b3390a7725: Pull complete
0f6b9d2879c1: Pull complete
Digest: sha256:1d9f6fd716526764a4ba6e389b635cbef8c5d413216bf5d00ca5175fc0d9bac8
Status: Downloaded newer image for frolvlad/alpine-scala:latest
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
1290    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
$ docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
2f0243478e1f: Pull complete
d8909ae88469: Pull complete
820f09abed29: Pull complete
01193a8f3d88: Pull complete
Digest: sha256:8e2324f2288c26e1393b63e680ee7844202391414dbd48497e9a4fd997cd3cbf
Status: Downloaded newer image for ubuntu:latest
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
1456    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
$ docker pull python
Using default tag: latest
latest: Pulling from library/python
357ea8c3d80b: Pull complete
52befadefd24: Pull complete
3c0732d5313c: Pull complete
ceb711c7e301: Pull complete
4211bb537697: Pull complete
71f9074c0739: Pull complete
3e5349707036: Pull complete
Digest: sha256:a755ad5a30b2146f9a132bf74107aa5be9a26b7cc77956b379af678e0f311b3c
Status: Downloaded newer image for python:latest
$ docker run ubuntu
$ docker run -it ubuntu bash
root@a2ae24adfd12:/# exit
$ docker run -itd ubuntu bash
95f5bed47a790d2c8d60544cbc1fa870e1ef4b8c04653508bed34fb25e650536
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
95f5bed47a79        ubuntu              "bash"              9 seconds ago       Up 9 seconds                                    hungry_lalande
a2ae24adfd12        ubuntu              "bash"              28 seconds ago      Exited (0) 21 seconds ago                       sharp_borg
a567a8a7aed3        ubuntu              "/bin/bash"         53 seconds ago      Exited (0) 52 seconds ago                       ecstatic_poitras
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
2371    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
$ docker stop hungry_lalande
hungry_lalande
$ docker rm -v $(docker ps -a -q -f status=exited)
95f5bed47a79
a2ae24adfd12
a567a8a7aed3
$ docker rmi $(docker images -q)
Untagged: ubuntu:latest
Untagged: ubuntu@sha256:8e2324f2288c26e1393b63e680ee7844202391414dbd48497e9a4fd997cd3cbf
Deleted: sha256:f8d79ba03c00bbcd8079cf05b7526ac8f4f422744aad8c3747a29a38ed8c4a41
Deleted: sha256:b4119cffabb62bd6374f5b06adcb820344490252b621cb8f6a8fad7f6f61b3ae
Deleted: sha256:83a17331cb4259e2690395586b641aabadf30256f25a032019b816a53a06ae6e
Deleted: sha256:30ae5404a524e7a713f86ebe832b1e198d1270d62b27c860721055b890a3ae9c
Deleted: sha256:d8d865b23727fa63dc7d4e9f02efc3e2433c8f220972689c6b6ba09e8136c162
Untagged: python:latest
Untagged: python@sha256:a755ad5a30b2146f9a132bf74107aa5be9a26b7cc77956b379af678e0f311b3c
Deleted: sha256:a08871375facdc1619fc6d19608a4555db479219b2853e2e11de05eddf97e4cb
Deleted: sha256:0878d93537f4951aa5fa1ca6fd7ef085aa993e520b5af01148d84704ef49abcf
Deleted: sha256:2d5707b912a332c89665a078a2dabb088cbb4e6982b0d3db36217c89640074da
Deleted: sha256:5f6d1bd4cff3a349d2b5989bf04295987b6d679c411357612523341d4b6f2264
Deleted: sha256:0e54104f61d3ae37dc10fbc275796123a40b54859776fb6e937bb5c39655c3e6
Deleted: sha256:d3c72419ca593d81f207884b70670fd69c4fa3bdecf926ecbbbd9d3bbdd6c7bf
Deleted: sha256:016fc440b1e6b49d889b0f325d2439c82c92f554d3f015db67ffde823bd2ff3e
Deleted: sha256:2f71b45e4e254ddceb187b1467f5471f0e14d7124ac2dd7fdd7ddbc76e13f0e5
Untagged: frolvlad/alpine-scala:latest
Untagged: frolvlad/alpine-scala@sha256:1d9f6fd716526764a4ba6e389b635cbef8c5d413216bf5d00ca5175fc0d9bac8
Deleted: sha256:ac1474efe52e017303f24209fff310c734eed81cdd947604545f160d03f2caad
Deleted: sha256:a08f99075408eb40ce74942b01d08003c8d923b8df0b0e593b73b13540850171
Deleted: sha256:5c86a5cfb5b67143c7ec277ac533176abb9906072eac82de512f8e822b06e769
Deleted: sha256:2615ff2c2f8c27913239080269f5be78d541b2564592856548b435f5daa42514
Deleted: sha256:4fe15f8d0ae69e169824f25f1d4da3015a48feeeeebb265cd2e328e15c6a869f
$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.15-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.954 GiB
Name: moby
ID: KQ7I:O3AX:CY77:DKU4:6LZR:Y6RF:ZM2P:U2YA:TZUV:3YDV:DQPU:OTJV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 18
 Goroutines: 29
 System Time: 2016-08-19T21:24:33.294921776Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
2371    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
@kommen
kommen commented Aug 19, 2016

Thanks for filing it over here @yakticus.

I also still have the problem and in the forum others have reported the same: https://forums.docker.com/t/docker-doesnt-release-disk-space-used-by-library-containers-com-docker-docker/15194

@lxhunter

+1

@noslouch noslouch referenced this issue in code-corps/code-corps-api Aug 20, 2016
Closed

docker-compose up dies on initial run #108

@simond
simond commented Aug 23, 2016

Yep, same issue here.

@yankcrime

I've found the quickest way (as mentioned in other issues / forum threads) to fix this is to reset the client which forces the recreation of Docker.qcow2. If you don't want to do that however, there is another fix, once you've cleaned up any dangling images or exited containers.

NB: You'll need to install qemu via Homebrew as this process requires qemu-img to recompress the qcow2 disk image.

Connect to the VM with screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty and then login as root

Fill up the disk with zeroes:

dd if=/dev/zero of=/var/tempfile

This'll take at least couple of minutes on SSD. I wouldn't bother if you're on spinning rust...

Then you need to rm /var/tempfile, logout of the VM, and quit the Docker client entirely. Now we can recompress the disk:

$ pwd
/Users/nick/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux
$ mv Docker.qcow2 Docker.qcow2.original
$ du -hs Docker.qcow2.original
12G     Docker.qcow2.original
$ qemu-img convert -O qcow2 Docker.qcow2.original Docker.qcow2
$ rm Docker.qcow2.original
$ du -hs Docker.qcow2
772M    Docker.qcow2
@geerlingguy

@yankcrime's workaround works... but it's a lot of steps to make Docker not continuously gobble up all the remaining space on my poor little MacBook's SSD. Between this and osxfs perf issues (#77) requiring similarly involved file sync workarounds, it's really hard to recommend Docker for Mac to anyone who's not an infra hacker at heart... I don't want to have to spend half my day helping other devs figure out why all their disk space is gone!

@kevinsimper

Just deleting it seems to be the easiest solution, but it would be nice if the client could reclaim space itself!

@gumho
gumho commented Sep 6, 2016

I've been running into this issue as well on the latest Stable Docker for Mac. @yankcrime's workaround does work. Would be nice to be able to provide an option to grow the disk size as well.

@creynders

👍 💯

@chrisclark

To add to this, I just deleted my qcow2 file and restart docker and disconnect completely from the internet, and docker ps shows no running containers, and docker images -a lists two containers, both of which are < 300mb...Docker will recreate the qcow2 file and I can just sit there, watching it grow in size, while Docker, and I, do absolutely nothing. There are no running containers, no nothing. What on earth is it even doing? I'm literally sitting here staring at the file in Finder (after re-enabling my network...) watching it grow from 200kb to...

...1.14GB
...1.15GB
...1.16GB

This is nuts. It'll happily just keep growing until it eats all of my available hard disk space.

@s80275
s80275 commented Sep 12, 2016

@chrisclark I have the same problem. There is no image, no container, qcow2 just keeping grow up, but until to 1.16GB. Seem 1.16GB is the VM basic requested working size.

@pointerxr

Same issue for me

@AzatKhalilov

Yep! The same issue for me. If often rebuild my images it's trouble,headache! After 5-6 rebuild image has 1G size but disk space decrease on 20-30G!!!!!

@PommeVerte
PommeVerte commented Sep 14, 2016 edited

Can we have an update on this? It's a really big issue that needs solving.
I can appreciate that it might not be an easy fix but some kind of feedback would be awesome.

@alex-ross

Using Docker to manage multiple development environments and this image often ends up to take 60GB for me. It's a lot on my 256GB SSD. All dev environments is using Docker Compose so I can reset docker once in a while but it's really annoying that I have to do so.

@djs55 djs55 added the osx/10.11.x label Sep 15, 2016
@anonymuse

I've been using Docker for Mac and the Docker Machine images in order to experiment effectively with Docker Swarm and other orchestration features -- the large qcow file and not-insubstantial disks of my other Docker Machine hosts make it difficult to fit onto a MBP sized HDD.

Any update here?

@wbednarski
wbednarski commented Sep 20, 2016 edited

If you can remove all images/containers then:

Stop Docker.

docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
docker volume rm $(docker volume ls |awk '{print $2}')
rm -rf ~/Library/Containers/com.docker.docker/Data/*

Start Docker, you have yours GB back.

@anonymuse

@wbednarski -- that's helpful if keeping images/containers around wasn't a big deal (as it might not be on a server) but it tends to be important in development (on my laptop.) I've read also that using the "Reset to factory defaults" button can clean up your installation too.

Thanks for pointing out your solution.

@wbednarski
wbednarski commented Sep 20, 2016 edited

Didn't try reset. I'll check out it soon :]

@anonymuse you can look at @yankcrime solution #371 (comment) if you want to keep images/containers.

@lijingpeng

@yankcrime I tried as you said, but nothing changed.

du -hs Docker.qcow2
17G Docker.qcow2

@anonymuse

Frank, how sure are you that you followed all of the steps correctly? Do
you have the output of your actions that we could take a peek at?

I forgot to run the dd myself first time around!

On Sep 23, 2016 12:07 PM, "Frank" notifications@github.com wrote:

@yankcrime https://github.com/yankcrime I tried as you said, but
nothing changed.

du -hs Docker.qcow2
17G Docker.qcow2


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#371 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACnE0XEI6hkeOnvHaxI2_WR4CYBR1gCgks5qs_kigaJpZM4Jo3w2
.

@djs55
Contributor
djs55 commented Sep 26, 2016

Thanks for the report.

The .qcow2 is exposed to the VM as a block device with a maximum size of 64GiB. As new files are created in the filesystem by containers, new sectors are written to the block device. These new sectors are appended to the .qcow2 file causing it to grow in size, until it eventually becomes fully allocated. It stops growing when it hits this maximum size.

Unfortunately when files are deleted from the filesystem, the block device doesn't find out, because the connection protocol we currently use virtio-blk doesn't support a "TRIM" or "DISCARD" operation. The block device keeps filling up with junk. A similar problem manifests on real hardware in SSDs -- they start slowing down as they fill up, unless TRIM is enabled in the OS and drivers.

We're hoping to fix this in several stages: (note this is still at the planning / design stage, but I hope it gives you an idea)

  1. we'll switch to a connection protocol which supports TRIM, and implement free-block tracking in a metadata file next to the qcow2. We'll create a compaction tool which can be run offline to shrink the disk (a bit like the qemu-img convert but without the dd if=/dev/zero and it should be fast because it will already know where the empty space is)
  2. we'll automate running of the compaction tool over VM reboots, assuming it's quick enough
  3. we'll switch to an online compactor (which is a bit like a GC in a programming language)

We're also looking at making the maximum size of the .qcow2 configurable. Perhaps 64GiB is too large for some environments and a smaller cap would help?

Let me know what you think! Thanks for your patience.

@miletbaker

@djs55 being able to configure the .qcow2 cap would be excellent! Thanks

@rogeriopradoj

We're also looking at making the maximum size of the .qcow2 configurable. Perhaps 64GiB is too large for some environments and a smaller cap would help?

@djs55, if possible, the same way as we can set the preference for CPUs and Memory IN General Preferences Docker Panel! We could mark the maximum we'd like to use!

@PommeVerte

@djs55 Sounds awesome. Like others have mentioned, being able to configure the cap would be a really good stop gag while a more thorough solution comes through.

@jemueller

Looking forward to these fixes. Having my HD randomly fill up, running DaisyDisk and seeing Docker eating up 50GB on my SSD with a relatively small project is not a good sight to behold.

Being able to lower the cap would be a huge help as a stop-gap measure until the automated tools to manage this in the background come out.

What is the expected ETA for the cap configuration tool and/or the various stages of the whole fix?

Thanks for working on this issue.

@jeff-lau

A little bit off topic, what happens when qcow2 hits 64GB? We use docker to stand up many dependent services on a fairly large software project. Some of our docker images are like 15GB in size, so 64Gb is not a lot for us. We are using docker for Mac and running into issues where docker images just seems to mysteriously disappear. Or docker wouldn't start unless we delete the old qcow2 file etc.. images disappearing is a big deal for us as it takes a fair amount of time to pull them again..

@torstenlehmann

Are named volumes in any way related to Docker.qcow2? Does removing Docker.qcow2 break any named volumes or connections between images/containers and named volumes? Just asking myself if I loose the data saved in my named volumes if I remove Docker.qcow2.

I'm using docker-compose for my images/containers and declared named volumes in docker-compose.yml.

Thanks in advance for some hints.

@justincormack
Member

Yes, the data in named volumes is stored in the Docker.qcow2 at present (it is possible to store it on the Mac but this is not the default, and has slower performance). You should back them up by copying via a container before changing the qcow if you have important data there.

@torstenlehmann

Thanks @justincormack for the fast response.

I'm using named volumes because the performance of host volumes on macOS was too bad. So now I have everything in named volumes running fast and watching Docker.qcow2 to grow.

Docker Beta hurts on two fundamentally important aspects here like @geerlingguy said beeing at a stage where it does not make its job silently. At the moment it's like a little docker pet.

But ok that's why it's called beta!

Apart from that it's a great tool for my use case. Thanks a lot!

@JamesChevalier

Yeah @jeff-lau - there's an upper limit to this file as well, as you're noticing.

I've got a StackOverflow question open about this problem here:
http://stackoverflow.com/questions/39402363/how-do-you-get-around-the-size-limitation-of-docker-qcow2-in-the-docker-for-mac

I've hit this limit with a very large postgres database I was trying to store in a volume. I was importing the data from a .sql file into the postgres docker container & the import would fail as the Docker.qcow2 file grew to its limit. I couldn't use mounted host directory volumes, because it's too slow. At the moment I'm working around the issue by not using docker for this database.

@Luukyb
Luukyb commented Oct 25, 2016

The workaround suggest by @yankcrime did not work in my case. Because you need to fill up the disk with zeroes, you need at least to be able to store the file at 64 GB, right?

@JamesChevalier
JamesChevalier commented Oct 26, 2016 edited

@Luukyb I had a good experience with these steps (that site's UI is funky - you may need to scroll down to see the comment from beccari on Aug 31). The process doesn't require running dd, but it does require you to think about your disk usage - you're effectively manually creating a new qcow2 file with a different maximum size, so if you have a drive that's e.g. 120gb and you set the qcow2 file's maximum size to 240gb then you might have some problems in the future with Docker taking up all of your available space. :-)

This is the process he suggests:

  • Create a new "template" image with desired size: qemu-img create -f qcow2 ~/data.qcow2 120G where the 120G is my new size.
  • Replace the default docker template /Application/Docker.app/Contents/Resources/moby/data.qcow2 with your brand new ~/data.qcow2 (I've saved the original just in case)
    • This data.qcow2 is the initial image which is copied to mentioned path ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
  • rm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
    • Unfortunately you have to wipe all your docker data as stated before
  • restart docker
  • Check disk free: docker run --rm alpine df -h
@Luukyb
Luukyb commented Oct 27, 2016

Thanks @JamesChevalier. In my case I have a SSD and 20GB to 30GB would be the maximum I would allow. I will try creating one of 20GB max size, and my understanding is that when it gets full I would be able to recompress the disk.

@jeremywadsack

@djs55 Are you sure about that maximum size?

$ du -sh ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 
 93G    ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
@djs55
Contributor
djs55 commented Nov 3, 2016

@jeremywadsack 64GiB ought to be the maximum size of .qcow2 files created by Docker for Mac. Could you try this:

$ /Applications/Docker.app/Contents/MacOS/qemu-img info ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
image: ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
file format: qcow2
virtual size: 64G (68719476736 bytes)
disk size: 60G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: true
    refcount bits: 16
    corrupt: false

The physical size of the file on the Mac should be less than or equal to the reported virtual size.

If the qcow2 was originally imported from toolbox then the original virtual size of the toolbox disk would have been preserved-- could this be the cause?

@jeremywadsack

I did upgrade from toolbox and it is definitely much more than 64GB:

$ /Applications/Docker.app/Contents/MacOS/qemu-img info ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
image: ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
file format: qcow2
virtual size: 200G (214748364800 bytes)
disk size: 93G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

The upgrade process left pieces of toolbox all over. Including the virtual box image, but that was only 36GB. (Maybe the virtual size was larger.)

How would I go about reducing the virtual size setting?

@askoretskiy

I have the same problem -- deletion of all images and containers did not help. I had to restore factory settings in Docker to free the space.

@rallen-temp

@wbednarski's comment worked for me, #371 (comment).

Delete + reset. Did lose all data, but gained some precious 50GB back.

➜  / du -h -d 1  ~/Library/Containers/com.docker.docker
 55G    /Users/rallen36/Library/Containers/com.docker.docker/Data
 55G    /Users/rallen36/Library/Containers/com.docker.docker

➜  / rm -rf ~/Library/Containers/com.docker.docker/Data/*
zsh: sure you want to delete all the files in /Users/rallen36/Library/Containers/com.docker.docker/Data [yn]? y

➜  / du -h -d 1  ~/Library/Containers/com.docker.docker
 32K    /Users/rallen36/Library/Containers/com.docker.docker/Data
 32K    /Users/rallen36/Library/Containers/com.docker.docker

➜  / docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
@blueredcat

This is still an issue. Very large qcow file.

@benduncan

+1 to have this fixed, the Docker utils should sit in the background and automatically manage the disk-space and free unused disk space in the Docker.qcow2

Anyone keen to help make the fixes and commit back?

@avatsaev

this is still an issue, had to uninstall docker from mac

@woegjiub

Could someone please lock comments for this?

These "+1" posts are unnecessary noise.

@simPod
simPod commented Nov 23, 2016

how do you want to lock comments on github? :D

@avatsaev

By solving the issue :D

@CHAAAAA CHAAAAA referenced this issue in kiwenlau/hadoop-cluster-docker Nov 23, 2016
Closed

How to delete the HDFS data in Docker containers #33

@izakp
izakp commented Nov 24, 2016 edited

This is the only major issue blocking our adoption of Docker for Mac - simply being able to configure the .qcow2 cap would be sufficient

@izakp
izakp commented Nov 24, 2016

@simPod yes we are experiencing that issue where mounted volumes are required and still using Dinghy but it's not so much of a blocker to Devs as their laptop disk filling up

@djs55
Contributor
djs55 commented Nov 24, 2016 edited

Quick update on this: configuring the qcow2 size cap is possible in the latest betas (e.g. 30.1) with the caveats:

  • that it won't grow automatically yet: the Docker.qcow2 needs to be deleted (loosing containers and images) and then the app restarted
  • there is no UI element yet

For example:

# my disk is currently 64GiB
$ /Applications/Docker.app/Contents/MacOS/qemu-img info ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
image: /Users/djs/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
file format: qcow2
virtual size: 64G (68719476736 bytes)
disk size: 28G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: true
    refcount bits: 16
    corrupt: false

# I can increase or decrease the default cap by:
$ cd ~/Library/Containers/com.docker.docker/Data/database/
$ git reset --hard
HEAD is now at 8635944 last-start-time changed at 1480009710
$ cat com.docker.driver.amd64-linux/disk/size
65536
$ echo 131072 > com.docker.driver.amd64-linux/disk/size
$ git add com.docker.driver.amd64-linux/disk/size
$ git commit -s -m 'increase size to 128GiB'
[master 60861de] increase size to 128GiB
 1 file changed, 1 insertion(+), 1 deletion(-)

# delete the disk -- this unfortunately deletes all images and containers
$ rm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
# now I need to restart the app

For anyone wanting to use an external drive, you can also change the location of the Docker.qcow2 by editing the database key com.docker.driver.amd64-linux/disk/path. There is no UI element for this yet either.

Regarding automatic shrinkage via TRIM: I'm working on this at the moment. There are early patches which expose the TRIM feature to the VM; and for the VM to invoke TRIM; the next step is to modify the qcow2 code to record the TRIMmed blocks somewhere where they can be read by a compactor. I'll let you know when there's some code available to test.

(edited to fix typo reported by @dojohnso below)

@dojohnso

@djs55 minor detail, but you left out a "/" in

echo 131072 > com.docker.driver.amd64-linux/disk size

otherwise, thank you!!

@djs55
Contributor
djs55 commented Dec 7, 2016

Quick update on this: beta 32 (released today) has experimental support for TRIM. It's not yet running continuously in the background but it can be manually triggered for testing using a command like:

docker run --rm --privileged --pid=host justincormack/debian nsenter -t 1 -m -n fstrim /var

The TRIMs are batched on the host and, when a threshold is hit, a disk compaction shrinks the file. There are known rough edges including

  • the compaction can be slow, particularly if there are many gigabytes to free
  • there may be a possible deadlock in the compaction code -- this seems rare and I'm trying to track it down at the moment

Since this is experimental, please be careful trying it if your VM contains data which is hard to recreate. If you do give it a go, let me know if it works (or not).

@yankcrime

Thanks @djs55 - certainly did the trick here, reducing usage from 16GB down to 4.5GB in a little under 6 minutes. This is on an SSD-equipped 2014 MBP:

~
$ du -hs ~/Library/Containers/com.docker.docker
 16G	/Users/nick/Library/Containers/com.docker.docker

~
$ docker run --rm --privileged --pid=host justincormack/debian nsenter -t 1 -m -n fstrim /var
Unable to find image 'justincormack/debian:latest' locally
latest: Pulling from justincormack/debian
43c265008fae: Pull complete
5b24ea945c69: Pull complete
Digest: sha256:c6cf0ea412a729633903ce59113274144e7044b2d1698be802c038987cb97b6f
Status: Downloaded newer image for justincormack/debian:latest

~ 5m 52s
$ du -hs ~/Library/Containers/com.docker.docker
4.5G	/Users/nick/Library/Containers/com.docker.docker
@ye
ye commented Dec 7, 2016

Thank you @djs55 !

It worked for me as well. It reduced the ~/Library/Containers/com.docker.docker down from around 18GB to 8.2GB in under 2 minutes (sorry didn't quite capture the log outputs but those are rough estimates) on an Early 2015 MBP.

@ggarnier
ggarnier commented Dec 7, 2016

@djs55 it's not working for me (OS X El Capitan):

$ docker run --rm --privileged --pid=host justincormack/debian nsenter -t 1 -m -n fstrim /var
fstrim: ioctl 0xc0185879 failed: Not supported
@MisterMartin
@ggarnier
ggarnier commented Dec 7, 2016

@MisterMartin I just checked for updates, and seems like I'm already using the latest version

@konradjurk

@ggarnier Are you in the beta channel?

@justincormack
Member

@ggarnier this is only on beta channel at present (as its still very beta!).

@jrwesolo
jrwesolo commented Dec 7, 2016

@djs55 This is awesome! You mentioned it does not run continuously in the background right now. Is the plan to have that be automatic before it exits the beta channel?

@ggarnier
ggarnier commented Dec 7, 2016

@konradjurk @justincormack oh, I see. I'm using the stable version. Sorry!

@dbourguignon

Worked for me perfectly too.

Down from 29Go to 6.4Go

@bric3
bric3 commented Dec 8, 2016

It works!
@djs55 is it necessary to use justincormack/debian for nsenter? For exemple can't we use walkerlee/nsenter it only ships with nsenter and fstrim is run on Moby (alpine) anyway .

@wojas
wojas commented Dec 11, 2016 edited

If I try to run fstrim /var with beta32 on El Capitan, the Docker CPU usage goes up to 100% and Docker for Mac completely stops responding after a minute or so (both the docker daemon and on the com.docker.driver.amd64-linux/tty socket).

I let it run overnight, but it seems it simply crashed. I cannot find any errors in the logs, not any tty output.

UPDATE: beta33.1 works like a charm!

@thaJeztah thaJeztah referenced this issue in docker/docker Dec 13, 2016
Merged

New Data Management commands #26108

2 of 2 tasks complete
@ak47
ak47 commented Dec 13, 2016

has no discernible affect on my machine:
osx: 10.11.6
docker: 1.13.0-rc3-beta32 (14523)

→ du -hs ~/Library/Containers/com.docker.docker
 95G    /Users/andrew/Library/Containers/com.docker.docker
→ docker run --rm --privileged --pid=host justincormack/debian nsenter -t 1 -m -n fstrim /var
→ du -hs ~/Library/Containers/com.docker.docker
 95G    /Users/andrew/Library/Containers/com.docker.docker
@keverw
keverw commented Dec 14, 2016

Yeah it didn't seem to have any affect for me either.

Kevins-MacBook-Pro:~ kevinwhitman$ du -hs ~/Library/Containers/com.docker.docker
 49G	/Users/kevinwhitman/Library/Containers/com.docker.docker
Kevins-MacBook-Pro:~ kevinwhitman$ docker run --rm --privileged --pid=host justincormack/debian nsenter -t 1 -m -n fstrim /var
Kevins-MacBook-Pro:~ kevinwhitman$ du -hs ~/Library/Containers/com.docker.docker
 49G	/Users/kevinwhitman/Library/Containers/com.docker.docker
Kevins-MacBook-Pro:~ kevinwhitman$ 
@jeremywadsack

No change for me either.

$ docker version
Client:
 Version:      1.13.0-rc3
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   4d92237
 Built:        Tue Dec  6 01:15:44 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.13.0-rc3
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   4d92237
 Built:        Tue Dec  6 01:15:44 2016
 OS/Arch:      linux/amd64
 Experimental: true

$ du -hs ~/Library/Containers/com.docker.docker
 95G	/Users/jeremywadsack/Library/Containers/com.docker.docker
$ docker run --rm --privileged --pid=host justincormack/debian nsenter -t 1 -m -n fstrim /var
$ du -hs ~/Library/Containers/com.docker.docker
 95G	/Users/jeremywadsack/Library/Containers/com.docker.docker

Could it be that we have already filled our volumes? How would I check that? This reports 112GB which is more than the reported size of the volume.

$ docker images | awk '{if ($8=="MB") { print $7 } else { print $7*1024 } }' | paste -s -d + - | bc
115689.38
@Waldz
Waldz commented Dec 17, 2016 edited

👍 💯
19G

cd ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/
ls -lah Docker.qcow2
@pditommaso

41G here! 👍

@ElForastero
ElForastero commented Dec 18, 2016 edited

➜ www cd ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux
➜ com.docker.driver.amd64-linux ls -lah Docker.qcow2
-rw-r--r-- 1 forastero staff 11G Dec 18 11:45 Docker.qcow2

@simPod
simPod commented Dec 18, 2016

@djs55 when do you expect this to be in stable version? thanks!

@djs55
Contributor
djs55 commented Dec 20, 2016

Here's a quick update:

  • beta 33 had automatic compaction turned on but the was a performance bug, the compaction took too long and blocked the I/O, the VM attempted to reset the AHCI controller and this triggered a bug in the hypervisor and the whole VM deadlocked :(
  • beta 33.1 has fixed the performance bug, but for safety the default has been changed to compact offline, whenever the VM shuts down. There is still an fstrim running every 15 minutes from cron which is fast -- all it does is mark the blocks as free. When the VM shuts down we run a compaction via the qcow-tool CLI and the file should shrink then.

@simPod: hopefully the offline compaction (i.e. beta 33.1) will be in stable 1.13 early in the new year.

@bric3: any image with fstrim is fine. Note it runs every 15 minutes from cron now, so maybe that's sufficient.

For people who aren't seeing the file reduce in size: I suspect (but can't be 100% sure) that there are stopped containers or images still consuming space in the VM. Have a look with docker images and docker ps -a to see if there are more of those than you expect. Perhaps run the new command:

$ docker image prune
WARNING! This will remove all dangling images.
Are you sure you want to continue? [y/N] y
Deleted Images:
deleted: sha256:2ff0a8a6539d0a20a4f066865c429a2d6312a6132b99b5dfd887961edd4de8e0
deleted: sha256:48fc0d05769249150018b8218c062dbdd7dd762c0539d5683182e0df04dda981
deleted: sha256:74cd2ea090535e23706ff23257c9b2e925929c412f733af22422737198e5db96
deleted: sha256:4c1acf4f3d736d7a0177b10cd11a855fae144cfff8ef6504fd27ecf759c7682a
deleted: sha256:f9e0d3b5b14fddfd9920874cad1eb53a69c66094275c94fe544c9394224a2a8c
deleted: sha256:65f8481d87b9ada751c776ae44f5efa4570158390d82bb121e7a2f3ccd13b018
deleted: sha256:c9b1accfd7460e9ed9d4bd0270a5caad06f97c746de603ebd4cd0e0b11ed7e04
deleted: sha256:dd7da2a4a7e2e8d4cbab59df584d3ee8db1f4c164c4700fd1529a67238f03784
deleted: sha256:26936ac3e1559c4065fefa4c986f2bec9571836036b08af6998299afeaf7fc8e
deleted: sha256:cf88610937d834f11a3ee6ab711c0a6ceeaec68045524aa4dab794f709d63cc4
deleted: sha256:0b5467e16ea5582f824999a3cb95a49a42a2270338a29571c4e823d3d694501b
deleted: sha256:1654d6e9bcec111d6268b007cef5ea05663583fb61c1796548997ce1e33dcf01
deleted: sha256:76247d7ada099f78fb6ef4ccdb4e4377ab8d0a5828682ff85b339d5dd55c733e
deleted: sha256:c96dbdb64ed4102d1f0aa6ae15c6a24c9ffd736902e42918e79c3fbc74673797
deleted: sha256:5db7cb29d89908817a743d50f38689c0999fd6a6642ab1a59782be2edefdecbd
deleted: sha256:865ad332e4acc91c9b503ca258daf996e8379ff34410c57aed73635719f56027
deleted: sha256:b77c07ec7a1bfaec3f7163c169919d6a836aa687ddc5ed1b60db98bb1cd26094
deleted: sha256:1f485d1c6a13339d503c134a1d98f6cdc2d9dff6b396f03aa401a8f2025fc067
deleted: sha256:516ff334aad20aa46995985dd1ab184708bca8c21dff9b4b625eeef76cb7147d
deleted: sha256:e49a643ecb5f61d6dc2bb21c463d76c77d67f2376a768a4d31bc03acf08d645b
deleted: sha256:a3a4474b03b3be17a67ba7745d328edef69fd83fdefb0c7bfbd0a84a3ccfbdbc
deleted: sha256:0d958c2b61920d1c484f681500162fea607387d1b15d66289a8d3ab4c218afaa
deleted: sha256:308592998a740f743b20e8b6e5c4f3a0e97591a2467c7d72b88d74343e5daab0
deleted: sha256:9a045d3548d46a70f008729304e724e9f242ad156afa31158f0dabd3838a8222
deleted: sha256:7f77f7815ac8db7f831815d446f18db32de98d61007c5cdfb5dce980232bd38f
deleted: sha256:9dde0c1520058256758c039e165fb8e67418d441045c4a2d2eed709383c1d2f8
deleted: sha256:d800c0d7346403c84a4e549e4e46a10a87064b4f72c588039ddce582995db15c
deleted: sha256:f828348a406fd2dcdaa06d18e86cf31eea95d6b55d40b2e5f3e845b9329895b4
deleted: sha256:b07acfcb0f9a999ed8b2f0c9e05f7d90cc04710f8bae4fa911aeddf43e832699
deleted: sha256:fc18587acbc9872ca60976405c762e6503325d596d6965d265e2e1019e9543f8
deleted: sha256:68f2fec945f578c5652eee3da928178f18aa0b47b838f38c72a11951163d8d63

Total reclaimed space: 96.91 MB

(Note this space is initially reclaimed in the VM, the fstrim command and then the VM reboot is needed before the space is freed on the host).

When 1.13 stable is out I'll look again at compacting the .qcow2 online. I'm not completely convinced that's worth the risk from a code complexity point of view -- let me know what you think about the offline compaction.

@luckydonald
luckydonald commented Dec 21, 2016 edited

All this from 29G to 15G in a couple of minutes with this strange trick sounds like them horrible advertisements on porn sites.

EDIT: I got 18GB back

@dojohnso

With the new beta, when i shut down my containers and restart docker, it took maybe 5 minutes, but it brought me down from 127G to 82G (I used the trick from @djs55 to up my container potential). Definitely a great step in the right direction.

@yvess
yvess commented Dec 21, 2016

I had the same issue with docker running in vmware fusion. I used zfs on osx to make it easier.
When you use zfs with compression (very fast) you gain on two sides. With compression it uses around half the space it would. When you do a maintenance with zero filling the zeros get compressed to nothing. So I made a symlink from Docker.qcow2 to a zfs filesystem. I would even prefer to use raw images instead of qcow2 images this would improve the performance and with zfs qcow2 is not needed.

@vyscond
vyscond commented Dec 23, 2016

@yvess Can you post a more detailed tutorial about your explanation? Sounds very interesting :3

@kbariotis
kbariotis commented Jan 2, 2017 edited

Same thing here with 22GB of Docker.qcow2. I just deleted all Docker containers, images and volumes through cli and it didn't shrink even a bit.

Mac OS Siera, Docker Version 1.12.5 (14777)

Removing it manually.

@Opa-
Opa- commented Jan 5, 2017

Same here, I just runned docker rm -f $(docker ps -qa) and docker rmi -f $(docker images -q) to delete everything and a restart or manual quit + start of Docker didn't purge the size of Docker.qcow2 (about 60 GB...) I had to remove it manually.
I'm running on Mac OS X El Capitan 10.11.6 (15G1212) and latest version of Docker:

Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 06:14:34 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 06:14:34 2016
 OS/Arch:      linux/amd64
@KeruHua
KeruHua commented Jan 5, 2017

@djs55 It does not work for me. Did I do anything wrong?
$ du -hs ~/Library/Containers/com.docker.docker 5.3G /Users/owner/Library/Containers/com.docker.docker $ docker run --rm --privileged --pid=host justincormack/debian nsenter -t 1 -m -n fstrim /var Unable to find image 'justincormack/debian:latest' locally latest: Pulling from justincormack/debian 43c265008fae: Pull complete 5b24ea945c69: Pull complete Digest: sha256:c6cf0ea412a729633903ce59113274144e7044b2d1698be802c038987cb97b6f Status: Downloaded newer image for justincormack/debian:latest $ du -hs ~/Library/Containers/com.docker.docker 6.2G /Users/owner/Library/Containers/com.docker.docker

By the way, my docker information is:

`Client:
Version: 1.13.0-rc4
API version: 1.25
Go version: go1.7.3
Git commit: 88862e7
Built: Sat Dec 17 01:34:17 2016
OS/Arch: darwin/amd64

Server:
Version: 1.13.0-rc4
API version: 1.25 (minimum version 1.12)
Go version: go1.7.3
Git commit: 88862e7
Built: Sat Dec 17 01:34:17 2016
OS/Arch: linux/amd64
Experimental: true`

And my computer is Macbook Pro Sierra 10.12.

@drs-m
drs-m commented Jan 10, 2017

This is a major issue. The file is over 50gb by now.

@djs55 djs55 added a commit to djs55/docker.github.io that referenced this issue Jan 12, 2017
@djs55 djs55 docker-for-mac: add a FAQ about reducing the qcow2 size
In Docker for Mac 1.12 the only way to free space on the host is to
delete the qcow2 which means all containers and images have to be
rebuilt.

In Docker for Mac 1.13 there is preliminary support for shrinking the
qcow2 file non-destructively using "TRIM" (as also used on SSDs).
Unfortunately this isn't (yet) fully automatic -- it runs in the
background and requires the app to be occasionally restarted.

Related to [docker/for-mac#371]

Signed-off-by: David Scott <dave.scott@docker.com>
b9df8fd
@djs55 djs55 referenced this issue in docker/docker.github.io Jan 12, 2017
Merged

docker-for-mac: add a FAQ about reducing the qcow2 size #1100

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment