New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker.qcow2 never shrinks - disk space usage leak in docker for mac #371

Open
yakticus opened this Issue Aug 19, 2016 · 251 comments

Comments

Projects
None yet
@yakticus

yakticus commented Aug 19, 2016

Expected behavior

Amount of disk space used under ~/Library/Containers/com.docker.docker/ should shrink when deleting images and/or containers

Actual behavior

Amount of disk space used under ~/Library/Containers/com.docker.docker/ doesn't shrink after deleting ~50 containers and ~141 images (out of a total of 194). My Docker.qcow2 was 41GB both before and after doing these deletions.

On the Diagnose & Feedback screen I see an error that looks relevant:

Docker for Mac: version: mac-v1.12.0.1
OS X: version 10.11.6 (build: 15G31)
logs: /tmp/5F7A165B-83AC-4514-AFC4-DB53430D2E9D/20160819-141130.tar.gz
[OK]     docker-cli
[OK]     app
[OK]     moby-syslog
[ERROR]  disk
         disk check failed with: Failure("exec: /Applications/Docker.app/Contents/Resources/bin/../../../Contents/MacOS/qemu-img check /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 > /tmp/5F7A165B-83AC-4514-AFC4-DB53430D2E9D/20160819-141130/qemu-img-check.stdout 2> /tmp/5F7A165B-83AC-4514-AFC4-DB53430D2E9D/20160819-141130/qemu-img-check.stderr: exit 2")
[OK]     virtualization
[OK]     system
[OK]     menubar
[OK]     osxfs
[OK]     db
[OK]     slirp
[OK]     moby-console
[OK]     logs
[OK]     vmnetd
[OK]     env
[OK]     moby
[OK]     driver.amd64-linux

Information

For some history, see moby/moby#23437

  • Diagnostic ID from "Diagnose & Feedback" in the menu.
    I might not have this anymore since I had to do a factory reset in order to reclaim the disk space. After the factory reset, it is: 5F7A165B-83AC-4514-AFC4-DB53430D2E9D
  • a reproducible case if this is a bug, Dockerfiles FTW

Easy to reproduce by pulling some images, observing the size of Docker.qcow2, deleting the images and seeing that the size of Docker.qcow2 does not change. Let me know if more info is needed

  • page URL if this is a docs issue or the name of a man page
  • host distribution and version (OSX 10.10.x, OSX 10.11.x)

OSX 10.11.6

Steps to reproduce the behavior

  1. pull several images
  2. run a couple of containers
  3. observe size of ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
  4. stop and delete all containers
  5. delete all images
  6. repeat step 3. Size has not shrunk despite there being no images or containers

Transcript:

$ docker pull frolvlad/alpine-scala
Using default tag: latest
latest: Pulling from frolvlad/alpine-scala
e110a4a17941: Pull complete
052edde983dc: Pull complete
44b3390a7725: Pull complete
0f6b9d2879c1: Pull complete
Digest: sha256:1d9f6fd716526764a4ba6e389b635cbef8c5d413216bf5d00ca5175fc0d9bac8
Status: Downloaded newer image for frolvlad/alpine-scala:latest
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
1290    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
$ docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
2f0243478e1f: Pull complete
d8909ae88469: Pull complete
820f09abed29: Pull complete
01193a8f3d88: Pull complete
Digest: sha256:8e2324f2288c26e1393b63e680ee7844202391414dbd48497e9a4fd997cd3cbf
Status: Downloaded newer image for ubuntu:latest
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
1456    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
$ docker pull python
Using default tag: latest
latest: Pulling from library/python
357ea8c3d80b: Pull complete
52befadefd24: Pull complete
3c0732d5313c: Pull complete
ceb711c7e301: Pull complete
4211bb537697: Pull complete
71f9074c0739: Pull complete
3e5349707036: Pull complete
Digest: sha256:a755ad5a30b2146f9a132bf74107aa5be9a26b7cc77956b379af678e0f311b3c
Status: Downloaded newer image for python:latest
$ docker run ubuntu
$ docker run -it ubuntu bash
root@a2ae24adfd12:/# exit
$ docker run -itd ubuntu bash
95f5bed47a790d2c8d60544cbc1fa870e1ef4b8c04653508bed34fb25e650536
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
95f5bed47a79        ubuntu              "bash"              9 seconds ago       Up 9 seconds                                    hungry_lalande
a2ae24adfd12        ubuntu              "bash"              28 seconds ago      Exited (0) 21 seconds ago                       sharp_borg
a567a8a7aed3        ubuntu              "/bin/bash"         53 seconds ago      Exited (0) 52 seconds ago                       ecstatic_poitras
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
2371    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
$ docker stop hungry_lalande
hungry_lalande
$ docker rm -v $(docker ps -a -q -f status=exited)
95f5bed47a79
a2ae24adfd12
a567a8a7aed3
$ docker rmi $(docker images -q)
Untagged: ubuntu:latest
Untagged: ubuntu@sha256:8e2324f2288c26e1393b63e680ee7844202391414dbd48497e9a4fd997cd3cbf
Deleted: sha256:f8d79ba03c00bbcd8079cf05b7526ac8f4f422744aad8c3747a29a38ed8c4a41
Deleted: sha256:b4119cffabb62bd6374f5b06adcb820344490252b621cb8f6a8fad7f6f61b3ae
Deleted: sha256:83a17331cb4259e2690395586b641aabadf30256f25a032019b816a53a06ae6e
Deleted: sha256:30ae5404a524e7a713f86ebe832b1e198d1270d62b27c860721055b890a3ae9c
Deleted: sha256:d8d865b23727fa63dc7d4e9f02efc3e2433c8f220972689c6b6ba09e8136c162
Untagged: python:latest
Untagged: python@sha256:a755ad5a30b2146f9a132bf74107aa5be9a26b7cc77956b379af678e0f311b3c
Deleted: sha256:a08871375facdc1619fc6d19608a4555db479219b2853e2e11de05eddf97e4cb
Deleted: sha256:0878d93537f4951aa5fa1ca6fd7ef085aa993e520b5af01148d84704ef49abcf
Deleted: sha256:2d5707b912a332c89665a078a2dabb088cbb4e6982b0d3db36217c89640074da
Deleted: sha256:5f6d1bd4cff3a349d2b5989bf04295987b6d679c411357612523341d4b6f2264
Deleted: sha256:0e54104f61d3ae37dc10fbc275796123a40b54859776fb6e937bb5c39655c3e6
Deleted: sha256:d3c72419ca593d81f207884b70670fd69c4fa3bdecf926ecbbbd9d3bbdd6c7bf
Deleted: sha256:016fc440b1e6b49d889b0f325d2439c82c92f554d3f015db67ffde823bd2ff3e
Deleted: sha256:2f71b45e4e254ddceb187b1467f5471f0e14d7124ac2dd7fdd7ddbc76e13f0e5
Untagged: frolvlad/alpine-scala:latest
Untagged: frolvlad/alpine-scala@sha256:1d9f6fd716526764a4ba6e389b635cbef8c5d413216bf5d00ca5175fc0d9bac8
Deleted: sha256:ac1474efe52e017303f24209fff310c734eed81cdd947604545f160d03f2caad
Deleted: sha256:a08f99075408eb40ce74942b01d08003c8d923b8df0b0e593b73b13540850171
Deleted: sha256:5c86a5cfb5b67143c7ec277ac533176abb9906072eac82de512f8e822b06e769
Deleted: sha256:2615ff2c2f8c27913239080269f5be78d541b2564592856548b435f5daa42514
Deleted: sha256:4fe15f8d0ae69e169824f25f1d4da3015a48feeeeebb265cd2e328e15c6a869f
$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.15-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.954 GiB
Name: moby
ID: KQ7I:O3AX:CY77:DKU4:6LZR:Y6RF:ZM2P:U2YA:TZUV:3YDV:DQPU:OTJV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 18
 Goroutines: 29
 System Time: 2016-08-19T21:24:33.294921776Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
2371    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
@kommen

This comment has been minimized.

Show comment
Hide comment
@kommen

kommen Aug 19, 2016

Thanks for filing it over here @yakticus.

I also still have the problem and in the forum others have reported the same: https://forums.docker.com/t/docker-doesnt-release-disk-space-used-by-library-containers-com-docker-docker/15194

kommen commented Aug 19, 2016

Thanks for filing it over here @yakticus.

I also still have the problem and in the forum others have reported the same: https://forums.docker.com/t/docker-doesnt-release-disk-space-used-by-library-containers-com-docker-docker/15194

@lxhunter

This comment has been minimized.

Show comment
Hide comment
@lxhunter

lxhunter commented Aug 20, 2016

+1

@simond

This comment has been minimized.

Show comment
Hide comment
@simond

simond Aug 23, 2016

Yep, same issue here.

simond commented Aug 23, 2016

Yep, same issue here.

@yankcrime

This comment has been minimized.

Show comment
Hide comment
@yankcrime

yankcrime Aug 24, 2016

I've found the quickest way (as mentioned in other issues / forum threads) to fix this is to reset the client which forces the recreation of Docker.qcow2. If you don't want to do that however, there is another fix, once you've cleaned up any dangling images or exited containers.

NB: You'll need to install qemu via Homebrew as this process requires qemu-img to recompress the qcow2 disk image.

Connect to the VM with screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty and then login as root

Fill up the disk with zeroes:

dd if=/dev/zero of=/var/tempfile

This'll take at least couple of minutes on SSD. I wouldn't bother if you're on spinning rust...

Then you need to rm /var/tempfile, logout of the VM, and quit the Docker client entirely. Now we can recompress the disk:

$ pwd
/Users/nick/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux
$ mv Docker.qcow2 Docker.qcow2.original
$ du -hs Docker.qcow2.original
12G     Docker.qcow2.original
$ qemu-img convert -O qcow2 Docker.qcow2.original Docker.qcow2
$ rm Docker.qcow2.original
$ du -hs Docker.qcow2
772M    Docker.qcow2

yankcrime commented Aug 24, 2016

I've found the quickest way (as mentioned in other issues / forum threads) to fix this is to reset the client which forces the recreation of Docker.qcow2. If you don't want to do that however, there is another fix, once you've cleaned up any dangling images or exited containers.

NB: You'll need to install qemu via Homebrew as this process requires qemu-img to recompress the qcow2 disk image.

Connect to the VM with screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty and then login as root

Fill up the disk with zeroes:

dd if=/dev/zero of=/var/tempfile

This'll take at least couple of minutes on SSD. I wouldn't bother if you're on spinning rust...

Then you need to rm /var/tempfile, logout of the VM, and quit the Docker client entirely. Now we can recompress the disk:

$ pwd
/Users/nick/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux
$ mv Docker.qcow2 Docker.qcow2.original
$ du -hs Docker.qcow2.original
12G     Docker.qcow2.original
$ qemu-img convert -O qcow2 Docker.qcow2.original Docker.qcow2
$ rm Docker.qcow2.original
$ du -hs Docker.qcow2
772M    Docker.qcow2
@geerlingguy

This comment has been minimized.

Show comment
Hide comment
@geerlingguy

geerlingguy Sep 2, 2016

@yankcrime's workaround works... but it's a lot of steps to make Docker not continuously gobble up all the remaining space on my poor little MacBook's SSD. Between this and osxfs perf issues (#77) requiring similarly involved file sync workarounds, it's really hard to recommend Docker for Mac to anyone who's not an infra hacker at heart... I don't want to have to spend half my day helping other devs figure out why all their disk space is gone!

geerlingguy commented Sep 2, 2016

@yankcrime's workaround works... but it's a lot of steps to make Docker not continuously gobble up all the remaining space on my poor little MacBook's SSD. Between this and osxfs perf issues (#77) requiring similarly involved file sync workarounds, it's really hard to recommend Docker for Mac to anyone who's not an infra hacker at heart... I don't want to have to spend half my day helping other devs figure out why all their disk space is gone!

@kevinsimper

This comment has been minimized.

Show comment
Hide comment
@kevinsimper

kevinsimper Sep 5, 2016

Just deleting it seems to be the easiest solution, but it would be nice if the client could reclaim space itself!

kevinsimper commented Sep 5, 2016

Just deleting it seems to be the easiest solution, but it would be nice if the client could reclaim space itself!

@gumho

This comment has been minimized.

Show comment
Hide comment
@gumho

gumho Sep 6, 2016

I've been running into this issue as well on the latest Stable Docker for Mac. @yankcrime's workaround does work. Would be nice to be able to provide an option to grow the disk size as well.

gumho commented Sep 6, 2016

I've been running into this issue as well on the latest Stable Docker for Mac. @yankcrime's workaround does work. Would be nice to be able to provide an option to grow the disk size as well.

@creynders

This comment has been minimized.

Show comment
Hide comment
@creynders

creynders commented Sep 8, 2016

👍 💯

@chrisclark

This comment has been minimized.

Show comment
Hide comment
@chrisclark

chrisclark Sep 9, 2016

To add to this, I just deleted my qcow2 file and restart docker and disconnect completely from the internet, and docker ps shows no running containers, and docker images -a lists two containers, both of which are < 300mb...Docker will recreate the qcow2 file and I can just sit there, watching it grow in size, while Docker, and I, do absolutely nothing. There are no running containers, no nothing. What on earth is it even doing? I'm literally sitting here staring at the file in Finder (after re-enabling my network...) watching it grow from 200kb to...

...1.14GB
...1.15GB
...1.16GB

This is nuts. It'll happily just keep growing until it eats all of my available hard disk space.

chrisclark commented Sep 9, 2016

To add to this, I just deleted my qcow2 file and restart docker and disconnect completely from the internet, and docker ps shows no running containers, and docker images -a lists two containers, both of which are < 300mb...Docker will recreate the qcow2 file and I can just sit there, watching it grow in size, while Docker, and I, do absolutely nothing. There are no running containers, no nothing. What on earth is it even doing? I'm literally sitting here staring at the file in Finder (after re-enabling my network...) watching it grow from 200kb to...

...1.14GB
...1.15GB
...1.16GB

This is nuts. It'll happily just keep growing until it eats all of my available hard disk space.

@DavidCommonTW

This comment has been minimized.

Show comment
Hide comment
@DavidCommonTW

DavidCommonTW Sep 12, 2016

@chrisclark I have the same problem. There is no image, no container, qcow2 just keeping grow up, but until to 1.16GB. Seem 1.16GB is the VM basic requested working size.

DavidCommonTW commented Sep 12, 2016

@chrisclark I have the same problem. There is no image, no container, qcow2 just keeping grow up, but until to 1.16GB. Seem 1.16GB is the VM basic requested working size.

@marvin-w

This comment has been minimized.

Show comment
Hide comment
@marvin-w

marvin-w Sep 13, 2016

Same issue for me

marvin-w commented Sep 13, 2016

Same issue for me

@AzatKhalilov

This comment has been minimized.

Show comment
Hide comment
@AzatKhalilov

AzatKhalilov Sep 13, 2016

Yep! The same issue for me. If often rebuild my images it's trouble,headache! After 5-6 rebuild image has 1G size but disk space decrease on 20-30G!!!!!

AzatKhalilov commented Sep 13, 2016

Yep! The same issue for me. If often rebuild my images it's trouble,headache! After 5-6 rebuild image has 1G size but disk space decrease on 20-30G!!!!!

@PommeVerte

This comment has been minimized.

Show comment
Hide comment
@PommeVerte

PommeVerte Sep 14, 2016

Can we have an update on this? It's a really big issue that needs solving.
I can appreciate that it might not be an easy fix but some kind of feedback would be awesome.

PommeVerte commented Sep 14, 2016

Can we have an update on this? It's a really big issue that needs solving.
I can appreciate that it might not be an easy fix but some kind of feedback would be awesome.

@alex-ross

This comment has been minimized.

Show comment
Hide comment
@alex-ross

alex-ross Sep 14, 2016

Using Docker to manage multiple development environments and this image often ends up to take 60GB for me. It's a lot on my 256GB SSD. All dev environments is using Docker Compose so I can reset docker once in a while but it's really annoying that I have to do so.

alex-ross commented Sep 14, 2016

Using Docker to manage multiple development environments and this image often ends up to take 60GB for me. It's a lot on my 256GB SSD. All dev environments is using Docker Compose so I can reset docker once in a while but it's really annoying that I have to do so.

@anonymuse

This comment has been minimized.

Show comment
Hide comment
@anonymuse

anonymuse Sep 20, 2016

I've been using Docker for Mac and the Docker Machine images in order to experiment effectively with Docker Swarm and other orchestration features -- the large qcow file and not-insubstantial disks of my other Docker Machine hosts make it difficult to fit onto a MBP sized HDD.

Any update here?

anonymuse commented Sep 20, 2016

I've been using Docker for Mac and the Docker Machine images in order to experiment effectively with Docker Swarm and other orchestration features -- the large qcow file and not-insubstantial disks of my other Docker Machine hosts make it difficult to fit onto a MBP sized HDD.

Any update here?

@wbednarski

This comment has been minimized.

Show comment
Hide comment
@wbednarski

wbednarski Sep 20, 2016

If you can remove all images/containers then:

Stop Docker.

docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
docker volume rm $(docker volume ls |awk '{print $2}')
rm -rf ~/Library/Containers/com.docker.docker/Data/*

Start Docker, you have yours GB back.

wbednarski commented Sep 20, 2016

If you can remove all images/containers then:

Stop Docker.

docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
docker volume rm $(docker volume ls |awk '{print $2}')
rm -rf ~/Library/Containers/com.docker.docker/Data/*

Start Docker, you have yours GB back.

@anonymuse

This comment has been minimized.

Show comment
Hide comment
@anonymuse

anonymuse Sep 20, 2016

@wbednarski -- that's helpful if keeping images/containers around wasn't a big deal (as it might not be on a server) but it tends to be important in development (on my laptop.) I've read also that using the "Reset to factory defaults" button can clean up your installation too.

Thanks for pointing out your solution.

anonymuse commented Sep 20, 2016

@wbednarski -- that's helpful if keeping images/containers around wasn't a big deal (as it might not be on a server) but it tends to be important in development (on my laptop.) I've read also that using the "Reset to factory defaults" button can clean up your installation too.

Thanks for pointing out your solution.

@wbednarski

This comment has been minimized.

Show comment
Hide comment
@wbednarski

wbednarski Sep 20, 2016

Didn't try reset. I'll check out it soon :]

@anonymuse you can look at @yankcrime solution #371 (comment) if you want to keep images/containers.

wbednarski commented Sep 20, 2016

Didn't try reset. I'll check out it soon :]

@anonymuse you can look at @yankcrime solution #371 (comment) if you want to keep images/containers.

@lijingpeng

This comment has been minimized.

Show comment
Hide comment
@lijingpeng

lijingpeng Sep 23, 2016

@yankcrime I tried as you said, but nothing changed.

du -hs Docker.qcow2
17G Docker.qcow2

lijingpeng commented Sep 23, 2016

@yankcrime I tried as you said, but nothing changed.

du -hs Docker.qcow2
17G Docker.qcow2

@anonymuse

This comment has been minimized.

Show comment
Hide comment
@anonymuse

anonymuse Sep 24, 2016

Frank, how sure are you that you followed all of the steps correctly? Do
you have the output of your actions that we could take a peek at?

I forgot to run the dd myself first time around!

On Sep 23, 2016 12:07 PM, "Frank" notifications@github.com wrote:

@yankcrime https://github.com/yankcrime I tried as you said, but
nothing changed.

du -hs Docker.qcow2
17G Docker.qcow2


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#371 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACnE0XEI6hkeOnvHaxI2_WR4CYBR1gCgks5qs_kigaJpZM4Jo3w2
.

anonymuse commented Sep 24, 2016

Frank, how sure are you that you followed all of the steps correctly? Do
you have the output of your actions that we could take a peek at?

I forgot to run the dd myself first time around!

On Sep 23, 2016 12:07 PM, "Frank" notifications@github.com wrote:

@yankcrime https://github.com/yankcrime I tried as you said, but
nothing changed.

du -hs Docker.qcow2
17G Docker.qcow2


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#371 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACnE0XEI6hkeOnvHaxI2_WR4CYBR1gCgks5qs_kigaJpZM4Jo3w2
.

@davidthornton

This comment has been minimized.

Show comment
Hide comment
@davidthornton

davidthornton Feb 11, 2018

@djs55 The new raw format on stable is working a treat. I really love the iteration speed core Docker OS teams can produce quality software.

I did have one question though; my only options for virtual disk size are 32gb and 64gb, however my colleague who has an older mac (but upgraded his disk) is up to 256gb. I have tried uninstalled and reinstalling Docker for mac. Is this a hardware restriction on my machine? I am not fully across why I would be restricted to 64gb (and hence don't know if it's a Docker or OS limitation).

Thank you for your contributions to the open source community!

davidthornton commented Feb 11, 2018

@djs55 The new raw format on stable is working a treat. I really love the iteration speed core Docker OS teams can produce quality software.

I did have one question though; my only options for virtual disk size are 32gb and 64gb, however my colleague who has an older mac (but upgraded his disk) is up to 256gb. I have tried uninstalled and reinstalling Docker for mac. Is this a hardware restriction on my machine? I am not fully across why I would be restricted to 64gb (and hence don't know if it's a Docker or OS limitation).

Thank you for your contributions to the open source community!

@davidthornton

This comment has been minimized.

Show comment
Hide comment
@davidthornton

davidthornton Feb 11, 2018

Update: Docker is truly a revolution.

I just spent three hours syncing large files off my computer, making on the order of 100gb available to macOS. After removing and re-installing Docker for mac (making sure all my container state - i.e. databases was backed up etc.) I have the option to resize my sparse file to 256gb 🎉🎉🎉🚀💥🔥🎉😍

davidthornton commented Feb 11, 2018

Update: Docker is truly a revolution.

I just spent three hours syncing large files off my computer, making on the order of 100gb available to macOS. After removing and re-installing Docker for mac (making sure all my container state - i.e. databases was backed up etc.) I have the option to resize my sparse file to 256gb 🎉🎉🎉🚀💥🔥🎉😍

@rsn8887

This comment has been minimized.

Show comment
Hide comment
@rsn8887

rsn8887 Feb 11, 2018

I just don’t understand why docker even needs this file. I thought the whole idea of docker was that it is not a vm. So why does it create a vm then?

rsn8887 commented Feb 11, 2018

I just don’t understand why docker even needs this file. I thought the whole idea of docker was that it is not a vm. So why does it create a vm then?

@davidthornton

This comment has been minimized.

Show comment
Hide comment
@davidthornton

davidthornton Feb 11, 2018

It is a lightweight virtual machine, which stores the entire state of any number of operating systems and user land applications in a single file. That file is called Docker.raw which is a sparse file on newer macOS SSDs.

davidthornton commented Feb 11, 2018

It is a lightweight virtual machine, which stores the entire state of any number of operating systems and user land applications in a single file. That file is called Docker.raw which is a sparse file on newer macOS SSDs.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 11, 2018

Member

I thought the whole idea of docker was that it is not a vm. So why does it create a vm then?

Correct, containers are not VM’s, and containers share the same kernel (and resources).

However, Docker for Mac enables you to run Linux containers on a macOS machine; for that, it needs a Linux kernel to run those containers, and a very minimal VM (based on LinuxKit) is used

Member

thaJeztah commented Feb 11, 2018

I thought the whole idea of docker was that it is not a vm. So why does it create a vm then?

Correct, containers are not VM’s, and containers share the same kernel (and resources).

However, Docker for Mac enables you to run Linux containers on a macOS machine; for that, it needs a Linux kernel to run those containers, and a very minimal VM (based on LinuxKit) is used

@davidthornton

This comment has been minimized.

Show comment
Hide comment
@davidthornton

davidthornton Feb 11, 2018

Oh yeah I forgot about that @thaJeztah because it's so good/transparent it makes me feel like I'm on a linux machine 😂

davidthornton commented Feb 11, 2018

Oh yeah I forgot about that @thaJeztah because it's so good/transparent it makes me feel like I'm on a linux machine 😂

@velconia

This comment has been minimized.

Show comment
Hide comment
@velconia

velconia Mar 2, 2018

Same issue for me, thanks

velconia commented Mar 2, 2018

Same issue for me, thanks

@qiuhuaqi

This comment has been minimized.

Show comment
Hide comment
@qiuhuaqi

qiuhuaqi Mar 5, 2018

Looks like Docker reverted the disk image format to qcow2 in recent stable release (17.12.0-ce-mac55 release note)

Since we are stuck with qcow2 for a while, I'm wondering how big should I set the cap to. What happens if you hit the limit?

qiuhuaqi commented Mar 5, 2018

Looks like Docker reverted the disk image format to qcow2 in recent stable release (17.12.0-ce-mac55 release note)

Since we are stuck with qcow2 for a while, I'm wondering how big should I set the cap to. What happens if you hit the limit?

@DebdiptoM

This comment has been minimized.

Show comment
Hide comment
@DebdiptoM

DebdiptoM Mar 15, 2018

Still a problem . Is there a JIRA issue we can vote for to have it fixed?

DebdiptoM commented Mar 15, 2018

Still a problem . Is there a JIRA issue we can vote for to have it fixed?

@pgayvallet

This comment has been minimized.

Show comment
Hide comment
@pgayvallet

pgayvallet Mar 26, 2018

@davidthornton we are limiting the resize options depending on the current space left on your device.

pgayvallet commented Mar 26, 2018

@davidthornton we are limiting the resize options depending on the current space left on your device.

@davidthornton

This comment has been minimized.

Show comment
Hide comment
@davidthornton

davidthornton Mar 27, 2018

@pgayvallet thanks :) I figured that out pretty quickly after I made some space!

I do however have a question re edge/qcow - my APFS raw file has saved my life (it works perfectly) BUT I am getting kicked off Docker Cloud in “51 days” so I need Kubernetes support but if I switch to edge to get it, will I lose APFS support?

Ideally I’d like APFS and k8s support before you retire the Docker Cloud runtime... Obviously no one from Docker sales have responded to me tweets or the hacker news article - can you give me any updates on whether Docker for Mac will have any of those features in the next 51 days?

davidthornton commented Mar 27, 2018

@pgayvallet thanks :) I figured that out pretty quickly after I made some space!

I do however have a question re edge/qcow - my APFS raw file has saved my life (it works perfectly) BUT I am getting kicked off Docker Cloud in “51 days” so I need Kubernetes support but if I switch to edge to get it, will I lose APFS support?

Ideally I’d like APFS and k8s support before you retire the Docker Cloud runtime... Obviously no one from Docker sales have responded to me tweets or the hacker news article - can you give me any updates on whether Docker for Mac will have any of those features in the next 51 days?

@akaghzi

This comment has been minimized.

Show comment
Hide comment
@akaghzi

akaghzi Jun 15, 2018

this is what worked for me:
docker images
this will give you list of images on your mac
docker image prune
docker container prune
repeat last 2 steps
restart docker

akaghzi commented Jun 15, 2018

this is what worked for me:
docker images
this will give you list of images on your mac
docker image prune
docker container prune
repeat last 2 steps
restart docker

@joadr

This comment has been minimized.

Show comment
Hide comment
@joadr

joadr Jun 22, 2018

@qiuhuaqi I've found out about this qcow2 file by reaching the limit. Your machines start telling they don't have enough disk space. Nothing terrible, because you can delete the file and get space again, or increase the qcow file limit. But, you have to keep an eye from time to time to avoid wasting extra time.

joadr commented Jun 22, 2018

@qiuhuaqi I've found out about this qcow2 file by reaching the limit. Your machines start telling they don't have enough disk space. Nothing terrible, because you can delete the file and get space again, or increase the qcow file limit. But, you have to keep an eye from time to time to avoid wasting extra time.

@djs55

This comment has been minimized.

Show comment
Hide comment
@djs55

djs55 Jun 26, 2018

Contributor

@davidthornton Docker for Mac edge contains the latest features including APFS raw file support and k8s support. k8s support will be added to a future stable version too.

Contributor

djs55 commented Jun 26, 2018

@davidthornton Docker for Mac edge contains the latest features including APFS raw file support and k8s support. k8s support will be added to a future stable version too.

@davidthornton

This comment has been minimized.

Show comment
Hide comment
@davidthornton

davidthornton Jun 26, 2018

Thanks! I managed to migrate everything in time. I'm now running raw file support and k8s on Docker for Mac Edge and k8s in production on AWS. Apart from Docker Cloud retiring, things are running pretty smoothly now.

davidthornton commented Jun 26, 2018

Thanks! I managed to migrate everything in time. I'm now running raw file support and k8s on Docker for Mac Edge and k8s in production on AWS. Apart from Docker Cloud retiring, things are running pretty smoothly now.

@djs55

This comment has been minimized.

Show comment
Hide comment
@djs55

djs55 Jun 26, 2018

Contributor

The current state is:

  • on edge 18.05.0-ce-mac67 we default to Docker.raw on macOS 10.13.4 (and later) running on APFS, otherwise we use Docker.qcow2
  • on stable 18.03.1-ce-mac65 we default to Docker.qcow2. The next version of stable will default to Docker.raw on macOS 10.13.4 (and later) running on APFS.

Note that APFS (needed by Docker.raw) is not supported by all Apple hardware. I believe it's currently only supported on SSD drives, although it might work on fusion drives / HDDs in the future.

Both Docker.raw and Docker.qcow2 support TRIM for space reclamation. A TRIM command is issued after every image delete event, but can also be manually triggered with

docker run --privileged --pid=host justincormack/nsenter1 /sbin/fstrim /var/lib/docker

TRIM on Docker.raw frees space immediately. TRIM on Docker.qcow2 only marks the blocks as free; they will be physically released later by a garbage collection process-- this can take several minutes.

I recommend:

  • use Docker.raw if possible. This requires an SSD, a recent version of macOS (10.13.4+) and running the edge version. Note if you are currently using a Docker.qcow2 you will need to reset to factory defaults and rebuild your images.
  • monitor images (docker image ls), stopped containers (docker container ls -a) and prune old ones regularly (docker system prune). It's easy to build up stale objects by accident particularly if you use docker run a lot without the --rm argument.
  • use ls -s to query the size of the Docker.raw to measure the allocated blocks, not the theoretical size of the disk
Contributor

djs55 commented Jun 26, 2018

The current state is:

  • on edge 18.05.0-ce-mac67 we default to Docker.raw on macOS 10.13.4 (and later) running on APFS, otherwise we use Docker.qcow2
  • on stable 18.03.1-ce-mac65 we default to Docker.qcow2. The next version of stable will default to Docker.raw on macOS 10.13.4 (and later) running on APFS.

Note that APFS (needed by Docker.raw) is not supported by all Apple hardware. I believe it's currently only supported on SSD drives, although it might work on fusion drives / HDDs in the future.

Both Docker.raw and Docker.qcow2 support TRIM for space reclamation. A TRIM command is issued after every image delete event, but can also be manually triggered with

docker run --privileged --pid=host justincormack/nsenter1 /sbin/fstrim /var/lib/docker

TRIM on Docker.raw frees space immediately. TRIM on Docker.qcow2 only marks the blocks as free; they will be physically released later by a garbage collection process-- this can take several minutes.

I recommend:

  • use Docker.raw if possible. This requires an SSD, a recent version of macOS (10.13.4+) and running the edge version. Note if you are currently using a Docker.qcow2 you will need to reset to factory defaults and rebuild your images.
  • monitor images (docker image ls), stopped containers (docker container ls -a) and prune old ones regularly (docker system prune). It's easy to build up stale objects by accident particularly if you use docker run a lot without the --rm argument.
  • use ls -s to query the size of the Docker.raw to measure the allocated blocks, not the theoretical size of the disk

djs55 added a commit to djs55/docker.github.io that referenced this issue Jun 26, 2018

Docker for Mac: add page on disk space
The disk space usage of Docker on Mac has been difficult to manage
and has confused lots of people, see

docker/for-mac#371

This patch adds a top-level section to the Docker for Mac docs
alongside networking which explains

- where the space has gone
- how to reclaim some of it

In future I'd like to add more about why some systems use `Docker.raw`
and some `Docker.qcow2` but we're in a transitional state at the moment.
It'll be simpler to explain after the next stable version has been
released.

Signed-off-by: David Scott <dave.scott@docker.com>

djs55 added a commit to djs55/docker.github.io that referenced this issue Jun 26, 2018

Docker for Mac: add page on disk space
The disk space usage of Docker on Mac has been difficult to manage
and has confused lots of people, see

docker/for-mac#371

This patch adds a top-level section to the Docker for Mac docs
alongside networking which explains

- where the space has gone
- how to reclaim some of it

In future I'd like to add more about why some systems use `Docker.raw`
and some `Docker.qcow2` but we're in a transitional state at the moment.
It'll be simpler to explain after the next stable version has been
released.

Signed-off-by: David Scott <dave.scott@docker.com>
@gotoin

This comment has been minimized.

Show comment
Hide comment
@gotoin

gotoin Jul 10, 2018

Why hasn't the problem been fixed for two years? my hard disk is exploding!

~~!@

gotoin commented Jul 10, 2018

Why hasn't the problem been fixed for two years? my hard disk is exploding!

~~!@
@BretFisher

This comment has been minimized.

Show comment
Hide comment
@BretFisher

BretFisher Jul 10, 2018

@gotoin have you not tried the latest post from @djs55 ? The problem is solved on modern hardware and Edge release. Stable is about to get its quarterly update to be in line with Edge fixes.

BretFisher commented Jul 10, 2018

@gotoin have you not tried the latest post from @djs55 ? The problem is solved on modern hardware and Edge release. Stable is about to get its quarterly update to be in line with Edge fixes.

@gotoin

This comment has been minimized.

Show comment
Hide comment
@gotoin

gotoin Jul 10, 2018

@BretFisher Ok, thanks. i'll try

gotoin commented Jul 10, 2018

@BretFisher Ok, thanks. i'll try

@durbanski

This comment has been minimized.

Show comment
Hide comment
@durbanski

durbanski Jul 13, 2018

In my case (macOS Sierra 10.12.6), docker Version 18.03.1-ce-mac65 (24312).
Docker.qcow can grow up to 100 Gigabytes per day. Insane.
Single ajax request to the server, makes Docker.qcow 300 Megabytes bigger each time request is done. If this is expected behavour, then I'm not sure about docker at all (using docker for a year now).
Cleaning all the time is PITA.

durbanski commented Jul 13, 2018

In my case (macOS Sierra 10.12.6), docker Version 18.03.1-ce-mac65 (24312).
Docker.qcow can grow up to 100 Gigabytes per day. Insane.
Single ajax request to the server, makes Docker.qcow 300 Megabytes bigger each time request is done. If this is expected behavour, then I'm not sure about docker at all (using docker for a year now).
Cleaning all the time is PITA.

@joshenders

This comment has been minimized.

Show comment
Hide comment
@joshenders

joshenders Jul 14, 2018

joshenders commented Jul 14, 2018

@eladchen

This comment has been minimized.

Show comment
Hide comment
@eladchen

eladchen Jul 25, 2018

@BretFisher It seems the edge build is still not the solution. (I don't think it should ever be)

Checkout issue-2625 & Edge Change Log

Is there a solution in the horizon? This is a major issue, working with docker comes with a big disk penalty at the moment.

eladchen commented Jul 25, 2018

@BretFisher It seems the edge build is still not the solution. (I don't think it should ever be)

Checkout issue-2625 & Edge Change Log

Is there a solution in the horizon? This is a major issue, working with docker comes with a big disk penalty at the moment.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 25, 2018

Member

@eladchen
#2625 was resolved (it was a bug in macOS); the version of Docker for Mac edge you're linking to is not the latest edge release; Docker for Mac 18.05 and up default to the .raw format again (see
#371 (comment) a few comments up). Note that Docker 18.06 will be available (or already is), which also uses .raw (on supported systems)

Member

thaJeztah commented Jul 25, 2018

@eladchen
#2625 was resolved (it was a bug in macOS); the version of Docker for Mac edge you're linking to is not the latest edge release; Docker for Mac 18.05 and up default to the .raw format again (see
#371 (comment) a few comments up). Note that Docker 18.06 will be available (or already is), which also uses .raw (on supported systems)

@brock-bouchard

This comment has been minimized.

Show comment
Hide comment
@brock-bouchard

brock-bouchard Jul 25, 2018

  • I just wiped all of my data and upgraded to Docker for Mac 18.06.0-ce-mac70, enabling the raw format by default. Now while decompressing a large .tar.gz archive, I'm running into #2388 which appears to still be outstanding. It would seem the raw format has trouble with large images and files (?)

  • Is there a way to reset Docker for Mac to qcow other than downgrading, resetting data, and re-upgrading?

  • FWIW with raw I'm still seeing storage leaking even after docker system prune --volumes. I still need to run docker run --privileged --pid=host justincormack/nsenter1 /sbin/fstrim /var/lib/docker to reclaim space, however it does return immediately with raw as @djs55 mentioned.

brock-bouchard commented Jul 25, 2018

  • I just wiped all of my data and upgraded to Docker for Mac 18.06.0-ce-mac70, enabling the raw format by default. Now while decompressing a large .tar.gz archive, I'm running into #2388 which appears to still be outstanding. It would seem the raw format has trouble with large images and files (?)

  • Is there a way to reset Docker for Mac to qcow other than downgrading, resetting data, and re-upgrading?

  • FWIW with raw I'm still seeing storage leaking even after docker system prune --volumes. I still need to run docker run --privileged --pid=host justincormack/nsenter1 /sbin/fstrim /var/lib/docker to reclaim space, however it does return immediately with raw as @djs55 mentioned.

@djs55

This comment has been minimized.

Show comment
Hide comment
@djs55

djs55 Jul 25, 2018

Contributor

@brock-bouchard to switch back to qcow, edit

~/Library/Group Containers/group.com.docker/settings.json

replace the .raw extension with .qcow and restart the app.

Re: the storage leaking -- the automatic fstrim is only triggered after an image deletion. If you have a lot of file churn within a container it won't run. I'll investigate to see how best to run it more often.

Contributor

djs55 commented Jul 25, 2018

@brock-bouchard to switch back to qcow, edit

~/Library/Group Containers/group.com.docker/settings.json

replace the .raw extension with .qcow and restart the app.

Re: the storage leaking -- the automatic fstrim is only triggered after an image deletion. If you have a lot of file churn within a container it won't run. I'll investigate to see how best to run it more often.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment