Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker.raw reserving too much size #2297

Closed
aitrusgit opened this issue Dec 7, 2017 · 70 comments

Comments

@aitrusgit
Copy link

commented Dec 7, 2017

Expected behavior

Docker.raw should not grow much larger than the space used by docker containers, images, volumes.

Actual behavior

I have very few images downloaded locally, with no active containers and just a two volumes with some data.

➜  com.docker.driver.amd64-linux docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              19                  0                   3.628GB             3.628GB (100%)
Containers          0                   0                   0B                  0B
Local Volumes       2                   0                   803.5MB             803.5MB (100%)
Build Cache                                                 0B                  0B

Nevertheless Docker.raw size is huge!

➜  com.docker.driver.amd64-linux ls -l Docker.raw
-rw-r--r--@ 1 cf  staff  68719476736 Dec  7 10:35 Docker.raw

Information

Diagnose info:

Docker for Mac: version: 17.11.0-ce-mac40 (b2149617ed8d7263c9c035e25fe71d147169348c)
macOS: version 10.13.1 (build: 17B1003)
logs: /tmp/F397F806-6CA7-40E3-9C36-79B5F0ED519D/20171207-103808.tar.gz
[OK]     db.git
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     kubernetes
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk
@aitrusgit

This comment has been minimized.

Copy link
Author

commented Dec 7, 2017

I also tried to do a full system prune

➜  com.docker.driver.amd64-linux docker volume prune
WARNING! This will remove all volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
033ce3294aa395bbb8c468eba823622d65b5b1d7f042c4a7ebf29826aed6ea40
2901af7dd8d50ae74ad761fe33cc558881d46ab323e7f6ee6eb87c3337ed8508

Total reclaimed space: 803.5MB
➜  com.docker.driver.amd64-linux docker system prune -a
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all images without at least one container associated to them
        - all build cache
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: postgres:9.5.4
untagged: postgres@sha256:1480f2446dabb1116fafa426ac530d2404277873a84dc4a4d0d9d4b37a5601e8
deleted: sha256:2417ea518abc0db32cf2fb0d021ce57dab2e16f480ac924000e
52afd07d3b0a4
...
deleted: sha256:b51149973e6a6c4fb1091ef34ff70002ee307e971b9982075cf226004a93c9b7

Total reclaimed space: 3.628GB

Nevertheless Docker.raw size didn't shrink even after restarting docker engine

@aitrusgit

This comment has been minimized.

Copy link
Author

commented Dec 7, 2017

I even tried removing Docker.raw file manually and, after restarting docker, it gets recreated with the same size:

➜  com.docker.driver.amd64-linux ls -l Docker.raw
-rw-r--r--@ 1 cf  staff  68719476736 Dec  7 10:44 Docker.raw

Is that file size independent of my volumes, images, containers? 64 GB is quite a big chunk of data for an small SSD HD.

@djs55

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2017

ls -l shows the "size" of the file, not the number of sectors allocated by the filesystem. Try ls -sl:

$ cd ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux
$ ls -sl Docker.raw 
16248952 -rw-r--r--@ 1 user  staff  ` Dec  7 13:57 Docker.raw

In my case Docker.raw is only using 16248952 filesystem sectors which is much less than the maximum file size of 68719476736 bytes. If I download some images then the 16248952 increases, and if I docker system prune then it decreases again.

I think this is working as expected so I'll close the ticket but nevertheless it is a little confusing. Perhaps we could present the current space usage in the UI somehow-- I'll escalate this suggestion to our UI team.

Thanks for the report and for using Docker for Mac!

@djs55 djs55 closed this Dec 7, 2017

@codesman

This comment has been minimized.

Copy link

commented Jan 15, 2018

Just updated to 17.12.0-ce-mac47 on 10.13.2 on a 13" MBA, reset the preferences and deleted all the things.
Preferences say the virtual disk image size is 64GB(3.5GB allocated)
Previously, my .qcow2 file was ~6.5GB
Now ls -sl Docker.raw reads 6840456 -rw-r--r--@ 1 user staff 68719476736 Jan 14 11:03 Docker.raw - 64GB. When I use GrandPerspective, it also reports that the .raw file is 64GB in size, when it used to report that the qcow2 was 6.5GB and it had some images in it, not a fresh install. Looks like there might still be a problem here.

@brown131

This comment has been minimized.

Copy link

commented Jan 15, 2018

I'm on the same version as codesman, and I too am seeing the same problem. My Docker.raw file is 64G even after I cleared all images in my environment.

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Jan 16, 2018

The 64GB you see is the "logical" size of the image, not the physical size; see Docker.raw consumes an insane amount of disk space! in the documentation:

Docker uses the raw format on Macs running the Apple Filesystem (APFS). APFS supports sparse files, which compress long runs of zeroes representing unused space. The output of ls is misleading, because it lists the logical size of the file rather than its physical size. To see the physical size, add the -ks switch; to see the logical size in human readable form, add -lh:

@tjsousa

This comment has been minimized.

Copy link

commented Feb 6, 2018

Adding to this information, I believe the issue is that most tools in the Apple ecosystem still rely on information from ls to report on available disk space.

In particular, Apple's own Software Update tool will refuse to install upgrades without reclaiming that space, and removing Docker.raw effectively unblocked the situation for me (by "freeing" 16GB, my predefined size for Docker image).

@stepmuel

This comment has been minimized.

Copy link

commented Feb 14, 2018

I was using rsync to do simple backup of my home directory when I discovered the humongous size of Docker.raw. I was able to solve my problem by moving the file to /scratch under Preferences > Disk. It also had an option to switch the size from 64GB to 32GB, which is better, but still very large. I'd rather not backup GBs of zeros or else loose data I actually do want.

Since the file doesn't actually use disk space until it is "filled", I guess this is fine. But additional options for smaller file sizes might still be beneficial in certain situations. All in all I even doubt that the benefit of preallocating all that space is worth the weird side effects and head aches many user will experience because of it. Keep in mind that many not-very-technical users are instructed to install docker for a variety of reasons.

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Feb 14, 2018

@stepmuel it’s a sparse file, and the usual approach for VM disk images.

This article may be of interest to you; https://gergap.wordpress.com/2013/08/10/rsync-and-sparse-files/amp/

@matschaffer

This comment has been minimized.

Copy link

commented May 16, 2018

I suspect this also causes Finder's "available" number to do strange things. Today I seemed to gain available space by copying data onto my laptop. 😕

@YRM64

This comment has been minimized.

Copy link

commented May 19, 2018

My comments are in congruence with statements posted by djs55; adding that it's unreasonable to think docker.raw consumes, or is reserving too much (disk?) size. Because Docker utilizes the raw format on Macs running the Apple File System (APFS), APFS supports sparse files "while compressing long runs of zeroes representing un-used space" (source: docs.docker.com).

Another option in determining actual disk usage, and complementing djs55:

$ du -h Docker.raw
2,2G Docker.raw

@sudo-bmitch

This comment has been minimized.

Copy link

commented Jul 31, 2018

After you delete a large file from inside the Linux VM, will the space be reclaimed from the Docker.raw file? From this stackoverflow question I'm suspecting the answer is no. My suspicion is that Linux does it's normal filesystem delete where it just unlinks the inode reference, but the underlying sectors on the disk are not zeroed out. And without those filesystem bits being zeroed out, APFS will never reclaim the sparse bits of the file. Is there a better option to reclaim that disk space other than wiping the Docker.raw file and starting fresh?

@jonnyijapan

This comment has been minimized.

Copy link

commented Sep 19, 2018

Still eating up way too much space, believe it or not...

@shannon-fluellen-aurea

This comment has been minimized.

Copy link

commented Sep 26, 2018

While I understand that the space isn't technically being used, it's still very annoying to have my Mac think I'm running out of space because it sees docker.raw eating up a quarter of my SSD. It causes low space notifications and, as mentioned by @tjsousa, will prevent things like software updates if the system thinks there isn't enough space b/c it views docker.raw as using what ls shows it using. I feel like there must be a way to solve this on Docker's end...

@kamil-jakubowski

This comment has been minimized.

Copy link

commented Sep 28, 2018

Same problem here.

$ docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              10                  6                   1.673GB             1.233GB (73%)
Containers          8                   0                   4.647GB             4.647GB (100%)
Local Volumes       5                   5                   3.529GB             0B (0%)
Build Cache         0                   0                   0B                  0B
$ ls -klsh Docker.raw
37883956 -rw-r--r--@ 1 Ciastek  staff    45G 28 wrz 15:01 Docker.raw

Docker consumes 36GB.
It does not react on deleting volumes/containers/files in containers...
Restart of docker / system does not work.
Only working - resizing of docker available space.

@ivanlara

This comment has been minimized.

Copy link

commented Oct 3, 2018

Same issue, my 128GB Mac keeps telling me that I'm running out of space every 2 minutes or so because of this, how do I take care of this 64GB Docker.raw space illusion.

@aviris

This comment has been minimized.

Copy link

commented Oct 3, 2018

I understand how this is supposed to work, but the bottom line is that both ls -sl and du -h docker.raw return that my docker.raw file is 64GB. Doesn't matter how many or few containers I have, as far as macOS 10.13 or 10.14 are concerned, my docker.raw file is 64GB.

Moving my docker.raw file to an external drive freed up a total of 64 GB on the internal drive.

screenshot 10-03-2018 at 3 18 19 p

@jackfruhecolab

This comment has been minimized.

Copy link

commented Oct 4, 2018

I have the same issue as everyone else.

Also I have an actual apple sparse image that I use for something else, that image always shows the actual used size and not the provisioned size so it would seem that Docker is not using an actual apple sparse image...

@shaneHowearth

This comment has been minimized.

Copy link

commented Oct 4, 2018

I am experiencing the same issue but it is causing issues.

$ uname -rms
Darwin 17.7.0 x86_64
$ docker --version
Docker version 18.06.1-ce, build e68fc7a

When I try to run my postgres container I am told that I don't have enough disk space on the device. There's ~35 GB on the disk available, so it's referring to the container's disk space.

When I resize the amount of space available via docker preferences I have made 80GB available (62 GB on disk) everything is happy again, but none of my containers are close to using up the previously set disk allocation

This has only really started being an issue in the last 4-5days
s@work ~/go/src/project (develop) $ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
postgres 10.4-alpine 962ed899c609 2 months ago 72.9MB
docker.elastic.co/elasticsearch/elasticsearch 6.0.1 28259852697e 10 months ago 508MB
openlabs/docker-wkhtmltopdf-aas latest c61ee94c7fcb 3 years ago 747MB
s@work ~/go/src/project (develop) $ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
72918cb8860c openlabs/docker-wkhtmltopdf-aas "usr/local/bin/gunic…" 24 minutes ago Up 24 minutes 0.0.0.0:8800->80/tcp tap-wkhtmltopdf-aas
b9b5789999b5 docker.elastic.co/elasticsearch/elasticsearch:6.0.1 "/usr/local/bin/dock…" 25 minutes ago Up 25 minutes 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp tap-elasticsearch
7366131f37b3 postgres:10.4-alpine "docker-entrypoint.s…" 25 minutes ago Up 25 minutes 0.0.0.0:5432->5432/tcp tap-database

@fabiosussetto

This comment has been minimized.

Copy link

commented Oct 9, 2018

I don't understand how come this issue has been closed.
This is clearly a big problem, as Docker for Mac is eating up all disk space. The fact that the space may be just "virtually" used doesn't change a thing from the user's point of view.

In the FAQs (https://docs.docker.com/docker-for-mac/faqs/#disk-usage), it says:

"This is an illusion. [...] The output of ls is misleading, because it lists the logical size of the file rather than its physical size."

It may as well be an illusion, but I can't install software updates and MacOS is constantly complaining about running out of space.

Kindly reconsider opening this issue, as it's a very, very annoying thing that imo should not happen in stable software. I understand it may not be easy to fix, but this is Docker for Mac at the end of the day - saying that it's kind of MacOs fault is a bit of a stretch, at best.

Thanks.

Edit: ping @djs55
Would you please consider at least reopening this issue?
Regarding: "I think this is working as expected": people here have their macbooks unusable because of this disk space issue.

@mgrandi

This comment has been minimized.

Copy link

commented Oct 9, 2018

same thing as @aviris , i just did a git clone + docker compose on https://github.com/Runscope/requestbin#readme , and it installed like 2 images, which aren't that big:

[2018-10-09 16:19:39] markgrandi@Gypaetus:~$ docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              3                   2                   323.4MB             58.79MB (18%)
Containers          2                   0                   96.4kB              96.4kB (100%)
Local Volumes       2                   1                   1.238kB             0B (0%)
Build Cache         0                   0                   0B                  0B

but then the file is huge, according to ls -lah and GrandPerspective

[2018-10-09 16:20:46] markgrandi@Gypaetus:~$ ls -lah /Users/markgrandi/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw
-rw-r--r--  1 markgrandi  staff    60G Oct  9 16:20 /Users/markgrandi/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw

even if its not actually using up that much space because of an APFS sparse file, its still causing the "low disk" notification popup every 5 minutes which is highly annoying

screenshot_1043

@jjharsch

This comment has been minimized.

Copy link

commented Oct 11, 2018

Has anyone found a solution to this? Is this a non-issue on Docker Toolbox?

As it currents stands, Docker for Mac is making my 128gb SSD macbook unusable.

@fabiosussetto

This comment has been minimized.

Copy link

commented Oct 11, 2018

@jjharsch
warning: you are going to lose all your containers and images if you do the following (no way around this, sigh!)

Go to Preferences > Disk and reduce the disk image size allocation. I tried yesterday and it did reduce the .raw file size on my disk

@jjharsch

This comment has been minimized.

Copy link

commented Oct 11, 2018

@NoICE

This comment has been minimized.

Copy link

commented Oct 14, 2018

@aviris You have the docker.raw image on some external disk? Is that disk APFS? This sparse file works only on APFS.

The same applies if you have HFS(+) volume.

I have now a similar issue. I used Migration Assistant to copy all my files from one mac to another, and the migration assistant copied WHOLE 64GB docker.raw, ignoring that it's a sparse file.

So now I have allocated 64GB (according to du and Docker for Mac GUI "64GB (64.0GB on disk)" and I don't know how to re-enable sparse nature of it without deleting it.

@leehinde

This comment has been minimized.

Copy link

commented Oct 18, 2018

It'll eat up your time machine back up quickly as well. I'm in the process of moving it to an external drive/higher level folder to make blocking it in Time Machine easier. The system is sure acting like it's moving 64GB. :-)

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Oct 18, 2018

@leehinde you should be able to exclude it from Time Machine backups in the configuration panel (de-select the "Include VM in Time Machine Backups" checkbox)

screen shot 2018-10-18 at 18 29 47

@jjharsch

This comment has been minimized.

Copy link

commented Oct 18, 2018

Has anyone figured out how to make the max disk image size less than 16gb?

The MacOS "Your disk is almost full" is still being triggered at ~12gb free, despite only 4.7gb being used. This is a problem as it clears caches and indexes, including the spotlight index.

screen shot 2018-10-18 at 12 48 43 pm

@jackfruhecolab

This comment has been minimized.

Copy link

commented Oct 18, 2018

When I changed mine from 64 to 16 today and that did not free up the expected 48GB (OSX 10.13.6) My file size shrunk, but get info on my hard drive still showed roughly the same free space on the disk overall.

Perhaps this means that file wasn't actually taking all the space as we'd thought?

@elcolie

This comment has been minimized.

Copy link

commented Dec 4, 2018

@thaJeztah

However, Apple / macOS looks at the size it can grow to, instead of the size that it actually uses, and > therefore producing the errors that "there's no space left".

How to force docker update that can grow size and let osx know?

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Dec 4, 2018

@elcolie that size is the size that's configured in the preferences ("disk image size") #2297 (comment)

@elcolie

This comment has been minimized.

Copy link

commented Dec 4, 2018

@thaJeztah I read it. It is hard code!

@joshlewis

This comment has been minimized.

Copy link

commented Dec 22, 2018

In my opinion, this should be re-opened until a workable solution can be found.

@ssddawei

This comment has been minimized.

Copy link

commented Jan 16, 2019

@kamil-jakubowski I think we need to break down the space usage to see where it's used. There are lots of places where extra space is required including

  • the Linux swap file
  • the Linux filesystem metadata (both ext4 and overlayfs)
  • logs
  • Docker metadata
  • containers and images

Could you first try triggering a TRIM with

docker run --privileged --pid=host justincormack/nsenter1 /sbin/fstrim /var/lib/docker

(from #371 (comment))

and then query the disk usage from Linux's perspective with

$ docker run -it --privileged --pid=host justincormack/nsenter1 /bin/df -h /var/lib/docker
Filesystem                Size      Used Available Use% Mounted on
/dev/sda1                58.4G      1.1G     54.4G   2% /var/lib

On my system I have the default 64G setting. The Linux filesystem is using (64 - 58.4 = 5.6 G) as overhead and 1.1G is currently used for user data.

Regarding the 1.1G of user data, according to the docker engine I only have 100kB of images:

$ docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              1                   0                   100.8kB             100.8kB (100%)
Containers          0                   0                   0B                  0B
Local Volumes       0                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B

The command

$ docker run -it --privileged --pid=host justincormack/nsenter1 /usr/bin/du -h /var/lib

shows what else is written on the disk. In my case there's a bunch of general overhead files plus the default 1G swap file for Linux. This all adds up to the 1.1G (approximately, with some rounding)

However on the host I'm using about 2.2G:

$ ls -sk ~/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw
2303508 /Users/.../Library/Containers/com.docker.docker/Data/vms/0/Docker.raw

Subtracting the 1G of Linux swap + 0.1G of images (+ misc stuff like logs, overlay filesystems etc), this leaves about 1.1G. I believe this is probably part of the 5.6G of filesystem overhead that Linux has reserved internally.

Let me know where your space seems to be allocated.

it work perfectly. in my case , fstrim can reduce the Docker.raw from 31GB to 4.2GB which is 64GB totoal i was setted.

@adibaiceanu

This comment has been minimized.

Copy link

commented Jan 31, 2019

@fabiosussetto
### warning: you are going to lose all your containers and images if you do the following (no way around this, sigh!)

> Go to Preferences > Disk and reduce the disk image size allocation. I tried yesterday and it did reduce the .raw file size on my disk

..a build and start again will do the job!
Thanks again!

@davidkuridza

This comment has been minimized.

Copy link

commented Mar 8, 2019

Downsizing the disk did not help in my case, free space on the drive stayed the same. I went and uninstalled Docker (from preferences) and 150GB of space was released right away.

@blueabysm

This comment has been minimized.

Copy link

commented Apr 15, 2019

I don't understand how come this issue has been closed.
This is clearly a big problem, as Docker for Mac is eating up all disk space. The fact that the space may be just "virtually" used doesn't change a thing from the user's point of view.

In the FAQs (https://docs.docker.com/docker-for-mac/faqs/#disk-usage), it says:

"This is an illusion. [...] The output of ls is misleading, because it lists the logical size of the file rather than its physical size."

It may as well be an illusion, but I can't install software updates and MacOS is constantly complaining about running out of space.

Kindly reconsider opening this issue, as it's a very, very annoying thing that imo should not happen in stable software. I understand it may not be easy to fix, but this is Docker for Mac at the end of the day - saying that it's kind of MacOs fault is a bit of a stretch, at best.

Thanks.

Edit: ping @djs55
Would you please consider at least reopening this issue?
Regarding: "I think this is working as expected": people here have their macbooks unusable because of this disk space issue.

Not only from the user's point of view, but also the operating system's. My laptop complains for lacking disk space all the time.

@jjharsch

This comment has been minimized.

Copy link

commented Apr 15, 2019

@vesper8

This comment has been minimized.

Copy link

commented Apr 15, 2019

There is a lot of confusion in this thread. Docker.raw may say it is taking up 64GB, but it is not, actually. Not at all. And this includes what your system says and thinks.

At least I can only speak for OSX here. I just did the test.

I inspected this 64gb file and it said it only took ~7gb "on disk". This means that Finder, and the OSX system in general, only thinks of this file as taking 7gb. Certainly not 64gb. If you delete it, or if you have Docker resize it to say 32gb, what will happen is 7gb will be free'd up and all your containers destroyed and will have to be rebuilt.

Now in many cases Docker may be taking a lot more than 7gb as in my case.. and you might be making the point that it's taking too much, well that's probably because you have too many downloaded containers you don't need anymore. Docker has been doing self-intelligent trimming for a while now. Once your containers are powered down, they should be trimmed so they take a little space as possible "on disk". As for the 64GB omg.. just ignore it, it's not really taking that much space and your system doesn't think so either.. you're just low on space because that's the DEV life and we all are ;)

@shaneHowearth

This comment has been minimized.

Copy link

commented Apr 15, 2019

vesper8 This is not my experience. The Mac I was using at the time reported that all the disk had been consumed, despite every attempt I made to prune/delete/tidy up docker containers and the issue persisted across reboots.

The amount of disk reported to being consumed grew, despite no new containers being added and little to no changes to the containers that were supposed to be there.

The system was reporting that all the disk was being used by docker, despite docker's protestations claiming only a fraction of the disk was being used by it.

At the time the only way to 'fix' things was a complete uninstall/reinstall of docker and containers being worked on.

It seems to me your comment should really be "Unable to repeat on my machine" because the rest of your comment does not match with my experience at all.

@vesper8

This comment has been minimized.

Copy link

commented Apr 16, 2019

@shaneHowearth I think the problem is that there are many people in this thread discussing very different issues. I was only pointing out that the virtual space (the 64gb part) is not taking up space and only whatever it says "on disk usage" is what's used up. As @NoICE pointed out above, this is true of AFPS filesystems and may not be true if you're on any other filesystem.

In your case, from what I read above, your "on disk" usage was actually high, so it was not so much a case of Docker reserving space but a problem with the containers growing inside and possibly not being cleaned up properly after you brought the containers down. It's my understanding that this auto-cleanup is something that was added not that long ago.

@myleshk

This comment has been minimized.

Copy link

commented Apr 16, 2019

@shaneHowearth vesper8 is right. Mac reported disk space shortage simply because your disk is indeed fully consumed. Try docker image prune (or even with -a) and docker container prune, which worked for me.
And I agree that this is not a simple bug, but a feature request for auto-cleanup function.

@3141592

This comment has been minimized.

Copy link

commented Apr 22, 2019

Still seems like a problem. After all the docker "cleanup" commands I could find I am seeing:

$ du -h /Users//Library/Containers/com.docker.docker/Data/vms/0/Docker.raw
56G /Users//Library/Containers/com.docker.docker/Data/vms/0/Docker.raw

@myleshk

This comment has been minimized.

Copy link

commented Apr 23, 2019

@3141592 You are seeing the actual size.
Try more cleanup commands https://linuxize.com/post/how-to-remove-docker-images-containers-volumes-and-networks/.
If still the same, then you are really running a large number/size of docker instances. Check with docker ps and shutdown unused ones.

@jeremymouton

This comment has been minimized.

Copy link

commented Apr 28, 2019

Ran docker image prune and docker container prune, all Docker images have been cleared. Docker.raw is still 56GB.

@elizabethbeard

This comment has been minimized.

Copy link

commented Apr 29, 2019

Running into similar issues as @jeremymouton and everyone else here. When I tried to move the disk image to an external drive, Docker crashed. Will be cutting my disk images size to 16GB but doesn't seem like that really resolves the issue whatsoever.

-rw-r--r-- 1 lizbeard staff 60G Apr 29 09:46 Docker.raw

@metacollin

This comment has been minimized.

Copy link

commented May 5, 2019

Ran docker image prune and docker container prune, all Docker images have been cleared. Docker.raw is still 56GB.

How did you check the size of Docker.raw? If you used du or ls or looked at its size in a finder window, then you did not determine the size of Docker.raw correctly and the 56GB size is almost certainly inaccurate.

Still seems like a problem. After all the docker "cleanup" commands I could find I am seeing:

$ du -h /Users//Library/Containers/com.docker.docker/Data/vms/0/Docker.raw
56G /Users//Library/Containers/com.docker.docker/Data/vms/0/Docker.raw

du -h cannot be used to determine the true size of Docker.raw. It is not 56GB. Get info on the file and check the size in the red box:

docker size

Note, my Docker.raw is zero bytes if freshly created. It shows 64GB in the finder size, but this is not correct. If you delete it, it frees up no space on my drive. Just to make this explicitly clear, I duplicated my '64GB' Docker.raw file 30 times, causing it to 'consume' '2TB'. My SSD is 500GB.

blah

Clearly, the raw file is not consuming all of the space you think it is.

This is assuming you're using APFS of course. If not, ignore everything I just said.

If your Docker.raw file is 56GB, then it will say 56GB here

@rubenvarela

This comment has been minimized.

Copy link

commented May 6, 2019

As a counter point to @metacollin

Docker.raw is 64GB on disk and I am using APFS.

Screen Shot 2019-05-05 at 8 09 34 PM

Full diskutil list output,

/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.1 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk3         499.8 GB   disk0s2

/dev/disk1 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *256.1 GB   disk1
   1:                        EFI EFI                     209.7 MB   disk1s1
   2:                 Apple_APFS Container disk2         255.9 GB   disk1s2

/dev/disk2 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +255.9 GB   disk2
                                 Physical Store disk1s2
   1:                APFS Volume MacOS                   180.6 GB   disk2s1
   2:                APFS Volume Preboot                 18.9 MB    disk2s2
   3:                APFS Volume Recovery                519.5 MB   disk2s3
   4:                APFS Volume VM                      2.1 GB     disk2s4

/dev/disk3 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +499.8 GB   disk3
                                 Physical Store disk0s2
   1:                APFS Volume MacData                 439.9 GB   disk3s1

@djs55

This comment has been minimized.

Copy link
Contributor

commented May 6, 2019

@metacollin thanks for the post. I wish it were that easy to store more than 500GB on my SSD :)

For anyone who has a Docker.raw really taking up too much space on disk even after docker image prune and docker container prune, see if you can determine where the space is allocated in the VM:

  • Open a shell with a command like docker run --rm -it --privileged --pid=host walkerlee/nsenter -t 1 -m -u -i -n /bin/sh
  • Explore the filesystem, in particular /var/lib where the Docker.raw is mounted. For example:
/ # df -h /var/lib
Filesystem                Size      Used Available Use% Mounted on
/dev/sda1                58.4G      5.8G     49.7G  10% /var/lib

It's possible that /var/lib has become genuinely full of something unexpected, such as excessive log messages due to a bug. It would help to know if the filesystem is full from Linux's point of view, and if so which directory or directories are responsible.

@leviticusg2015

This comment has been minimized.

Copy link

commented May 7, 2019

@metacollin Thanks for the reply. Initially the Mac Finder info did verify it was 56G. I unistalled Docker Desktop, deleted the Docker.raw and reinstalled Docker Desktop, and all seems OK now. While Finder says 64 GB, the File Info says 46 GB and my Mac is no longer acting like it's out of space:
Size: 63,999,836,160 bytes (46.3 GB on disk)

@rubenvarela

This comment has been minimized.

Copy link

commented May 12, 2019

Trying @djs55's idea,

After downloading and logging into the container, the output of du

/ # du -hd 1 .
3.1M	./bin
6.3M	./boot
2.5G	./containers
0	./dev
4.5K	./EFI
829.5K	./etc
2.0K	./home
56.9M	./lib
8.0K	./media
2.0K	./mnt
6.0K	./opt
0	./proc
2.0K	./root
672.0K	./run
1.3M	./sbin
2.0K	./srv
0	./sys
0	./tmp
119.4M	./usr
2.4G	./var
5.1G	.

and the output of df,

/ # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               477.4M    477.4M         0 100% /
tmpfs                   199.9M    672.0K    199.3M   0% /run
tmpfs                   199.9M         0    199.9M   0% /tmp
tmpfs                   999.5M         0    999.5M   0% /var
dev                      10.0M         0     10.0M   0% /dev
shm                     999.5M         0    999.5M   0% /dev/shm
cgroup_root              10.0M         0     10.0M   0% /sys/fs/cgroup
tmpfs                   999.5M         0    999.5M   0% /containers/onboot/001-sysfs/tmp
tmpfs                   999.5M         0    999.5M   0% /containers/onboot/003-sysctl/tmp
tmpfs                   999.5M         0    999.5M   0% /containers/onboot/004-format/tmp
tmpfs                   999.5M         0    999.5M   0% /containers/onboot/005-extend/tmp
tmpfs                   999.5M         0    999.5M   0% /containers/onboot/006-mount/tmp
/dev/sda1                58.4G      1.8G     53.6G   3% /var/lib
tmpfs                   999.5M         0    999.5M   0% /containers/onboot/007-swap/tmp
tmpfs                   999.5M         0    999.5M   0% /containers/onboot/008-move-logs/tmp
tmpfs                   999.5M         0    999.5M   0% /containers/onboot/009-mount-docker/tmp
/dev/sr2                961.1M    961.1M         0 100% /containers/services/docker
tmpfs                   999.5M         0    999.5M   0% /containers/onboot/010-bridge/tmp
tmpfs                   999.5M         0    999.5M   0% /containers/onboot/011-vpnkit-9pmount-vsock/tmp
tmpfs                   999.5M      4.0K    999.5M   0% /containers/onboot/012-ip/tmp
tmpfs                   999.5M      4.0K    999.5M   0% /containers/services/acpid/tmp
overlay                 999.5M      4.0K    999.5M   0% /containers/services/acpid/rootfs
tmpfs                   999.5M         0    999.5M   0% /containers/services/diagnose/tmp
overlay                 999.5M         0    999.5M   0% /containers/services/diagnose/rootfs
tmpfs                   999.5M    244.0K    999.3M   0% /containers/services/docker/tmp
overlay                 999.5M    244.0K    999.3M   0% /containers/services/docker/rootfs
tmpfs                   199.9M    672.0K    199.3M   0% /run/desktop/linuxkit-external-logging.sock
tmpfs                   999.5M         0    999.5M   0% /containers/services/host-timesync-daemon/tmp
overlay                 999.5M         0    999.5M   0% /containers/services/host-timesync-daemon/rootfs
tmpfs                   999.5M         0    999.5M   0% /containers/services/kmsg/tmp
overlay                 999.5M         0    999.5M   0% /containers/services/kmsg/rootfs
tmpfs                   999.5M         0    999.5M   0% /containers/services/sntpc/tmp
overlay                 999.5M         0    999.5M   0% /containers/services/sntpc/rootfs
tmpfs                   999.5M         0    999.5M   0% /containers/services/socks/tmp
overlay                 999.5M         0    999.5M   0% /containers/services/socks/rootfs
tmpfs                   999.5M         0    999.5M   0% /containers/services/trim-after-delete/tmp
overlay                 999.5M         0    999.5M   0% /containers/services/trim-after-delete/rootfs
tmpfs                   999.5M         0    999.5M   0% /containers/services/vpnkit-forwarder/tmp
overlay                 999.5M         0    999.5M   0% /containers/services/vpnkit-forwarder/rootfs
tmpfs                   999.5M         0    999.5M   0% /containers/services/vsudd/tmp
overlay                 999.5M         0    999.5M   0% /containers/services/vsudd/rootfs
tmpfs                   999.5M         0    999.5M   0% /containers/services/write-and-rotate-logs/tmp
overlay                 999.5M         0    999.5M   0% /containers/services/write-and-rotate-logs/rootfs
overlay                  58.4G      1.8G     53.6G   3% /var/lib/docker/overlay2/ba8b2289a74df14ee776c9df61087719c330784a8a8ea59953c7ec60c0aef93f/merged
overlay                  58.4G      1.8G     53.6G   3% /var/lib/docker/overlay2/9fb9845412112d52435d7c02606d26cab84122d69023174eb91701ddb193d30f/merged
shm                      64.0M         0     64.0M   0% /var/lib/docker/containers/c3cfc52bb7712817c6559218e4c4c580ea39c147959ca3f0e6737efb8441020e/mounts/shm
shm                      64.0M         0     64.0M   0% /var/lib/docker/containers/e3b63dac01a48901559215789bca07fd8385c31d7a26fd271511e5c78a304a19/mounts/shm
overlay                  58.4G      1.8G     53.6G   3% /var/lib/docker/overlay2/e8588458d2e6f192f40f7717b481630f05e13f746fa398fff35123472b0addfc/merged
shm                      64.0M         0     64.0M   0% /var/lib/docker/containers/e0398b742aa0399fd8d128ba7a762f2e371786e0dc60bde6d310ec1c035a847a/mounts/shm
/ #

and then /var/lib

/ # df -h /var/lib
Filesystem                Size      Used Available Use% Mounted on
/dev/sda1                58.4G      1.7G     53.7G   3% /var/lib
@adipascu

This comment has been minimized.

Copy link

commented Aug 3, 2019

@jjharsch You can set the max disk usage under 16GB by editing the diskSizeMiB field from ~/Library/Group\ Containers/group.com.docker/settings.json.

image

Watch out, this breaks the UI, the slider will no longer work.

@jjharsch

This comment has been minimized.

Copy link

commented Aug 3, 2019

@Lzolcsi

This comment has been minimized.

Copy link

commented Aug 13, 2019

@jjharsch You can set the max disk usage under 16GB by editing the diskSizeMiB field from ~/Library/Group\ Containers/group.com.docker/settings.json.

image

Watch out, this breaks the UI, the slider will no longer work.

wow, that really helped me a lot. Finally I can move it to a 16GB SD card... :)

@onacit

This comment has been minimized.

Copy link

commented Aug 14, 2019

The illusion keeps making side effects.
The OS itself keep complaining about running out of storage and some applications, like Outlook, go to offline.

@stepmuel

This comment has been minimized.

Copy link

commented Aug 14, 2019

Native sparse files are an exotic new feature on macOS. Using them to that extent is confusing and makes system administration harder. Since Docker used to work before APFS, there must be a way to do it the old way. If there are good reasons that justify all the hassle, please let us know, so we can at least have an informed discussion!

As far as I see it, APFS sparse files are mostly intended to speed up creation of large files. Once a section is written to, its space will never be freed again. It is not some sort of "zero compression" like the documentation has us think. Also, it makes sense for software to assume a file with a size of 64 GB actually intends to use that space at some point. Your design implies otherwise, which is controversial for those who understand, and confusing for those who don't.

Docker is used by thousands of people who just want their Mac to work. Please don't make all of us have to deal with that weirdness. Just because a file system feature exists doesn't mean you should use it.

@djs55

This comment has been minimized.

Copy link
Contributor

commented Aug 19, 2019

Once a section is written to, its space will never be freed again.

In fact when images are deleted inside the Linux VM we trigger an fstrim which calls fcntl(F_PUNCHHOLE) on macOS which causes the blocks to be freed again on the host. The actual space usage of the file should therefore go up and down, just as it used to with the Docker.qcow2. (If this isn't happening in practice then there must be a bug)

The main advantages of the sparse file are:

  1. container I/O is much faster
  2. reclaiming space is instant and reliable (previously we would mark the Docker.qcow2 blocks as free and run a concurrent GC to move blocks and shrink the file. If the VM is keeps allocating then the collector will take a long time to shrink the file. The file would often get very big, exceeding the maximum configured limits)

The Docker.qcow2 code still exists and is used when the disk image is written to a non-APFS filesystem. So one way to force the use of qcow2 is to write the disk to an external drive formatted with some other filesystem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.