WSL2 distro rapidly expands to fill drive without using space and cannot be shrunk

[sjshamsi]

Version

Microsoft Windows [Version 10.0.22000.593]

WSL Version

• WSL 2
• WSL 1

Kernel Version

5.10.16

Distro Version

Ubuntu 20.04

Other Software

No response

Repro Steps

I am running a Python script calculating some values which are then put in a dictionary and pickled on the disk.

python script_saving_dictionaries
.
.
.
input/output error

Expected Behavior

Expected disk usage by WSL distro to increase by the amount occupied by my saved files (7MB at point of failure)

Actual Behavior

I got to saving ~400 dictionaries with a total size of 7MB, which was what I expected.

However soon I encounter an input/output error due to low disk space. The Ubuntu ext4.vhdx seems to occupy 103GB now from the original ~40GB. This occupies all my disk space and my task fails.

I tried my best to confirm if I was saving some junk somewhere prompting this behaviour but cannot find it. Using du -hs --exclude mnt/ on path ~/../../ (in an effort to see all disk space usage by Ubuntu) returns a size of ~39GB, and I am unable to find what's occupying the rest of the space.

I uninstalled and reinstalled Ubuntu to the same result.

Using optimize-vhd -Path ...\ext4.vhdx -Mode full or using diskpart does not reclaim the space.

Diagnostic Logs

No response

[sjshamsi]

In case it helps, here are disk space usage from Windows and from ncdu. Both indicate total disk space usage way smaller than the ext4.vhdx occupied ~100GB (besides the ~/../../proc directory being 127TB on Windows I'm not sure what happened there).

[OneBlue]

Hi @sjshamsi. Do you have free space on the linux filesystem ?

What's the output of df -h inside wsl ?

[sjshamsi]

Hi @OneBlue, thansk for the comment. So in this meantime I reimaged my machine but it does exhibit the same problem. Currently Windows displays 33.1GB free of 186GB C drive, and ext4.vhdx occupies 118GB.

On WSL, df -h displays:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        251G   36G  203G  16% /
none            1.9G   32K  1.9G   1% /mnt/wslg
none            1.9G  4.0K  1.9G   1% /mnt/wsl
tools           187G  154G   34G  83% /init
none            1.9G     0  1.9G   0% /dev
none            1.9G     0  1.9G   0% /run
none            1.9G     0  1.9G   0% /run/lock
none            1.9G     0  1.9G   0% /run/shm
none            1.9G     0  1.9G   0% /run/user
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
drivers         187G  154G   34G  83% /usr/lib/wsl/drivers
lib             187G  154G   34G  83% /usr/lib/wsl/lib
none            1.9G   76K  1.9G   1% /mnt/wslg/versions.txt
none            1.9G   76K  1.9G   1% /mnt/wslg/doc
drvfs           187G  154G   34G  83% /mnt/c
drvfs            50G   13G   38G  25% /mnt/d

I am quite new to Linux filesystems but I was concerned with /dev/sdc displaying 16% use. Please let me know if I can provde more details.

[OneBlue]

Interesting. Does running fstrim -a in wsl, then wsl wsl --shutdown and Optimize-Vhd again helps ? I wonder if filesystem is fragmented

[sjshamsi]

Oh wow that totally worked! Running fstrim -a before running 'optimize-vhd' shrinks the 'ext4.vhdx' drive to ~38GB which is about the size I expect. df -h now returns:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        251G   36G  203G  16% /
none            1.9G   32K  1.9G   1% /mnt/wslg
none            1.9G  4.0K  1.9G   1% /mnt/wsl
tools           187G   74G  114G  40% /init
none            1.9G     0  1.9G   0% /dev
none            1.9G     0  1.9G   0% /run
none            1.9G     0  1.9G   0% /run/lock
none            1.9G     0  1.9G   0% /run/shm
none            1.9G     0  1.9G   0% /run/user
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
drivers         187G   74G  114G  40% /usr/lib/wsl/drivers
lib             187G   74G  114G  40% /usr/lib/wsl/lib
none            1.9G   64K  1.9G   1% /mnt/wslg/versions.txt
none            1.9G   64K  1.9G   1% /mnt/wslg/doc
drvfs           187G   74G  114G  40% /mnt/c
drvfs            50G   13G   38G  25% /mnt/d

I wonder if there anything in particular that might have caused this. At any rate, thanks so much for your help!

[OneBlue]

Happy to hear that, closing.

[kloh-fr]

@OneBlue Thank you so much for the fstrim -a trick! ❤️

I was in the exact same situation than @sjshamsi and I got back 130+GB of free space thanks to you. 🕺👯💃

[okibcn]

Using optimize-vhd -Path ...\ext4.vhdx -Mode full or using diskpart does not reclaim the space.

Those methods are not very effective as both are designed to optimize NTFS partitions and their performance in ext4 filesystems is far from good. In some cases, they can even increase the size.

You can try wslcompact utility. It leaves images about 2-3% larger than the used space reported by df /.

[winston01]

In case you're using Win 10 Home, you may not have Opimize-Vhd available. Alternative way of compacting the virtual drive is described here:

https://answers.microsoft.com/en-us/windows/forum/all/optimize-vhd-not-found-in-windows-10-home/a727b760-0f82-4d0f-8480-d49eeaeb11a2

[ryanjlind]

Unfortunately I am having this issue and the fstrim command followed by diskpart compact does not work for me.

If I run df -h in Ubuntu, it shows

/dev/sdc 336G 321G 0 100% /

First of all, I don't really understand how 321/336 = 100%, but that's not really important. What's important is that I literally have almost nothing in here and it makes no sense that it's taking up 300+ GB of space. The contents of my /home directory (where I save everything) is only 17GB. I just cannot figure out what is going on here.

I don't have optimize-vhd available, but I ran fstrim -a and then followed the instructions posted by @winston01 and it made no difference. I am completely out of space in Ubuntu and not even able to save files or do any work, and I've been deleting stuff left and right to make space that haven't made any difference.

[ethanw-tm]

@ryanjlind I had this same issue in WSL2 Ubuntu, and all defragmentation, optimize-vhd, and fstrim -a attempts did nothing. My /dev/sdc was full to 100%, and even though I deleted gigabytes from my home directory, it would always fill up soon after, even though I had only ~40 GB in my home dir. Unless I manually deleted files from the file explorer in \wsl$, I couldn't even open a terminal properly due to the disk being full.

For me, it ended up being a docker issue; I finally narrowed my du -sh calls down to /var/lib/docker/overlay2 (had to sudo su to enter the docker dir), which was the entirety of the problem at almost 200G. Running docker system prune freed about 30G, but I had to run docker system prune -a in order to free the rest.

Hope this helps. Cheers!

[ryanjlind]

@ryanjlind I had this same issue in WSL2 Ubuntu, and all defragmentation, optimize-vhd, and fstrim -a attempts did nothing. My /dev/sdc was full to 100%, and even though I deleted gigabytes from my home directory, it would always fill up soon after, even though I had only ~40 GB in my home dir. Unless I manually deleted files from the file explorer in \wsl$, I couldn't even open a terminal properly due to the disk being full.

For me, it ended up being a docker issue; I finally narrowed my du -sh calls down to /var/lib/docker/overlay2 (had to sudo su to enter the docker dir), which was the entirety of the problem at almost 200G. Running docker system prune freed about 30G, but I had to run docker system prune -a in order to free the rest.

Hope this helps. Cheers!

Holy cow dude. That did it. That was the problem. Docker. Thank you soooo much!

It took like 15 minutes to clear all the junk out of Docker, and freed up 200GB. Man!

Thanks again. Really appreciate it.

[marinac-dev]

I had the same issue /usr/lib/wsl/drivers was huge like 216GB and fstrim -a with Optimize-VHD worked like a charm!

PS: Commenting, so others can find it easier with search👍🏻

[micha-1987]

Hello I am having the same issue.
I installed WSL 2 without any problems on my Win machine (Win 10 Enterprise 21H2 Build 19044.3930).
But after installing Docker I can see that the Ubuntu distro starts using more and more disk space.
After one hour it increased by more than 10GB and it doesnt stop which you can see in the following pictures.


It is really eating up my disk space without any obvious reason.
It is a fresh Docker installation without any containers running.

Unfortunately Optimize-VHD is not found and docker system prune -a comes up with 0B released since nothing is running.
I am having other Docker Deamons running on different Win machines without having this problem.
Do you have any other ideas how I can stop this?