[wsl2] filesystem performance is much slower than wsl1 in /mnt

[ioweb-gr]

I decided to open this as a separate issue because although it's related to the generic issue of filesystem performance it's directly related to WSL 2 while the other issues are for WSL 1 and it's showing very conflicting results.

• Your Windows build number: (Type ver at a Windows Command Prompt)
  Version 10.0.18917 Build 18917

• What you're doing and what's happening:
  I'm testing filesystem write speed in /mnt using dd command. Performing the following tests

WSL2

root@LUCIANO-PC:/home/# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 25.939 s, 40.4 MB/s

WSL1

root@LUCIANO-PC:/home/# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 47.4897 s, 442 MB/s

On / it's actually the reverse. WSL2 is more than 2 times faster than WSL1.

• What's wrong / what should be happening instead:
  I would expect the filesystem performance in /mnt to at least be on the same level but it's over 10 times slower.

Another interesting fact is that if I mount the same drive as a cifs share I get 3x performance

WSL 2 (cifs share)

root@LUCIANO-PC:/mnt/sambae# dd if=/dev/zero of=/mnt/sambae/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 84.001 s, 125 MB/s

An update of the current status because it's way too hidden in this thread.

Latest status report: #4197 (comment)_

[billziss-gh]

On / it's actually the reverse. WSL2 is more than 2 times faster than WSL1.

Can you post the results when doing the same test on a file on / for WSL1 and WSL2?

This is straight I/O on a single file where I would expect to see more or less the same perf between WSL1 and WSL2. My expectation was that WSL2 perf is better than WSL1 only when doing "namespace" operations (i.e. working on lots of small files, listing them, stat'ing them, etc.)

As for the results on /mnt/e they are not very surprising, since I/O has to go through both the Linux and Windows file system stack in likely a new and unoptimized piece of code.

[ioweb-gr]

I see,

I'm assuming you're still refering to / and not /mnt folders about namespace operations.
I've repeated the write tests in both environments a few times now and there's no consistent result.

WSL 2

root@LUCIANO-PC:/home/# dd if=/dev/zero of=/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 7.0873 s, 1.5 GB/s
root@LUCIANO-PC:/home/# dd if=/dev/zero of=/testfile bs=1M count=20000

20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 51.3779 s, 408 MB/s
root@LUCIANO-PC:/home/#
root@LUCIANO-PC:/home/# dd if=/dev/zero of=~/testfile bs=1M count=20000
^C12360+0 records in
12360+0 records out
12960399360 bytes (13 GB, 12 GiB) copied, 12.1766 s, 1.1 GB/s

WSL1

root@LUCIANO-PC:/home/# dd if=/dev/zero of=~/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 29.7096 s, 353 MB/s
root@LUCIANO-PC:/home/#

I'm thinking this has to do something with OS Caching because in task manager even though I can see the file being written, the disk usage write speed is unchanged while in wsl1 I can see it constantly at around 350mb/sec .

As another interesting fact, the vmmem usage while writing the file is increasing rapidly. By the 3rd copy action all my RAM is used up (32GB).
It goes back down a while after quitting my shell. So maybe the performance is bottlenecked by this bug as well.

[paultyng]

I'm also experiencing this after an upgrade to WSL2, only slow performance under /mnt, not /, slower than WSL1. I can post numbers as well if necessary. This is on Microsoft Windows [Version 10.0.18936.1000].

[definelicht]

I'm experiencing the same issue, /mnt is extremely slow compared to WSL1 (seconds to run git status on a relatively small repository).

[paultyng]

When I run a git status on a clone of https://github.com/hashicorp/terraform in /mnt I get output like this:

It took 6.97 seconds to enumerate untracked files. 'status -uno'
may speed it up, but you have to be careful not to forget to add
new files yourself (see 'git help status').

[ioweb-gr]

This is an example from the command

pv files.tar.gz | tar -zxf -

The speed is amazingly slow in wsl2 on /mnt

[definelicht]

Do we know what's at the root of this, and whether there are any other workarounds than downgrading?

[benhillis]

@definelicht - Yep we're working on improving the performance. In the meaning working out of your root file system (the ext4 volume) will have MUCH better performance.

[ioweb-gr]

Indeed after the last update I can already see much better performance on WSL2.

root@LUCIANO-PC:~# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 41.277 s, 254 MB/s

That's on an SSD and it's like 6 times faster than before. Still slower then WSL1 but it's definitely improved.

--correction---
I see much better performance on sequential writes. Untar seems very slow still extracting data at bytes/sec while in wsl1 I can see speeds at kb/sec for the same part

[tuananh]

Indeed after the last update I can already see much better performance on WSL2.

root@LUCIANO-PC:~# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 41.277 s, 254 MB/s

That's on an SSD and it's like 6 times faster than before. Still slower then WSL1 but it's definitely improved.

--correction---
I see much better performance on sequential writes. Untar seems very slow still extracting data at bytes/sec while in wsl1 I can see speeds at kb/sec for the same part

I got

1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.57894 s, 293 MB/s

but git status still taking more than 2 seconds on quite small repo

[definelicht]

I think this is a latency rather than a bandwidth issue, surfacing when a large number of files are accessed (e.g., when running git status).

[tuananh]

@definelicht - Yep we're working on improving the performance. In the meaning working out of your root file system (the ext4 volume) will have MUCH better performance.

can you share the cause of it? I'm curious

[gbatalski]

can confirm a very bad performance on /mnt (mvn install takes forever), on /tmp it takes only 2 minutes for the same project

[nono-pxbsd]

After switch configuration from WSL to WSL 2 (Build 18975), without any change on my distro, the server respond time incresed.

Configuration : Debian buster Apache2.X / PHP-FPM 7.3 / MySQL
Root : /
Web directories : /c/repositories/xxx

WSL 1 : 0.5 sec
WSL 2 : 8 sec

With Docker (Ubuntu 18.04) WSL Tech preview i could not test on WSL 1, but WSL 2 takes over 30 sec... I suppose the issue is the same.

[realtica]

This is my performance in a medium project:

GIT_TRACE_PERFORMANCE=1 git status -sb -uno

13:20:04.630972 read-cache.c:2267 performance: 0.017249300 s: read cache .git/index
13:20:07.766270 preload-index.c:146 performance: 3.135180100 s: preload index
Refresh index: 100% (3463/3463), done.
13:20:53.531359 read-cache.c:1575 performance: 48.900269600 s: refresh index
13:20:53.981829 read-cache.c:2267 performance: 0.016242500 s: read cache /mnt/e/p/keyed_au/.git/ modules/Files/media/nusoap/index
13:20:54.383135 read-cache.c:1575 performance: 0.401222700 s: refresh index
13:20:54.426399 diff-lib.c:251 performance: 0.000005000 s: diff-files
13:20:54.468000 unpack-trees.c:1554 performance: 0.000087700 s: traverse_trees
13:20:54.468266 unpack-trees.c:465 performance: 0.000164900 s: check_updates
13:20:54.468346 unpack-trees.c:1649 performance: 0.000438000 s: unpack_trees
13:20:54.468368 diff-lib.c:537 performance: 0.001873400 s: diff-index
13:20:54.569793 read-cache.c:2989 performance: 0.014679900 s: write index, changed mask = 2
13:20:54.586550 trace.c:477 performance: 0.736638600 s: git command: /usr/lib/git-core/git status --porcelai n=2 -uno
13:20:55.277184 read-cache.c:2267 performance: 0.031572000 s: read cache /mnt/e/p/keyed_au/.git/ modules/Files/modules/unittest/index
13:20:55.958706 read-cache.c:1575 performance: 0.681302500 s: refresh index
13:20:56.007094 diff-lib.c:251 performance: 0.000003500 s: diff-files
13:20:56.041768 unpack-trees.c:1554 performance: 0.000081200 s: traverse_trees
13:20:56.042014 unpack-trees.c:465 performance: 0.000162100 s: check_updates
13:20:56.042084 unpack-trees.c:1649 performance: 0.000399800 s: unpack_trees 13:20:56.042102 diff-lib.c:537 performance: 0.000454400 s: diff-index 13:20:56.126615 read-cache.c:2989 performance: 0.012095500 s: write index, changed mask = 2 13:20:56.141174 trace.c:477 performance: 1.261971900 s: git command: /usr/lib/git-core/git status --porcelain=2 -uno 13:20:56.148677 diff-lib.c:251 performance: 2.586112000 s: diff-files 13:20:56.190888 unpack-trees.c:1554 performance: 0.000022200 s: traverse_trees 13:20:56.191028 unpack-trees.c:465 performance: 0.000027800 s: check_updates 13:20:56.191106 unpack-trees.c:1649 performance: 0.000639200 s: unpack_trees 13:20:56.191175 diff-lib.c:537 performance: 0.001413100 s: diff-index
13:20:56.215068 trace.c:477 performance: 51.628126000 s: git command: git status -sb -uno

File System exFat, the result is the same is other file system...
WSL 2 git=51.62 sec
git for windows=0.8 sec

[paslandau]

I'm currently using the normal Docker Desktop with linux containers to run a PHP projekt during local development. The repo is in a normal Windows directory and gets mounted via volume bind in the docker container, so that I can easily manipulate the files through an IDE.

The same setup on a native linux machine has always been multiple times faster and the "problem" (from what I've read) has always been attributed to the Linux VM that is used on Docker Desktop.

Then WSL 2 was announced and I finally came around to set everything up. But when I ran our teststuite for the first time, I was pretty disappointed that the performance was actually much worse than before (40s on the Docker Deskop Linux VM setup vs 100s on the WSL2 setup). Subsequent investigation brought me to this thread.

From what I've read here so far:

Does it even make sense to switch to WSL 2 for local development yet if "bad performance" is your primary reason to do so?

A "no" would be a totally acceptable answer - I just don't want to waste any more time on this if it's simply not ready yet.

[stweedie]

I know that wsl2 is a preview, but is there any hope that accessing \wsl$ from windows or /mnt from wsl will reach acceptable speeds?

[gormulent]

I'd also like to ask about this, doing a git status on something in /mnt/c/.. takes about a minute, something seems wrong ..

[gbatalski]

same with 1903 (18990.1) absolutely useless...

[wusticality]

I am also seeing this, accessing Windows files from the Linux environment is several orders of magnitude slower in WSL2 than in WSL1.

[danrossi]

I can get another M.2 just for WSL. But I need to edit files and manage repositories via windows. So need access to the files. I use VS Code but not sure what that extension does.

[xzn]

@danrossi Is using WSL1 an option? If you need to work with files from Windows but need to use WSL to do it, the easiest way it seems is to use WSL1 for now. I pretty much keep both WSL1 and WSL2 versions installed for the primary WSL distro that I'm using because of this.

[AmineDjeghri]

@danrossi easiest way I've found is to not share data between WSL and windows. Run everything inside the linux VM. :( Or install a linux desktop directly. Not ideal for everyone, but at least it's fast. Jetbrains Gateway and VS Code WSL extensions make this mostly seamless. Convert your SSD to ext4 and mount it inside WSL. Avoid working with NTFS.

Exactly, I'm cloning now my git projects directly inside WSL instead of another location that needs to go through /mnt.

[AmineDjeghri]

I can get another M.2 just for WSL. But I need to edit files and manage repositories via windows. So need access to the files. I use VS Code but not sure what that extension does.

If you are moving your WSL2 to another drive, I suggest you to do an export of your actual WSL image. It takes seconds to minutes to export and import .

You can still edit files and manage repositories via windows even if they are stored inside WSL with explorer manager and sublime text or any tool you want

[danrossi]

I don't understand how to mount the files in VS Code but going to make a clean install move files over and give it a try not a problem. I've backed up my old instance.

[andrewleech]

I don't understand how to mount the files in VS Code

Using the official WSL support in vscode is the best way to manage this, especially if you're already using vscode. There's heaps of guides online, try this official one https://code.visualstudio.com/docs/remote/wsl-tutorial

[JoonSumisu]

Hello guys, so 5 years passed, the best way still get a new SSD and put everything in it? and this problem could be solved in future one day?

[GalileoFe]

Hello guys, so 5 years passed, the best way still get a new SSD and put everything in it? and this problem could be solved in future one day?

I find that the filesystem performance has not improved, and it is the same as it was 5 years ago. Last week, I used WSL2 to transfer 500GB of files, but the speed was only 300MB/s.

[JoonSumisu]

Hello guys, so 5 years passed, the best way still get a new SSD and put everything in it? and this problem could be solved in future one day?

I find that the filesystem performance has not improved, and it is the same as it was 5 years ago. Last week, I used WSL2 to transfer 500GB of files, but the speed was only 300MB/s.

same.... and I decide to get a new SSD only for my wsl, wish it can solve all the problem

[kirkbushell]

The problem is not, and has never been a HDD performance issue - it's how they changed how filesystems are accessed.

[trentbuck]

I saw a couple of comments suggesting WSL2 is using 9P still.
If that's really the case, has anyone considered virtiofsd (on the ntoskrnl host) and fuse.ko (on the linux guest)?
I've booted Debian 12 guests off virtiofsd. It was not fast, but it was still faster than qemu's internal 9P server.
(And obviously it let me entirely avoid making disk images and filesystems, and instead JUST have directories on the host.)
I looked for a separate ticket, but I couldn't find one. Since the whole point of virtiofs is "like 9p, but faster", I'll mention it here.

https://virtio-fs.gitlab.io/
https://gitlab.com/virtio-fs/virtiofsd/
https://docs.kernel.org/filesystems/virtiofs.html

PS: my main use case for wsl is things like du -haxt1G /mnt/C/*/steam | sort -hr, because I'm too lazy to learn powershell equivalents. So I'm quite happy to stick to WSL1 for now. I thought having ntoskrnl implement a linux personality was an elegant idea, and I like not having any linux or systemd in my Debian GNU/NTOSkrnl containers, and using GUI apps via GTK_BACKEND=broadwayd emacs or wireshark -platform webgl. :-)

[andrewleech]

has anyone considered virtiofsd (on the ntoskrnl host) and fuse.ko (on the linux guest)?

From a quick search I can only see reference to virtiofsd supporting windows as guests on linux hosts?
Do you know of any existing windows host drivers for it?

edit: This probably should already be possible to use to get access to linux files from windows (eg. mount wsl home folder as a windows drive) though not the other way around (accessing windows drive files from within linux)

[trentbuck]

Do you know of any existing windows host drivers for it?

As I understand it, virtiofsd is a rust userspace program that does ordinary file access (open/read/write/close &c), and talks to the guest via an AF_UNIX socket. If Windows can compile that and run it, I think it would work for /mnt/C. wsl.exe would need a small amount of plumbing to auto-start and auto-stop virtiofsd with the container, and to glue together the sockets/file-descriptors. For linux-on-linux, that is done by libvirtd (or just by hand).

For actually booting the guest off it (i.e. rootfstype=virtiofs root=my-cool-placeholder-name), for Linux-on-Linux, virtiofsd has to run as root (so it has CAP_DAC_OVERRIDE and CAP_FOWNER), because the guest is completely stock userspace and expects to have >1 user inside it, and virtiofsd (by default) has no uid mapping stuff. i.e. similar to how booting off nfs4.ko with sec=sys works. That MIGHT be solvable, but I think for the immediate issue of "/mnt/C is slow" isn't even in-scope?

This is speculation on my part - I have only ever used virtiosfd for linux-on-linux; I didn't even compile it myself, I just used what Debian shipped (which might even be the original pre-rust C version).

So I guess the immediate step would be to try to compile and run virtiofsd on Windows as a regular (unprivileged) user -- that is definitely outside my personal skillset :-)

[chocolata]

Hi, I'm also experiencing dramatic slowdowns when files are stored outside the Linux filesystem. Could anyone confirm that WSL 2.0.0 that has just been released solves this issue or is this unrelated?

https://devblogs.microsoft.com/commandline/windows-subsystem-for-linux-september-2023-update/

[myalcin81]

Microsoft Windows [Version 10.0.22621.2283] : wsl 2. I still experience same issue.

[zlocorp]

Hi, I'm also experiencing dramatic slowdowns when files are stored outside the Linux filesystem. Could anyone confirm that WSL 2.0.0 that has just been released solves this issue or is this unrelated?

https://devblogs.microsoft.com/commandline/windows-subsystem-for-linux-september-2023-update/

It's still very slow on WSL2 - 2.0.4

[zlocorp]

I continue to observe. On new WSL2 build nothing change :(

Windows 11 23H2
WSL2 - 2.0.7 (autoMemoryReclaim - on)

[sdhou]

Is this issue still planned to be fixed?

[TTimo]

Do you know of any existing windows host drivers for it?

As I understand it, virtiofsd is a rust userspace program that does ordinary file access (open/read/write/close &c), and talks to the guest via an AF_UNIX socket.

I don't know about virtiofsd but doing a simple network share mount (CIFS/SMB3) may be an option with reasonable performance and fewer hard technical problems to resolve.

[mike-2020]

WSL 2, version 2.0.9.0
dd file read from files under /mnt/d/, speed is around 30MB/s. The disk is a NVME SSD.
Looks like this issue has not been fixed after 4 years.

[ddjangl]

I want to use WSL 2 for its USB features such that I can flash embedded devices but this slow git performance is completely and entirely holding me back from moving up to WSL 2. Has anyone come up with viable workarounds? Is USB possible in WSL 1? Sheesh...

[Illutax]

Doesn't Microsoft understand that modern software development is heavily based on the ability to build/run containerised applications? They will loose the last couple windows devs with this shit show...

[odie5533]

Would alternatively be useful to have a way to create a Linux volume that lives outside of a single WSL2 instance. That way we could mount the volume to /home and it would be fast and external.