Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wsl2] filesystem performance is much slower than wsl1 in /mnt #4197

Open
ioweb-gr opened this issue Jun 19, 2019 · 76 comments
Open

[wsl2] filesystem performance is much slower than wsl1 in /mnt #4197

ioweb-gr opened this issue Jun 19, 2019 · 76 comments
Labels

Comments

@ioweb-gr
Copy link

@ioweb-gr ioweb-gr commented Jun 19, 2019

I decided to open this as a separate issue because although it's related to the generic issue of filesystem performance it's directly related to WSL 2 while the other issues are for WSL 1 and it's showing very conflicting results.

  • Your Windows build number: (Type ver at a Windows Command Prompt)
    Version 10.0.18917 Build 18917

  • What you're doing and what's happening:
    I'm testing filesystem write speed in /mnt using dd command. Performing the following tests

WSL2

root@LUCIANO-PC:/home/# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 25.939 s, 40.4 MB/s

WSL1

root@LUCIANO-PC:/home/# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 47.4897 s, 442 MB/s

On / it's actually the reverse. WSL2 is more than 2 times faster than WSL1.

  • What's wrong / what should be happening instead:
    I would expect the filesystem performance in /mnt to at least be on the same level but it's over 10 times slower.

Another interesting fact is that if I mount the same drive as a cifs share I get 3x performance

WSL 2 (cifs share)

root@LUCIANO-PC:/mnt/sambae# dd if=/dev/zero of=/mnt/sambae/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 84.001 s, 125 MB/s

@billziss-gh

This comment has been minimized.

Copy link

@billziss-gh billziss-gh commented Jun 20, 2019

On / it's actually the reverse. WSL2 is more than 2 times faster than WSL1.

Can you post the results when doing the same test on a file on / for WSL1 and WSL2?

This is straight I/O on a single file where I would expect to see more or less the same perf between WSL1 and WSL2. My expectation was that WSL2 perf is better than WSL1 only when doing "namespace" operations (i.e. working on lots of small files, listing them, stat'ing them, etc.)

As for the results on /mnt/e they are not very surprising, since I/O has to go through both the Linux and Windows file system stack in likely a new and unoptimized piece of code.

@ioweb-gr

This comment has been minimized.

Copy link
Author

@ioweb-gr ioweb-gr commented Jun 20, 2019

I see,

I'm assuming you're still refering to / and not /mnt folders about namespace operations.
I've repeated the write tests in both environments a few times now and there's no consistent result.

WSL 2

root@LUCIANO-PC:/home/# dd if=/dev/zero of=/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 7.0873 s, 1.5 GB/s
root@LUCIANO-PC:/home/# dd if=/dev/zero of=
/testfile bs=1M count=20000

20000+0 records in
20000+0 records out
20971520000 bytes (21 GB, 20 GiB) copied, 51.3779 s, 408 MB/s
root@LUCIANO-PC:/home/#
root@LUCIANO-PC:/home/# dd if=/dev/zero of=~/testfile bs=1M count=20000
^C12360+0 records in
12360+0 records out
12960399360 bytes (13 GB, 12 GiB) copied, 12.1766 s, 1.1 GB/s

WSL1

root@LUCIANO-PC:/home/# dd if=/dev/zero of=~/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 29.7096 s, 353 MB/s
root@LUCIANO-PC:/home/#

I'm thinking this has to do something with OS Caching because in task manager even though I can see the file being written, the disk usage write speed is unchanged while in wsl1 I can see it constantly at around 350mb/sec .

As another interesting fact, the vmmem usage while writing the file is increasing rapidly. By the 3rd copy action all my RAM is used up (32GB).
It goes back down a while after quitting my shell. So maybe the performance is bottlenecked by this bug as well.

@paultyng

This comment has been minimized.

Copy link

@paultyng paultyng commented Jul 12, 2019

I'm also experiencing this after an upgrade to WSL2, only slow performance under /mnt, not /, slower than WSL1. I can post numbers as well if necessary. This is on Microsoft Windows [Version 10.0.18936.1000].

@definelicht

This comment has been minimized.

Copy link

@definelicht definelicht commented Jul 16, 2019

I'm experiencing the same issue, /mnt is extremely slow compared to WSL1 (seconds to run git status on a relatively small repository).

@paultyng

This comment has been minimized.

Copy link

@paultyng paultyng commented Jul 18, 2019

When I run a git status on a clone of https://github.com/hashicorp/terraform in /mnt I get output like this:

It took 6.97 seconds to enumerate untracked files. 'status -uno'
may speed it up, but you have to be careful not to forget to add
new files yourself (see 'git help status').
@ioweb-gr

This comment has been minimized.

Copy link
Author

@ioweb-gr ioweb-gr commented Jul 19, 2019

This is an example from the command

pv files.tar.gz | tar -zxf -

The speed is amazingly slow in wsl2 on /mnt

image

@definelicht

This comment has been minimized.

Copy link

@definelicht definelicht commented Jul 22, 2019

Do we know what's at the root of this, and whether there are any other workarounds than downgrading?

@benhillis

This comment has been minimized.

Copy link
Member

@benhillis benhillis commented Jul 22, 2019

@definelicht - Yep we're working on improving the performance. In the meaning working out of your root file system (the ext4 volume) will have MUCH better performance.

@ioweb-gr

This comment has been minimized.

Copy link
Author

@ioweb-gr ioweb-gr commented Jul 27, 2019

Indeed after the last update I can already see much better performance on WSL2.

root@LUCIANO-PC:~# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 41.277 s, 254 MB/s

That's on an SSD and it's like 6 times faster than before. Still slower then WSL1 but it's definitely improved.

--correction---
I see much better performance on sequential writes. Untar seems very slow still extracting data at bytes/sec while in wsl1 I can see speeds at kb/sec for the same part

@tuananh

This comment has been minimized.

Copy link

@tuananh tuananh commented Aug 12, 2019

Indeed after the last update I can already see much better performance on WSL2.

root@LUCIANO-PC:~# dd if=/dev/zero of=/mnt/e/testfile bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 41.277 s, 254 MB/s

That's on an SSD and it's like 6 times faster than before. Still slower then WSL1 but it's definitely improved.

--correction---
I see much better performance on sequential writes. Untar seems very slow still extracting data at bytes/sec while in wsl1 I can see speeds at kb/sec for the same part

I got


1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 3.57894 s, 293 MB/s

but git status still taking more than 2 seconds on quite small repo

@definelicht

This comment has been minimized.

Copy link

@definelicht definelicht commented Aug 12, 2019

I think this is a latency rather than a bandwidth issue, surfacing when a large number of files are accessed (e.g., when running git status).

@tuananh

This comment has been minimized.

Copy link

@tuananh tuananh commented Aug 14, 2019

@definelicht - Yep we're working on improving the performance. In the meaning working out of your root file system (the ext4 volume) will have MUCH better performance.

can you share the cause of it? I'm curious

@gbatalski

This comment has been minimized.

Copy link

@gbatalski gbatalski commented Aug 15, 2019

can confirm a very bad performance on /mnt (mvn install takes forever), on /tmp it takes only 2 minutes for the same project

@christmas-fr

This comment has been minimized.

Copy link

@christmas-fr christmas-fr commented Sep 11, 2019

After switch configuration from WSL to WSL 2 (Build 18975), without any change on my distro, the server respond time incresed.

Configuration : Debian buster Apache2.X / PHP-FPM 7.3 / MySQL
Root : /
Web directories : /c/repositories/xxx

WSL 1 : 0.5 sec
WSL 2 : 8 sec

With Docker (Ubuntu 18.04) WSL Tech preview i could not test on WSL 1, but WSL 2 takes over 30 sec... I suppose the issue is the same.

@realtica

This comment has been minimized.

Copy link

@realtica realtica commented Sep 13, 2019

This is my performance in a medium project:

GIT_TRACE_PERFORMANCE=1 git status -sb -uno

13:20:04.630972 read-cache.c:2267 performance: 0.017249300 s: read cache .git/index
13:20:07.766270 preload-index.c:146 performance: 3.135180100 s: preload index
Refresh index: 100% (3463/3463), done.
13:20:53.531359 read-cache.c:1575 performance: 48.900269600 s: refresh index
13:20:53.981829 read-cache.c:2267 performance: 0.016242500 s: read cache /mnt/e/p/keyed_au/.git/ modules/Files/media/nusoap/index
13:20:54.383135 read-cache.c:1575 performance: 0.401222700 s: refresh index
13:20:54.426399 diff-lib.c:251 performance: 0.000005000 s: diff-files
13:20:54.468000 unpack-trees.c:1554 performance: 0.000087700 s: traverse_trees
13:20:54.468266 unpack-trees.c:465 performance: 0.000164900 s: check_updates
13:20:54.468346 unpack-trees.c:1649 performance: 0.000438000 s: unpack_trees
13:20:54.468368 diff-lib.c:537 performance: 0.001873400 s: diff-index
13:20:54.569793 read-cache.c:2989 performance: 0.014679900 s: write index, changed mask = 2
13:20:54.586550 trace.c:477 performance: 0.736638600 s: git command: /usr/lib/git-core/git status --porcelai n=2 -uno
13:20:55.277184 read-cache.c:2267 performance: 0.031572000 s: read cache /mnt/e/p/keyed_au/.git/ modules/Files/modules/unittest/index
13:20:55.958706 read-cache.c:1575 performance: 0.681302500 s: refresh index
13:20:56.007094 diff-lib.c:251 performance: 0.000003500 s: diff-files
13:20:56.041768 unpack-trees.c:1554 performance: 0.000081200 s: traverse_trees
13:20:56.042014 unpack-trees.c:465 performance: 0.000162100 s: check_updates
13:20:56.042084 unpack-trees.c:1649 performance: 0.000399800 s: unpack_trees 13:20:56.042102 diff-lib.c:537 performance: 0.000454400 s: diff-index 13:20:56.126615 read-cache.c:2989 performance: 0.012095500 s: write index, changed mask = 2 13:20:56.141174 trace.c:477 performance: 1.261971900 s: git command: /usr/lib/git-core/git status --porcelain=2 -uno 13:20:56.148677 diff-lib.c:251 performance: 2.586112000 s: diff-files 13:20:56.190888 unpack-trees.c:1554 performance: 0.000022200 s: traverse_trees 13:20:56.191028 unpack-trees.c:465 performance: 0.000027800 s: check_updates 13:20:56.191106 unpack-trees.c:1649 performance: 0.000639200 s: unpack_trees 13:20:56.191175 diff-lib.c:537 performance: 0.001413100 s: diff-index
13:20:56.215068 trace.c:477 performance: 51.628126000 s: git command: git status -sb -uno

File System exFat, the result is the same is other file system...
WSL 2 git=51.62 sec
git for windows=0.8 sec

@paslandau

This comment has been minimized.

Copy link

@paslandau paslandau commented Sep 15, 2019

I'm currently using the normal Docker Desktop with linux containers to run a PHP projekt during local development. The repo is in a normal Windows directory and gets mounted via volume bind in the docker container, so that I can easily manipulate the files through an IDE.

The same setup on a native linux machine has always been multiple times faster and the "problem" (from what I've read) has always been attributed to the Linux VM that is used on Docker Desktop.

Then WSL 2 was announced and I finally came around to set everything up. But when I ran our teststuite for the first time, I was pretty disappointed that the performance was actually much worse than before (40s on the Docker Deskop Linux VM setup vs 100s on the WSL2 setup). Subsequent investigation brought me to this thread.

From what I've read here so far:

Does it even make sense to switch to WSL 2 for local development yet if "bad performance" is your primary reason to do so?

A "no" would be a totally acceptable answer - I just don't want to waste any more time on this if it's simply not ready yet.

@stweedie

This comment has been minimized.

Copy link

@stweedie stweedie commented Sep 19, 2019

I know that wsl2 is a preview, but is there any hope that accessing \wsl$ from windows or /mnt from wsl will reach acceptable speeds?

@gormulent

This comment has been minimized.

Copy link

@gormulent gormulent commented Sep 24, 2019

I'd also like to ask about this, doing a git status on something in /mnt/c/.. takes about a minute, something seems wrong ..

@gbatalski

This comment has been minimized.

Copy link

@gbatalski gbatalski commented Sep 27, 2019

same with 1903 (18990.1) absolutely useless...

@wusticality

This comment has been minimized.

Copy link

@wusticality wusticality commented Sep 27, 2019

I am also seeing this, accessing Windows files from the Linux environment is several orders of magnitude slower in WSL2 than in WSL1.

@toxmc

This comment has been minimized.

Copy link

@toxmc toxmc commented Nov 13, 2019

I will be use WSL1 until WSL2 solve this problem.

@nhanledev

This comment has been minimized.

Copy link

@nhanledev nhanledev commented Nov 19, 2019

Just try out the WSL 2 today on Build 19013.vb_release.191025-1609 and notice the performance drop on /mnt/. I have to continue to work in a VM until this is fixed, because there are no way to work with PHPStorm/WebStorm without accessing files in mounted drives.

@ioweb-gr

This comment has been minimized.

Copy link
Author

@ioweb-gr ioweb-gr commented Nov 21, 2019

Fyi. With every passing version it's getting worse instead of improving. I can't even run composer anymore because it times out checking out git repositories. Reverted to WSL1 today on my main instance

@kliakos

This comment has been minimized.

Copy link

@kliakos kliakos commented Nov 22, 2019

Any roadmap on fixing this issue?

@justeene

This comment has been minimized.

Copy link

@justeene justeene commented Nov 25, 2019

Folder /mnt/c is too slow.
image

@rudzboy

This comment has been minimized.

Copy link

@rudzboy rudzboy commented Nov 27, 2019

Rarely rollbacking has been so satisfiying.
How is this even released...

@fredgalvao

This comment has been minimized.

Copy link

@fredgalvao fredgalvao commented Dec 4, 2019

I've started noticing recently that disk activity on /mnt/d (SSD on C:, HDD on D:) is reaching 100% easily sometimes, even when there's nothing intentionally using anything on that disc or any folder/file inside it (see edit below) I couldn't trace what exactly is triggering this activity or being accessed there, but I can say that almost nothing has changed on my use pattern or on the linux image to justify this, other than constant fast-ring updates. It does feel, through my anedoctal evidence, that IO is not getting much better with time.

Let me know if there's any way I can monitor in details such a scenario involving WSL2 so that I can provide more information and maybe help identify bottlenecks.

I'm really hoping WSL2 will improve on IO the same way it has improved recently on memory usage (it's amazing now).

EDIT: I was so into the *nix mindset that I had forgotten I have access to the amazing Resource Monitor from Windows, which now shows that the thing getting constantly bombarded is pagefyle.sys, though not only by vmmem. I can say now that this issue on my case is not really immediately IO then, but overall resource consumption (that is only augmented by a suboptimal IO environment).
The thing is, I have RAM available for everything to fit in there, but a lot of disk activity still routes to windows virtual memory...

@effolkronium

This comment has been minimized.

Copy link

@effolkronium effolkronium commented Dec 4, 2019

just use docker inside linux vm ;)

Also i don'k understand, why there is no ways to get access to windows drives without network just with forwarding read&writes to nt kernel...

seems like the project is developing by early-courses students

@digitalfiz

This comment has been minimized.

Copy link

@digitalfiz digitalfiz commented Dec 5, 2019

Still experiencing this in December... wsl2 is pretty much unusable if you need to work out of any thing not already in the linux file system.

@ArthurJam

This comment has been minimized.

Copy link

@ArthurJam ArthurJam commented Dec 6, 2019

Just +1

@cpbotha

This comment has been minimized.

Copy link

@cpbotha cpbotha commented Dec 6, 2019

I did some measurements using the latest slow ring WSL2 as of today and wrote them up in this blog post: https://vxlabs.com/2019/12/06/wsl2-io-measurements/

WSL 2 is currently about 5x slower than WSL 1 accessing files on NTFS.

However, WSL 2 is showing such great potential everywhere else, so I really hope that MS is able to address this issue also!

@daniel-pospisil

This comment has been minimized.

Copy link

@daniel-pospisil daniel-pospisil commented Dec 8, 2019

The same here - starting a node project (using AdonisJS framework) takes about 30s on WSL2 while around 5s on WSL1. I have to go back until this is solved..

@gabtzi

This comment has been minimized.

Copy link

@gabtzi gabtzi commented Dec 8, 2019

Is.this impossible to fix?

@ctivanovich

This comment has been minimized.

Copy link

@ctivanovich ctivanovich commented Dec 11, 2019

Yeah, somehow the newest insider build got even slower.

@ioweb-gr

This comment has been minimized.

Copy link
Author

@ioweb-gr ioweb-gr commented Dec 11, 2019

So far I've converted all my machines to WSL1 again until this gets better. Please do let us know when there are updates regarding this so we can retest for you.

@thomascooper

This comment has been minimized.

Copy link

@thomascooper thomascooper commented Dec 23, 2019

I just wanted to +1 this as well

@arryanggaputra

This comment has been minimized.

Copy link

@arryanggaputra arryanggaputra commented Dec 25, 2019

+1

@chx

This comment has been minimized.

Copy link

@chx chx commented Dec 25, 2019

My primary use case is running phpstorm in the Windows GUI and Linux command line tools on the same set of files. I do not look forward to switch to phpstorm on Linux as support is (obviously) lesser and one of the reasons I didn't even try native Linux on the current laptop is hotplugging GPUs all the time (TB3 eGPU) and so I really do not wish to touch X unless there's some X Server which doesn't go hardware level but then how's performance? Anyways, I hope this gets resolved. It seems we got from the frying pan into fire.

@sameronline

This comment has been minimized.

Copy link

@sameronline sameronline commented Dec 25, 2019

Agreed @chx , This file access performance issues is just killer and making the wsl2 not usable, sticking to wsl1 for now

@moigagoo

This comment has been minimized.

Copy link

@moigagoo moigagoo commented Dec 25, 2019

Please cut it with those “me too” comments. The issue is there, well-documented, acknowledged, probably being addressed. There's absolutely no need to constantly remind all 44 subscribers that there's still nothing new here.

The issue affects you, we get it. Just add a reaction to the initial comment, subscribe, and do NOT comment unless you have anything useful to add to what's already been said.

@findjobs-deploy

This comment has been minimized.

Copy link

@findjobs-deploy findjobs-deploy commented Dec 26, 2019

Moigagoo: thank you for fighting for all of the subscribers that can't handle the notification spam, but other people lending their support to an issue to attempt to get a major problem fixed is not a problem for everyone. Open-source github issues users generally follow the accepted "+1/me too" pattern, but for those who are inconvenienced by the notifications, you can just instead bookmark this page and unsubscribe to the notifications. I am not looking to start an internet fight or trolling competition, but personally I appreciate the "+1"s here and on many issues I track, so I am defending those following this path. stop notifications

I am also affected by the issue referenced here and hope that this has enough traction that it is being working on actively. Look forward to testing the fix when its available.

@anedisi

This comment has been minimized.

Copy link

@anedisi anedisi commented Dec 26, 2019

i kinda don't mind the notifications, and somehow was hoping for some sort of the official comment. for me i jumped back to linux and invested more time there. i just cannot work with this performance and i got tired of all the workarounds. im suprised that they would release this kind of quality even for beta.

@ioweb-gr

This comment has been minimized.

Copy link
Author

@ioweb-gr ioweb-gr commented Dec 28, 2019

I think that some regular status updates (if there are any) on the progress of this issue from people who are overseeing this project would probably stop people spamming

@craigloewen-msft

This comment has been minimized.

Copy link
Member

@craigloewen-msft craigloewen-msft commented Dec 31, 2019

Hi everyone, thank you for the discussion here on /mnt performance in WSL2. We realize this is a dealbreaker for many of you commenting here, and we are looking into ways to improve this, and make the file system performance faster in /mnt drives.

As of right now, if you can move your project over to the Linux file system, please consider doing so and using a WSL2 distro running Linux applications accessing files in your Linux file system for faster file performance. If your use case requires you to access Windows files, please feel free to move back to a WSL1 distro, or consider having a WSL1 distro and a WSL2 distro side by side for your different applications.

We'll be sure to post any updates on this thread as soon as they are available. Thank you for your patience!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.