-
Notifications
You must be signed in to change notification settings - Fork 413
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slow disk write speed for shared folders mounted via vmhgfs-fuse #246
Comments
I'm seeing similar performance. I'd like to know the same as @smattheis . VMWare team - it's been 4 months since this issue was filed, anything insights? |
Could you please confirm in your test how is /tmp mounted in the guest and in the host? Is /tmp going to disk on both sides or memory? There may not be anything wrong in your setup because HGFS over FUSE is expected to slower. However, we will have to look into any possible improvements in our implementation. Thanks for reporting this issue. I have filed an internal bug to look into any performance improvement possible there. |
Of course, /tmp is a directory on the virtual disk mounted as / into the guest OS. |
Sorry not getting to this before now. I am actively going to look into this and see what is going on here. |
I have seen the same behavior previously with
|
Same here indeed:
|
I have same issue.. with CentoOS |
Just to clarify and ensure that everyone is aware of what is going on when you run these benchmark tests. It is great to have people run them and check out the performance. Points to consider:
Thanks |
Thanks @lousybrit for the insights and many thanks for working on the problem. Some comments: (1) I reported the problem because for me a use case broke after I switched from VMWare Fusion version 8 to version 10. The use case was that operations on a local Git repository, which I mounted as shared folders between host and guest, timed out and corrupted my repository. I did some tests and found that the mentioned performance characteristics might be the explanation especially because it was no problem to work on non-shared folder within the VM. In conclusion, I think VMWare is a tool mostly for developers and having a local Git repo in a shared folder is something very convenient and maybe a very frequent use case. I hope that this is enough to catch the attention of VMWare to keep working on the issue. From my side, I will do some Git operation tests to maybe have a reproducible case that you could test and work with. |
@lousybrit Thanks so much for working on this! My use case was running an EDA tool (Quartus) on the Linux guest with the database in a shared folder. This is necessary in my case because some work is performed on Mac and some on Linux. I have no control over the type if I/O Quartus performs, so I can't change the block size. The net effect is that Quartus takes orders of magnitude longer to run which renders it unusable with a shared folder. For me the telling behavior here is that the fuse process consumes essentially 100% of the CPU when running Quartus. Same is the case when running DD with smaller block sizes. So this is is not the case of communications latencies between guest and host. There is a huge amount of CPU resources being used here on the guest, much more than what seems reasonable for packaging up the data to send to the host even one byte at a time. If I had to guess, this feels like a spin lock, or some other type non-productive loop in Fuse. |
Hi there, Inline responses below.
This seems like a useful test case for us.
It really does depend on the file IO profile. Here with dd we can see that it is a number of write operations of a consistent block size when small adds up to a larger number of round-trips to the server. That would cause a file update to take longer and so overall time latency for the file update. That could be an issue or not, it is impossible to say exactly without the Git profile and issue. It would be the same as running Git with your "local" checked out repository to be on a real networked server that is remote and is slow in responding. I have not looked into that kind of set up with Git.
I don't think I have enough information yet to fully know why Git was broken from Fusion 8 to 10.
Agreed and that would be awesome. Thanks |
Thanks very much for the background and information here. Do you have a single CPU VM? Thanks
|
@lousybrit I just replicated on an 8 core guest to confirm. Fuse taking 88% on the guest. Not sure if that percent is of a single core or all cores combined. The host is reporting Thanks!
|
This is just one of the cores. As you can see see total CPU usage is summarized in the header in %Cpu line and that is 10.6% only. |
User space: 10.6% However, if your application is waiting for FUSE to complete an IO, it can't do anything with the available/idle CPUs but wait for the IO to finish. |
Apologies to "bump" an old thread. Is there some official or "official-like" response on the matter for its status? Any workarounds apart from duplicating files? |
I'm having the same issue Centos Guest on Windows 10 host. This is really important, would like to get some feed back. |
I'm having the same issue with Ubuntu 18.04 + Windows 10 host. One thing I noticed, enabling |
There has been multiple answers about this from myself and other VMware colleagues. Can you either give some context about the application in question with regard performance and what steps you use to reproduce the issue you are referring too? It should also be noted that we set big_writes to true as default so setting this explicitly on the command line should not have any effect. It might be dictated by what application you are using and any arguments you are passing. Can you supply more information? Thanks |
This is a generic response, which doesn't really help us, in return, answer:
My case is sharing a workspace from my Host (Ubuntu 16.04) home directory, to my Guest (Ubuntu 18.04) home directory. Most notably, |
My case is Centos 7 guest running on Windows 10 Host. Using LAMP stack on Centos. As Stavros Ntentos noted before, we don't need a generic response, but a real fix/patch. |
Same here, it's not unworkable, but slow enough to be annoying. Setup: Fedora 30 guest running in Windows 10 host. My development files (PHP) are located on the host, where my IDE is; the Linux guest runs the web server that reads the PHP files through the shared folder. This setup is much slower than when the files are located on the guest directly (which is not possible due to the IDE and other tools being located on the host—or would require a rather complicated sync process). Also, things such as I can definitely understand and accept some overhead due to the translation/encapsulation/transport of filesystem commands on the guest filesystem to commands on the host filesystem. It's just that the overhead is right now big enough to give a hint that some kind of optimization must be possible, or at least mandate some investigation. |
I have the same.... vmhgfs is really slow - not faster than vboxsf - and consumes nearly 100% of a core on transferring data. Perhaps I go better with |
Any news on this issue? Debian 10 guest on a Windows 7 host. If I use dd to make a write speed test, I get around 950 MB/s directly on the VM disk and only 170 MB/s on the mounted shared drive. It's really slow! |
@StephaneArguin Consider yourself lucky, as you're probably transferring reasonably big files, hence you still get a decent transfer speed and "only" a ~5x performance penalty. When working with many small files though, I get ~25x slower speeds in the shared folder, compared to an ext4 folder in the VM ( Plus, I have to note that symlinks are still not properly handled, and that inotify notifications are not forwarded to the guest machine. That's 3 reasons why shared folders are almost unusable for any serious software development using VMware shared folders at the moment. If VMware wants to win the battle against upcoming WSL2, they'd better fix shared folders, and quick. Microsoft is fixing WSL issues at lightning speed right now, and whoever gets this right first will probably win developers' business. At least, they'll win mine. I'd be ready to put quite some money in an hypervisor that would fix the 3 issues above, right now. |
@BenMorel My test was just the creation of one 1G file using 'dd'. But like you, I have the same problem when I'm using Git : it's really slowwww. Almost unusable. And I also have an issue if I try to untar a file with symlinks in it. I think I'll recommend my client to stop paying for VMWare Workstation licences and simply use VirtualBox for developer's machines because there is no advantage to pay for this. Or better, authorizing developers to use Linux machines (99.9% Java development) and save VMWare and Windows licenses. :) |
EXACTLY THAT. In dev you have several systems with LOTS of small files... and shared folder suck at this |
Just upgraded to macOS Catalina on my iMac with SSD drive. Using Fusion 11.5.0. Having the same experience as all describe. vmhgfs-fuse -Vvmhgfs-fuse: version 1.6.9.0 FUSE library version: 2.9.7 |
Same here Guest system: Ubuntu 16.04.2 LTS FUSE library version: 2.9.4 |
I would really like to switch from virtualbox to vmware, but this issue is making that impossible. I cannot wait for 10 seconds for every git command to complete. Back to virtualbox I go... VMWare Fusion Version 11.5.3 (15870345) on macOS Mojave Guest system Ubuntu 18.04 $ vmhgfs-fuse -V FUSE library version: 2.9.7 |
@dinkelk because of that issue I've switched to parallels. My colleague switched and same result. Way better performance. |
@mbm-michal That is a good idea. Unfortunately, for my team, I need a virtual environment that works on a Windows, Mac, and Linux host. I tested this morning, the same slowness exists on a Windows 10 host :( |
same( |
Why aren't you people using a network share like samba instead? it's more flexible and more secure anyways. |
Because this is slow TOO - we´re only always (I think so) talking about A LOT of SMALL files. The overhead of SMB and obviously of VMHGFS for accessing ONE file is huge compared to direct access. I would expect lightning fast access to host file system in a payd product. |
I immediately feel like 🤣🤣😂😂🤣😂🤣😂 |
Okay okay.... but I could swear: ~8 years ago with older versions of Vmware I had no such issues. Since I stopped using host fs access I´m not that interested in solving the issue anymore. |
Is there a better solution for this? Experiencing the same slowness with fuse. |
Hi Ben and Stephane
If you use dd to test performance as I mentioned in earlier posts, set the block size for transfer to something like 60kb. Defaults are small and small chunks do get poor performance as you have seen as the ratio of overhead to actual useful file data read is not great. Using larger chunks yields vastly different and improved throughput. Having said all of that, what happens in reality depends on other factors, such as the applications being used, which can differ greatly when comes to reading/writing file data. Secondly, it does depend on the profile of the file data e.g. lots of small files, repeated access to the same files, only large files, one single large file, etc. I am filing bugs with regard your specific setup and use cases you have mentioned so that are team which is working on this feature (not me by the way) can make sure that these areas are tested and improved. Otherwise, time and resources could be allocated to other or the wrong issues. It has been slow to get resources allocated to work on this feature which is important and can make a huge difference to the Workstation/Fusion product line. Now, I would hope current and future work will take this issues and address them. Thanks |
Hi all! |
I just don´t use it anymore. Switched to WSL2 with much less pain and much more speed. |
The answer is quite simple for Linux guests. Once upon a time VMWare provided a proper Linux kernel driver for hgfs. Not enough payback for the maintenance cost, so they went to user space FUSE-based code. Every hgfs operation requires several context switches user <-> kernel space. Until they provide a kernel driver again, hgfs will be essentially useless for many purposes. |
I was using sshfs for quite some time in openSUSE guest. I switched to hgfs after I upgraded to 15.2 where it mostly was working fine. Now I just upgraded to 15.3 (with updates to open-vm-tools or some other change somewhere) and the excessive CPU load is back. I'll go back to sshfs again. |
In '
Works just fine. Should add to this that I also use |
i try virtualbox(mount -t vboxsf github /mnt/) and vmware, both use shared floder, and my os is centos7.8, My code is placed in a shared folder. My compilation environment: c++, gcc10.2 ccache, test result: |
So ... 5 years later, this issue is still not fixed. What a bummer... unbelievable! |
I am now on VMWare Fusion 13 on Macbook Pro M1. Mounting the host machine over sshfs works really well and the problems I had before with frequent disconnections rarely happen now. I would recommend to use sshfs over vmhgfs-fuse any day. The directories work pretty much as if they were local. Just make sure any symlinks on the host that you rely on are relative. |
So you don't have the high CPU load anymore? What about the disk write speed? |
No, never had that problem when using sshfs. I lost patience with hgfs. Actually, I don't see why it's even necessary when sshfs is better. You just have to make the effort to set things up a bit, the ssh key on the host and some configuration in fstab. I actually also use Parallels on another machine, but I still stuck with using sshfs on that vm as well. |
Oh, about disk write speed. It's perfectly fine. I haven't made any speed tests but, like I wrote, it pretty much performs like a local disk. I haven't noticed any problems when it comes to speed. I just tried copying a 500 MB file, took less than 2.4 seconds. |
I'm using VMWare Fusion 10.1.1 on Mac OS X 10.12.6 with Ubuntu 16.04 (Linux ubuntu 4.4.0-121-generic) and observe that the speed for writing to shared folders mounted with vmhgfs-fuse is extremely different from writing to a folder of the guest OS.
I installed open-vm-tools-desktop with the following versions:
In a non-shared folder of the guest OS, I get the following speed:
However, in a shared folder, I get the following speed:
That's a huge difference which, unfortunately, breaks some of my use cases for shared folders. Do I have something wrong in my setup?
The text was updated successfully, but these errors were encountered: