virtio-fs service on a win10 QEMU guest fails before restart and user cannot modify created files

[marcello-pietrobon]

As probably well known virtio-fs on a Win10(20H2) Windows guest (and maybe other versions too) is unusable for some users.

The way this problem manifests at first is that a user is able to create/delete files or directories on a drive mapped through virtio-fs, but then he cannot modify them in any way (where in the case of a directory this means to add any file to it or remove it).

On a first look this problem appears of the same nature of few others that have been affecting some WinFsp filesystems, but on a further analysis it actually seems to be specific to the virtio-fs driver which, before restart, returns VirtFs->LocalUid == VirtFs->LocalGid == 0 (or better most probably does so).

Once the virtio FS service is restarted (after logging in into the VM at least) the returned uid/gid are correct and the file permissions correctly mapped to those of the underlying linux host filesystem.

A more complete description of this problem is found in virtio-fs with a win10 QEMU guest and WinFsp

I'm available to perform some testing or answer other questions if needed.

Thank you.

Edit: I wonder if the virtio-fs service should be started after some other windows service that I'm not aware of...

[daiaji]

Encountered a similar problem. After logging in to the WIN10 guest, the folder cannot be created in the drive letter mapped by virtio-fs, but everything is normal after restarting the virtio-fs service.
Does the virtio-fs service start prematurely?

[hammerg]

This issue should have been resolved in commit 57aedaa. Are you working with a version that includes this fix?

The virtio-fs service is depending on two other services. The WinFsp.Launcher and the VirtioFsDrv driver. So a manual installation of it should look something like:

C:\> sc create VirtioFsSvc binpath="(your binary location)\virtiofs.exe" start=auto depend="WinFsp.Launcher/VirtioFsDrv" DisplayName="Virtio FS Service"

[daiaji]

Thank you. Does it have a compiled executable?

[marcello-pietrobon]

It works!

Thank you so much for having taken care of this so promptly, very much appreciated.

This is what I did to compile:
I went to

• https://github.com/virtio-win/kvm-guest-drivers-windows/tree/57aedaa3009692e9d7d7f590d16490753129ba9e
• downloaded the zip
• installed vs community edition
• opened the solution and rebuilt only the VirtioFS folder by right-clicking on it
  (the batch files are a bit outdated, but I believe that if we use VS2019 we just need to replace in them 2017 with 2019)
  copied the content of the generated 'viofs\Install\Win10' dir into the 'VioFs' directory where it was installed.

[marcello-pietrobon]

There's only one small issues left, I write it here as memo for anyone who may face this problem, but I'm content already with the fix... and for what it concerns me this issue can be closed.

Here's the description of the remaining issue:
Everything works, except with some zip files. Some cannot be copied. It's very weird.
This is how it goes:

• I create a zip from a dir that I created. Then copy the zip file into the shared folder z:. I can copy or move this zip file into another dir, no problem.
• I create a zip file from a dir that I didn't create (for example coming from some installation). Then copy the zip file into the shared folder z. I cannot copy nor move this zip file into another dir. I can modify it there or delete it though. Even as administrator I cannot change its security properties in order to try making it copyable,

The difference between the two cases seems to me only that in the first case it was me who created the directory.

Edited:
I don't know if this is an issue with virtio-fs or with WinFsp.
but what I find asthonishing is that if I run fsptool-x64.exe or Icacls on both zip files they give exactly the same result. For example for test1.zip is:

z:>"c:\Program Files (x86)\WinFsp\bin\fsptool-x64.exe" perm z:\test1.zip
O:S-1-5-21-2819646085-1004851513-3197712829-1001G:S-1-5-21-2819646085-1004851513-3197712829-513D:P(A;;0x1f01bf;;;S-1-5-21-2819646085-1004851513-3197712829-1001)(A;;0x1201af;;;S-1-5-21-2819646085-1004851513-3197712829-513)(A;;0x1200a9;;;WD) (perm=197609:197121:0775)

z:>Icacls z:\test1.zip
z:\test.zip DESKTOP-MYSELF\myself:(M,WDAC,WO)
DESKTOP-MYSELF\None:(RX,WD,AD,WA)
Everyone:(RX)

[ipilcher]

@marcello-pietrobon Can you provide a bit more detail on the process you used to build the driver?

For example, I haven't been able to get VisualStudio installed, as I have no idea which "workloads" I need to install. And assuming that I'm able to get the driver built, I'm not at all clear on how to install the updated driver; you wrote that you copied the generated files into the VioFs directory, but I don't have any directory by that name in my Windows 10 guest. Also, don't I have to sign the drivers somehow in order to install them?

[YanVugenfirer]

Hi @ipilcher The build environment instalaltion instructions are here: https://github.com/virtio-win/kvm-guest-drivers-windows/wiki/Building-the-drivers-using-Visual-Studio-2017

How to test sign during the build: https://docs.microsoft.com/en-us/windows-hardware/drivers/develop/signing-a-driver-during-development-and-testing

How to enable test signing on Windows: https://docs.microsoft.com/en-us/windows-hardware/drivers/install/the-testsigning-boot-configuration-option (tl;dr: run "Bcdedit.exe -set TESTSIGNING ON" with administration privileges and reboot).

[daiaji]

@marcello-pietrobon Thank you, I will test it later.

[ipilcher]

I was finally able to work through the process of building and signing the driver. My initial smoke testing seems to confirm that the latest code fixes this problem.

Could we please get a signed release with this fix included? The process to build and sign the driver is quite involved (not to mention the need to put the guest into "test mode" in order to use a self-signed driver).

[marcello-pietrobon]

Sorry I couldn't reply sooner.

I'm waiting too for a new release with this problem fixed, as meanwhile I have to run the Win VM with "Bcdedit.exe -set TESTSIGNING ON"

I used very simple and partial instructions in order to save time and trouble, so if someone else has to go through this...

The instructions here are an addition to those I wrote few posts above here.

The source code is in c, so you just install the standard C/C++ part of the Visual Studio Community edition installer (the 'Desktop development with C++' tab in the picture you see in
https://docs.microsoft.com/en-us/cpp/build/vscpp-step-0-installation?view=msvc-160 )
and you also make sure to install the latest 'Windows 10 SDK' that you see in there (or other Windows version you are using).

Make also sure to go on the 'Individual Components' tab of the installer and select 'Windows Driver Kit'.

Now you are set to load the viofs/viofs.sln solution and compile the viofs project in there.

The trick that saves you a lot of time is to just compile the viofs part of the kvm-guest-drivers-windows code (that is not the entire library) by using the viofs/viofs.sln solution that you find in there, as just said.

On top of Visual Studio you can clearly see and choose the Platform and Configuration you need to compile.

Once compiled it appears an 'install' directory under the viofs project, and with the files in there you overwrite those of the original installation (backup them first).

At this point, I didn't care signing the drivers, I just used the command "Bcdedit.exe -set TESTSIGNING ON"

It's even simpler than it looks like written here.

Hope this helps,

If somebody wants I can attach the binaries I generated for myself. I didn't change any line of code of course but I don't want to take any responsibility for any problem whatsoever, there shouldn't be any though.

M

[TecEssence]

@YanVugenfirer , hello
What is the release cycle of the virtio-win installation package? can the new version of the installation package be released as soon as possible? The latest on (https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/) is still the version in November 2020

[YanVugenfirer]

@TecEssence we are building new packages based on the added features and fixed bugs.

I will check how soon a new version can be uploaded.

Best regards,
Yan.

[TecEssence]

@YanVugenfirer ,hello

We look forward to the new virtio-win installation package, when will it be released?

Best regards,
Qing.

[YanVugenfirer]

Hello,

We are expecting the package during next two week.

Best regards,
Yan.

[fubar-1]

Hello @YanVugenfirer,

Has there been any progress on releasing an updated & signed package with bug fixes?

Thanks,
Dave

[YanVugenfirer]

Hello @Dave--G,

I am pushing the release of the package. There is a problem with a signature now. MS retired cross-signed certificates, and we are working to attestation sign the release. Most probably the next release will be unsigned, yet.

Best regards,
Yan.

[fubar-1]

Thanks for the update @YanVugenfirer,

This is unfortunate. I understand the certificate matter is out of your hands. This sounds like a significant development. What is virtio-win's long term strategy to resolve this issue? Can you point us to where this is being publicly discussed?

[YanVugenfirer]

@Dave--G

We are working in two directions:

1. Automating attestation singing for Windows 10. This way Windows 10 drivers will be signed with attestation signing provided by Microsoft and could be installed as the drivers previously signed with a cross-signing certificate.
2. Try to influence Microsoft to reverse their policy for older OSes.

Here are some of the public discussions:

• https://community.osr.com/discussion/292723/what-we-know-about-cross-signing-for-down-level-windows-versions-as-of-today-2-march-2021
• https://community.osr.com/discussion/comment/301169

[oosoG]

I'm also suffering from this bug.
Because it seems that there are only certification problems for older versions of Windows. For the moment, can't someone create the virtiofs.exe driver (+viosoc?) for WINDOWS 10 ? (eventually publishing them on a different location)

[oosoG]

While waiting for better. Isn't there someone who can share the compiled files? (signed, testsigned or not signed) @marcello-pietrobon @ipilcher ?

[YanVugenfirer]

@oosoG unsigned drivers build from the current master for Windows 8 and Windows 10 32 and 64bit: https://www.dropbox.com/s/x86xam26uuuc8z7/VirtioFS-Install.zip?dl=0

[oosoG]

Thank you @YanVugenfirer for sharing Its working (question: #590)

[fubar-1]

Thanks @YanVugenfirer for the unsigned build. This will save some time in setup testing this evening.

Can you give us any expectation for when the next signed build will be available?

[jwatt]

It looks like the contents of https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/ got an update a few days ago, and after installing that in Windows the VirtIO FS Device driver has version 100.85.104.20400 and says it's signed by 'Microsoft Windows Hardware Compatibility Publisher'.

[ilmari-lauhakangas]

I updated the driver to the latest and while I can rename and modify files, git does not work. My use case is bisecting LibreOffice regression bugs. I can't launch executables, I can't do checkouts. It might initialise some files and then fall on its face.

If I do

git clone -o bundle win32-5.0.git-bundle win32-5.0

I get

fatal: remote did not send all necessary objects

Can others reproduce the problem with git? I am using git 2.29.2.windows.2 under Powershell 7. I also updated WinFSP to v1.9.

Simplest test:

PS Z:\> git init test
fatal: cannot copy 'C:/Program Files/Git/mingw64/share/git-core/templates/hooks/commit-msg.sample' to 'Z:/test/.git/hooks/commit-msg.sample': No such file or directory

Shared dir is defined as

<filesystem type="mount" accessmode="passthrough">
  <driver type="virtiofs"/>
  <source dir="/home/user/libobugs"/>
  <target dir="libobugs"/>
  <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</filesystem>

[arthurrasmusson]

I am also experiencing this bug on Windows 10 LTSC on an Arch Linux host. The virtiofs.exe Windows service appears to crash during some operations and others refuse the ability to write or delete files - I am able to create files and sometimes running executables works however sometimes it does not - possibly in cases where the .exe wants to modify something. I've tried changing file permissions in the shared directory to 777 to see if that might mitigate the problems and the socket is in group KVM. I'm not clear on if what I am experiencing is the same very same issue but on the latest public drivers I installed yesterday this issue appears to still exist.

[Striffly]

I also have problems when I perform operations with the virtio-fs shared folder :

List of actions made inside guest :

• Create / delete file inside shared folder : works
• Copy file from local folder to shared folder : works
• Copy file from shared folder to local folder : fail, crash service
• Duplicate file inside shared folder (copy in same place inside shared folder) : fail, crash service

Host config :

• Arch Linux
• qemu 6.2.0-2
• virt-manager 7.10.0-2

Guest config :

• Windows 10 Guest with virtio devices (https://github.com/Fmstrat/winapps/blob/main/docs/KVM.md) & virtio-fs shared folder (https://wiki.archlinux.org/title/libvirt#Virtio-FS)
• virtio-win-0.1.215 ISO with drivers & wirtio-win-guest-tools installed
• spice-guest-tools installed
• WinFSP
• Virtio FS Service launched

[YanVugenfirer]

@viktor-prutyanov Please take a look

[viktor-prutyanov]

Hi @Striffly,

Thank you for detailed description.

Copy file from shared folder to local folder : fail, crash service
Duplicate file inside shared folder (copy in same place inside shared folder) : fail, crash service

Could you please try adding queue="1024" to your libvirt XML as follows:

...
<filesystem type="mount" accessmode="passthrough">
  <driver type="virtiofs" queue="1024"/>
  ...
</filesystem>
...

If it doesn't work, could you please share your libvirt virtiofs-related XML and clarify, what was the approximate size of the file you copying?

[viktor-prutyanov]

Hi @arthurrasmusson,

I am able to create files and sometimes running executables works however sometimes it does not - possibly in cases where the .exe wants to modify something.

Could you please clarify, what executables are you using? If these are some private things, have you ever had such a problem with some common executables or even with standard command-line utilities? For now, there is very little information. I can only suggest you to add queue="1024" (as described above) if it hasn't already been done.

[ilmari-lauhakangas]

@viktor-prutyanov see my comment from August. Are you able to run 'git init' in a shared folder without problems?

[viktor-prutyanov]

Hi @ilmari-lauhakangas,

Sometimes command

Z:\> git init test

reproduces the error you got. But sometimes it works.

[viktor-prutyanov]

@ilmari-lauhakangas, PR #719 is supposed to fix your problem with git init

[viktor-prutyanov]

Hi @Striffly, @arthurrasmusson,

If your problems are still actual, I suggest you to open separate issues with your problems.
I have to write this because current issue seems to be solved and should be closed.

[agrisvv]

i can say some more comments:
work:

• copy 300 mb from and to.
• duplicate files with different name.
• creating new files and folders, but in Host os the show perms like drwxrwxr-x and then cant write anything inside.

dose not work:

• edit same file. ( create new txt file, edit , and can't save). even with host perms 777.
• cant owerwrite file with same name. even with host perms 777

no problems with linux guest. event with 770 perms on files.
no changes with DistinctPermsForSameOwnerGroup 1
on windows file perms shows as everyone read only and s-1-5-0 as modify.

env:

• fedora 35 host
• libvirt runs ar qemu user.
• /shared_data owned by qemu
• host user same as linux user - name but windows user name are different.
• win10 21h2 x64
• winFsp 1.10.22006
• virtio package 215
• from xml

   <filesystem type="mount" accessmode="passthrough">
      <driver type="virtiofs" queue="1024"/>
      <source dir="/shared_data/untrusted"/>
      <target dir="shared_data"/>
    </filesystem>

[viktor-prutyanov]

Hi @agrisvv,

Could you please open new issue?

Also, please add information about your host filesystem.

[ilmari-lauhakangas]

* virtio package 215

This does not yet include the fix. Would be nice to get an updated ISO. I haven't tested the fix either.

[viktor-prutyanov]

* virtio package 215

This does not yet include the fix. Would be nice to get an updated ISO. I haven't tested the fix either.

You can build the driver by yourself. It became much easier with the advent of EWDK: https://github.com/virtio-win/kvm-guest-drivers-windows/wiki/Building-the-drivers-using-Windows-11--EWDK

[arthurrasmusson]

@viktor-prutyanov

Could you please clarify, what executables are you using?

I have GZDoom and Steam games in the shared path:
https://www.zdoom.org/downloads

[arthurrasmusson]

@viktor-prutyanov

I can only suggest you to add queue="1024"

I am not currently using Libvirt - I'm using LibVF.IO.
What does this correspond to in raw QEMU commands?

[viktor-prutyanov]

@arthurrasmusson, queue-size=1024. For example:

-device vhost-user-fs-pci,chardev=chr-vu-fs0,queue-size=1024,tag=mount_tag,bus=pci.6,addr=0x0

[arthurrasmusson]

@viktor-prutyanov

queue-size=1024. For example:

-device vhost-user-fs-pci,chardev=chr-vu-fs0,queue-size=1024,tag=mount_tag,bus=pci.6,addr=0x0

Since upgrading packages on Arch I cannot get the VM to start. I'm now using the following parameters on start:
-mem-path /dev/hugepages
-object memory-backend-memfd,id=mem,size=8G,share=on
-numa node,memdev=mem
-chardev socket,id=char0,path=/home/user/.local/libvf.io/sockets/virtio-fs-001.sock
-device vhost-user-fs-pci,chardev=char0,tag=lime,queue-size=1024

On start virtiofsd crashes on the host with the following error message:
"virtio_loop: Unexpectted poll revents 11"

The host is Arch Linux.

Is VirtIO-FS currently considered stable for use with Windows guests?

[viktor-prutyanov]

Since upgrading packages on Arch I cannot get the VM to start.

What is your QEMU version?

On start virtiofsd crashes on the host with the following error message: "virtio_loop: Unexpectted poll revents 11"

On virtiofsd start or on VM start?

Is VirtIO-FS currently considered stable for use with Windows guests?

No, it is a tech preview.

[arthurrasmusson]

@viktor-prutyanov

What is your QEMU version?

6.2.0

On virtiofsd start or on VM start?

On VM start.

[xiagao]

@viktor-prutyanov

queue-size=1024. For example:

-device vhost-user-fs-pci,chardev=chr-vu-fs0,queue-size=1024,tag=mount_tag,bus=pci.6,addr=0x0

Since upgrading packages on Arch I cannot get the VM to start. I'm now using the following parameters on start: -mem-path /dev/hugepages -object memory-backend-memfd,id=mem,size=8G,share=on -numa node,memdev=mem -chardev socket,id=char0,path=/home/user/.local/libvf.io/sockets/virtio-fs-001.sock -device vhost-user-fs-pci,chardev=char0,tag=lime,queue-size=1024

On start virtiofsd crashes on the host with the following error message: "virtio_loop: Unexpectted poll revents 11"

Seems the vm failed to start, the virtiofsd quit with "virtio_loop: Unexpected poll revents 11", so do you have any error msg from qemu-kvm?

The host is Arch Linux.

Is VirtIO-FS currently considered stable for use with Windows guests?

[viktor-prutyanov]

I have to close this issue because the original problem seems to be solved. Please open new separate issues with your problems, as I previously suggested.