Qrexec Agent Starts Incorrectly and too Early on Windows 10 HVM

[GWeck]

Qubes OS version
R4.0.1 fully updated

Affected component(s) or functionality
Qubes Windows Tools 4.0.1.3

Brief summary
When starting a Windows 10 (Version 1803 or 1909) HVM with QWT installed, both Qubes services start (Qubes RPC Agent and QubesDB daemeon), but most of the time Qubes complains that it cannot connect to the qrexec agent of the VM, and functions like file copy do not work. This seems to be a timing problem, as it does not always happen.
If, however, the Qubes RPC Agent start mode is set to Manual (or even disabled) and the agent was stopped (interactively or via the command net stop qrexecagent) before shutting down windows, this service does not start at the next boot. If then the service is started interactively, connection to Qubes is established immediately and the full functionality is available.

To Reproduce
Install QWT 4.0.1.3 on a freshly installed Windows 10 Pro systemrunning as HVM under Qubes. Set qvm-feature gui to 1 to get an interactive window. Set qvm-prefs qrecec_timeout to a larger value (600 or so) to reduce stress.
Reboot the VM and observe its state in the Qube manager.

Expected behavior
The state should always turn to green, and full functionality should be availabl,e like it is on Windows 7.

Actual behavior
If the state stays at yellow, which happens most of the time, the qrexec channel between host and guest is not working, although the correspondig Qubes RPC Agent service in the guest is running. If, on the other hand, the state turns to green, communication is established.

Additional context
The Qrexec Agent starts, irrespectively of its starttype, if it was active at the last shutdown before boot - even if its starttype is set to disabled. If it is running, it can be shut down interactively or via the command net stop qrexecagent. Trying to restart the service interactively or via the command net start qrexec agent crashes Windows with error 0x0000dead in xenbus.sys.
If the agent is not running at boot time, it can be started interactively. Trying to start it via the command line leads to the same crash. The VM has to be killed in order to be able to start it again.
Probably not connected to this, but still annoying: The logs complain that the RPC qubes.SetMonitorLayout fails due to File not Found. Creating a file with this name, e.g. by copying qubes.SetGuiMode under this name, in the qubes-rpc directory removes this error, but does not change the other behaviour.

Solutions you've tried
First set the starttype of Qubes RPC Agent to Manual. This can be done while it is running, even if there is no communication with the host system.
Before shutdown of the VM, stop the service Qubes RPC Agent.
After booting, start the service Qubes RPC Agent interactively - not by command, as this would crash the VM.
In most cases, you have then a fully functional system.

Relevant documentation you've consulted
Documentaion on QWT

Related, non-duplicate issues
Possibly #5332 and #3585

Log files on a successful start of Qubes RPC Agent with connection to the host
qrexec-agent-20191115-094051-6456.log
qrexec-wrapper-20191115-094051-6552.log
qrexec-wrapper-20191115-094051-6672.log
qubesdb-daemon-20191115-094014-2784.log

[GWeck]

If the Windows 10 VM is created as a template VM, it behaves quite the same as the HVM does. This is the same no matter whether the Xen network and disk drivers are installed or not. (If the Xen base drivers are not installed, the Qubes RPC Agent and the QubesDB Daemon cannot run at all, like with the HVM. So these drivers have to be installed.)

So the situation is as follows:

• If the Qubes RPC Agent is started at boot time, no communication with the host is possible. This occurs if the service starttype is set to automatic or if this service was running at the previous shutdown.
• If the service is restarted, the VM will crash.
• If the service is not running at boot time, it can be started. Most of the time, communication with the host is the established, but sometimnes the VM crashes instead.

If an AppVM based on this template is created, it behaves different:

• The Qubes RPC Agent is started at boot time, if it was runnung at the last shutdown of the template VM.
• If the service is running at boot time, no host communication is possible, just like this is the case with the template VM.
• If the service is not running at boot time, the AppVM will crash whenever the service is started. It was not possible to start it in any case.

I wonder if the newer Xen drivers (version 8.2.2), which are available now, are any better than the ones (version 8.2.1) currently used by QWT. It might be worth a try, but I was not able to run QWT with the newer drivers.

[GWeck]

If the Qubes RPC Agent is started manually, and Windows 10 crashes, communication with the host system is established shortly - the status indicator in the Qube Manager turns green, and a notification Domain has started is displayed - and then the VM crashes immediately. This occurs in the HVM, the template VM and in the AppVM in exactly the same way.

If a running Qubes RPC Agent is stopped and then restarted, the crash occurs immediately without communication being established, also in all types of VM.

During my tests, I once had a template VM starting the agent at boot time - the service starttype was set to Automatic - and getting communication with the host. After shutdown of this template VM, the corresponding AppVM started and had communication with the host. The AppVM could be started severeal times, and each time communication was established. After the next boot of the template VM, the situation returned to the previous behaviour, i.e. no communication was established if the service starttype was set to Automatic, and the AppVM would crash if the service starttype was set to Manual and the agent was started manually.

This is all rather confusing, but I hope it will help to track the problem down.

[brendanhoar]

I’ll ask the silly questions first: you have 4GB assigned for both mem and maxmem and have disabled memory balancing, correct?

[GWeck]

I had both initial and max memory set to 6144 MB. Memory balancing was on for the HVM and the template VM, and off for the AppVM.

I now changed this to initial memory 4000 MB on all VMs - max. memory stayed at 6144 MB when memory balancing was turned off - and switched memory balancing off on all VMs. The behaviour of all machines did not change - no communication if the agent was started automatically, and a mix of correct starts and crashes if it was started manually.

However, I found a certain pattern in the crashes:

• For the HVM and the template VM, the crashes do not occur randomly but on each second boot. So, if the agent starts correctly, I may leave it running when shutting down the VM; in this case it will run after the next boot, but etstablish no communication with the host. If I stop it before shutting down the VM, it will not run after the next boot and will crash the VM if it is started, but then after the next boot, it can be started manually and will establish host communication.
• For the AppVM, the behaviour depends on what happend before with the template VM. If the template VM crashed on the previous run, the agent will start correctly after each boot - just as is to be expected for the next boot of the template VM. If the template VM ran correctly on the previous run, the AppVM wil crash on each start of the agent.

So I suppose that shutting down the agent leaves something that will crash it on the next start, and that this crash somehow clears the offending data. This seems consistent with the fact that stopping the agent and then restarting it immediately crashes the VM. Anyhow, if the agent is started automatically, it runs, but without establishing host communication.

[GWeck]

Here are some log files that might help to investigate the errors.

Installation logs of Qubes Windows Tools
Qubes_Windows_Tools_4.0.1.3_20191120115727.log
Qubes_Windows_Tools_4.0.1.3_20191120115727_001_qubes_tools_WIN7x64_4.0.1.3.msi.log

Logs of Qubes RPC Agent starts without crash
qrexec-agent-20191123-101458-6308.log
qrexec-agent-20191123-101129-7124.log
The first file concerns a start with a dysfunctional qrexec agent, i.e. the agent is running but unable to connect to the host. The second file describes a successful start after which communication is established. The first 41 lines of both files are equivalent, i.e. up to WatchForEvents: waiting for event.
In line 42 of the successful start, we get WatchForEvents: event 1 and then see that the channel to the host is working, whereas in that line in the unsuccessful start, we get the error message SvcCtrlHandlerEx: code: 4 0x4, probably meaning that the service control handler got a problem, and after that, there is silence.

Log file of Qubes RPC Agent start crashing Windows and correspondig crash dump
qrexec-agent-20191123-105642-6628.log
112219-12312-01.zip
In the case of the crash, the log ends with line 11, SvcMain: start. It is unclear whether this was the cause of the crash or if the rest of the log simply could not be flushed to disk, because the crash was too quick. I have no idea if the dump file might be of any value, but at least it is there.

[GWeck]

If the Xen bus and storage drivers Version 9.0.0 are installed in a Windows 10 system without Qubes Windows Tools, and QWT 4.0.1.3 are installed after the Xen installation has finished, the Qubes interface works correctly. Files can be exchanged with other VMs, and the common clipboard works in both directions. So to get a working Windows 10 system under Qubes R4.0, the following steps should be performed:

• Install xenvbd and xenbus Version 9.0.0 from the Xen web site.

• After installation, reboot.

• Install Qubes Windows Tools 4.0.1.3, unselecting the Xen storage driver and the Move of user data to Drive D (which would probably lead to problems in Windows, anyhow).

• Shut down Windows.

• Set qvm-features gui 1.

• Reboot Windows. The VM starts, but does not show any window.

• Shutdown Windows from the Qube manager.

• Reboot Windows once more. Now the system is up, with QWT running correctly.

For me, this sequence worked for Windows 10 as template VM, and a correspondig AppVM worked too.

It might be a good idea to create a new version of QWT, containing these updated drivers, but unfortuately I was not able to do this.

[GWeck]

Installing the version 9.0.0 Xen drivers on Windows 7 (a system witout QWT - QWT uninstalled) leads to an unbootable system. The drivers install without error, but after reboot, the system aborts the reboot saying "Missing driver xenbus.sys".

So, for Windows 7, it seems that we have to stick to the current QWT 4.0.1.3.

[brendanhoar]

My recollection is that I had good luck with the latest version of the 8.x.x drivers before 9.0, using win7 on qubes r4.0.x. That version’s Pv storage driver seems stable and allows qubes-initiated block device attaches.