sys-net occasionally dies on resume from suspend

[andrewdavidwong]

Qubes OS version:

R3.2

Affected component(s):

sys-net

---

Steps to reproduce the behavior:

1. Close lid to suspend.
2. Wait for machine to suspend.
3. Open lid.
4. Machine resumes from suspend.
5. Unlock screen locker.

Expected behavior:

sys-net stays on and continues to provide network access normally.

Actual behavior:

Occasionally:

• NetworkManager in sys-net looks like it's trying to connect for a second, then the entire sys-net just dies (shows powered off in Qubes Manager).
• sys-firewall and AppVMs using sys-firewall for network access are still running normally, but of course they don't have network access.
• Restarting just sys-net does not restore network access to these other AppVMs.
• Instead, they must all be restarted, beginning with sys-net. (Maybe restarting just sys-net and sys-firewall would be enough, but in practice it's easier for me just to shut them all down and restart them all.)

General notes:

This has been going on for at least a few months. When reading #4657, I saw that it mentioned this problem with sys-net in passing. I thought we already had an issue on this (see below) but couldn't find one, so I'm filing this now.

---

Related issues:

I could have sworn we already had an issue about this, but after searching, I can't find one. These all look different:

#2964 was about losing network access when sys-net stays on
#3008/#3030 was about failing to connect to a network after resume when sys-net stays on
#3151 was about NetworkManager not running in sys-net after resume when sys-net stays on
#3738 was about the entire computer not resuming correctly from suspend.

Ah, maybe I was thinking of #4042, which is a similar report about sys-usb.

[brycepg]

Posted bounty ($100): https://www.bountysource.com/issues/67959138-sys-net-dies-on-resume-from-suspend

[unman]

@andrewdavidwong @brycepg I feel this must be hardware specific, since I dont see it on any of the machines that I still have access to running 3.2.1.
It might be helpful if you could post details of the hardware you are using, and specifically the NIC.
I'm assuming that you see this using both Fedora and Debian templates? If you haven't checked please do so.

[brycepg]

@unman

Laptop model: Lenovo Thinkpad T530
Wireless adapter: Intel Centrino Advanced-N 6205 [Taylor Peak] (rev 34)
I've only tested this issue on my fedora-28 template. Will try with debian-9
CPU: i5-3210m (no IOMMU)

EDIT: Strangely I cannot repro right now with a quick suspend. Will try out with longer suspend times.

[andrewdavidwong]

I should have clarified that this only happens occasionally, so it would be difficult to reproduce without suspending/resuming many times. I'll update the issue to reflect this.

Lenovo Thinkpad T450s:

Xen:		4.6.6
Kernel:		4.14.74-1

RAM:		20194 Mb

CPU:
  Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz
Chipset:
  Intel Corporation Broadwell-U Host Bridge -OPI [8086:1604] (rev 09)
VGA:
  Intel Corporation HD Graphics 5500 [8086:1616] (rev 09) (prog-if 00 [VGA controller])

Net:
  Intel Corporation Ethernet Connection (3) I218-LM (rev 03)
  Intel Corporation Wireless 7265 (rev 59)

[4vrel]

same issue, thinkpad x220, qubes 4.0

Maybe a script that shut down and restart all net-vm could be a temporary fix ? ( its pretty long and painful to do it manually )

[tasket]

This happened to me recently after re-doing some templates and VMs. What I discovered was that my wifi module suspend settings were gone.... I forgot to re-add them.

I agree with unman this is a hardware-specific issue. OTOH it would be nice if Qubes had some way of automatically populating this module information.

Here is what I use in /rw/config/suspend-module-blacklist for an Intel "Ultimate-N" card:

iwldvm
iwlwifi

[aslfv]

In my case, sys-net will always die on resume from suspend, if another VM is in a transient state (i.e. qube manager displaying the VM with a yellow dot). This is in particular true for the following two cases which others may be able to reproduce as well:

1. the other VM is starting while the machine is suspending.
2. A windows VM -- without QWT -- is running, while the machine is suspending.

The result in my case is the following output in sys-net:
[ 372.065513] ath10k_pci 0000:00:06.0: failed to wake target for read32 at 0x0003a028: -110
[ 372.657769] clocksource: timekeeping watchdog on CPU0: Marking clocksource 'tsc' as unstable because the skew is too large:
[ 372.657817] clocksource: 'xen' wd_now: 27c2a46c622 wd_last: 27c0c3c4a2a mask: ffffffffffffffff
[ 372.657856] clocksource: 'tsc' cs_now: ffffff9953059778 cs_last: ffffffecec397004 mask: ffffffffffffffff
[ 372.657902] tsc: Marking TSC unstable due to clocksource watchdog
Please also note that this may occasionally occur even for VMs that are not connected to a NetVM.

In the case of sys-usb the machine may even entirely fail to resume, and instead reboots.

I hope that this is relevant for this issue. If not please indicate how I can proceed. For the record, essentially the machine (DELL XPS 13 9360) is running R4.0.

[marmarek]

@aslfv your case looks like #3489