kswapd spinning at 100% CPU

[marmarek]

Qubes OS version:

R3.2

Affected TemplateVMs:

Fedora 25, but probably unrelated.
VM kernel version 4.9.45, but happens also on 4.9.56. And it also happened on 4.4.

---

Steps to reproduce the behavior:

No exact steps, but it happens under memory pressure in the VM.

Actual behavior:

kswapd spins at 100% CPU. Backtrace caught using sysrq:

[180421.153140] NMI backtrace for cpu 1
[180421.153146] CPU: 1 PID: 42 Comm: kswapd0 Tainted: G           O    4.9.45-21.pvops.qubes.x86_64 #1
[180421.153153] task: ffff880011f2d880 task.stack: ffffc90001080000
[180421.153157] RIP: e030:[<ffffffff811cdddf>] c [<ffffffff811cdddf>] shrink_node_memcg+0x6ef/0x790
[180421.153164] RSP: e02b:ffffc90001083c80  EFLAGS: 00000206
[180421.153168] RAX: 000000000000117e RBX: ffffc90001083e48 RCX: ffff880018840c00
[180421.153174] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000000001ac37
[180421.153180] RBP: ffffc90001083d68 R08: 0000000000000000 R09: 0000000000000003
[180421.153186] R10: 0000000000000002 R11: 0000000000000000 R12: 000000000000003c
[180421.153192] R13: ffff88001883f000 R14: ffff88006352a400 R15: ffffc90001083da8
[180421.153199] FS:  0000000000000000(0000) GS:ffff880013f00000(0000) knlGS:0000000000000000
[180421.153205] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[180421.153209] CR2: 000055c9bc126000 CR3: 00000000aa502000 CR4: 0000000000042660
[180421.153215] Stack:
[180421.153218]  00000000000000c9c 000000000000002bc 0000000000000001c 0000000000001bc2c
[180421.153225]  0101000000000001c 000000000001ac37c ffffc90001083cb0c ffffc90001083cb0c
[180421.153231]  ffffc90001083cc0c ffffc90001083cc0c ffffc90001083cd0c ffffc90001083cd0c
[180421.153237] Call Trace:
[180421.153241]  [<ffffffff8113338c>] ? css_next_descendant_pre+0x4c/0x60
[180421.153245]  [<ffffffff81233997>] ? mem_cgroup_iter+0x1d7/0x310
[180421.153250]  [<ffffffff811cdf61>] shrink_node+0xe1/0x320
[180421.153254]  [<ffffffff811cef2c>] kswapd+0x32c/0x700
[180421.153257]  [<ffffffff811cec00>] ? mem_cgroup_shrink_node+0x180/0x180
[180421.153262]  [<ffffffff810c0576>] kthread+0xe6/0x100
[180421.153266]  [<ffffffff810c0490>] ? kthread_park+0x60/0x60
[180421.153270]  [<ffffffff810c0490>] ? kthread_park+0x60/0x60
[180421.153274]  [<ffffffff817dead5>] ret_from_fork+0x25/0x30
[180421.153278] Code: c00 c00 cbe c03 c00 c00 c00 c4c c89 cef c48 c89 c44 c24 c28 ce8 c58 cb8 c00 c00 cbe c02 c00 c00 c00 c4c c89 cef c48 c89 c44 c24 c18 ce8 c46 cb8 c00 c00 c48 c8b c7c c24 c28 c<48> c03 c7c c24 c18 c49 c8d c55 c60 c49 c8d cb5 c60 c17 c00 c00 c31 cc9 c48 c83 c3a c

Sometimes echo 3 > /proc/sys/vm/drop_caches helps for some time. Giving the VM more memory (for example closing Firefox instance in other VM) helps too.

---

Related issues:

qubes-users thread

[aQubesUser]

I see this somewhat frequently. I have been restarting the VM to sort this out.

[aQubesUser]

I see this somewhat frequently. I have been restarting the VM to sort this out.

[donob4n]

Maybe this will be fixed soon https://lkml.org/lkml/2018/1/29/580

I think it is out of scope of Qubes, last time I saw it I think I was doing a backup of some VM's. Probably we can not do any workaround, even people without swap partition is reporting same problem.

[donob4n]

Maybe this will be fixed soon https://lkml.org/lkml/2018/1/29/580

I think it is out of scope of Qubes, last time I saw it I think I was doing a backup of some VM's. Probably we can not do any workaround, even people without swap partition is reporting same problem.