libvirt memory leak

[marmarek]

Qubes OS version:

R4.0

---

Steps to reproduce the behavior:

Start/stop a lot of VMs.

Expected behavior:

Libvirt memory usage stays at initial amount (~1GB virt, ~40MB res).

Actual behavior:

Libvirt use a lot of memory (~8GB virt, ~500MB res), to the point it can't allocate it anymore:

libxl_fork.c:612:libl__ev_child_fork: fork failed: Cannot allocate memory
libxl.c:1560:domain_destroy_callback: unable to destroy guest with domid 587
libxl.c:1489:domain_destroy_cb: destruction of domain 587 failed

General notes:

I suspect it is cause by one of our patches, so lets check it (both libvirt and libxl) first before reporting upstream.

[marmarek]

After few thousands VM start/stop cycles, valgrind says:

==12684== 3,234,825,604 (5,211,024 direct, 3,229,614,580 indirect) bytes in 50,106 blocks are definitely lost in loss record 1,479 of 1,480
==12684==    at 0x4C2FA1E: calloc (vg_replace_malloc.c:711)
==12684==    by 0x54B8D93: virAlloc (viralloc.c:144)
==12684==    by 0x162681: virNetMessageNew (virnetmessage.c:42)
==12684==    by 0x1257F3: remoteDispatchObjectEventSend (remote.c:3993)
==12684==    by 0x129587: remoteRelayDomainEventLifecycle (remote.c:329)
==12684==    by 0x5584DD1: virDomainEventDispatchDefaultFunc (domain_event.c:1779)
==12684==    by 0x5583726: virObjectEventStateDispatchCallbacks (object_event.c:710)
==12684==    by 0x5583726: virObjectEventStateQueueDispatch (object_event.c:724)
==12684==    by 0x5583726: virObjectEventStateFlush (object_event.c:825)
==12684==    by 0x5583726: virObjectEventTimer (object_event.c:555)
==12684==    by 0x54D94BD: UnknownInlinedFun (vireventpoll.c:457)
==12684==    by 0x54D94BD: virEventPollRunOnce (vireventpoll.c:653)
==12684==    by 0x54D7E40: virEventRunDefaultImpl (virevent.c:314)
==12684==    by 0x563DF7C: virNetDaemonRun (virnetdaemon.c:819)
==12684== by 0x1247EA: main (libvirtd.c:1541)

Full final report: https://gist.github.com/marmarek/e887339a31aa1bef7810319f0919b708

[marmarek]

After few thousands VM start/stop cycles, valgrind says:

==12684== 3,234,825,604 (5,211,024 direct, 3,229,614,580 indirect) bytes in 50,106 blocks are definitely lost in loss record 1,479 of 1,480
==12684==    at 0x4C2FA1E: calloc (vg_replace_malloc.c:711)
==12684==    by 0x54B8D93: virAlloc (viralloc.c:144)
==12684==    by 0x162681: virNetMessageNew (virnetmessage.c:42)
==12684==    by 0x1257F3: remoteDispatchObjectEventSend (remote.c:3993)
==12684==    by 0x129587: remoteRelayDomainEventLifecycle (remote.c:329)
==12684==    by 0x5584DD1: virDomainEventDispatchDefaultFunc (domain_event.c:1779)
==12684==    by 0x5583726: virObjectEventStateDispatchCallbacks (object_event.c:710)
==12684==    by 0x5583726: virObjectEventStateQueueDispatch (object_event.c:724)
==12684==    by 0x5583726: virObjectEventStateFlush (object_event.c:825)
==12684==    by 0x5583726: virObjectEventTimer (object_event.c:555)
==12684==    by 0x54D94BD: UnknownInlinedFun (vireventpoll.c:457)
==12684==    by 0x54D94BD: virEventPollRunOnce (vireventpoll.c:653)
==12684==    by 0x54D7E40: virEventRunDefaultImpl (virevent.c:314)
==12684==    by 0x563DF7C: virNetDaemonRun (virnetdaemon.c:819)
==12684== by 0x1247EA: main (libvirtd.c:1541)

Full final report: https://gist.github.com/marmarek/e887339a31aa1bef7810319f0919b708