VC4 kernel framework driver - memory leak

[nkichukov]

Hi all,
(redirected here via: raspberrypi#2293)

I have identified a memory leak in the VC4 kernel framework driver implementation on raspberry pi 3b. Details follow below:

kernel: 4.14.0-v7+
compiler: gcc-6.4
raspberry pi running in 32bit mode
OS: Gentoo Linux
Kernel configuration is attached, see below.

Whenever kodi(v17.6) is playing video, I can see that /proc/slabinfo reports number of kmalloc-128 constantly increasing and never releasing:
kmalloc-128 84916 84924 384 21 2 : tunables 0 0 0 : slabdata 4044 4044 0

In about 24 hours this has taken up about 600MB of the memory and the Pi needs to be restarted to avoid swapping. If I stop kodi playback and leave it idle, the kmalloc-128 remains the same until playback is resumed which starts increasing again. If I stop the X server and kodi altogether, the memory is not released.

The memleak pattern is like the one below

(from /sys/kernel/debug/kmemleak):

unreferenced object 0x95ce16c0 (size 128):
comm "kodi.bin", pid 2075, jiffies 891363 (age 2338.023s)
hex dump (first 32 bytes):
01 00 00 00 5c ed 1e 7f 00 00 00 00 00 00 00 00 ...............
d0 16 ce 95 d0 16 ce 95 00 3a fb b4 00 00 00 00 .........:......
backtrace:
[<802827c8>] kmem_cache_alloc_trace+0x234/0x2f8
[<7f1e461c>] vc4_submit_cl_ioctl+0x730/0xb50 [vc4]
[<7f0d796c>] drm_ioctl_kernel+0x78/0xb8 [drm]
[<7f0d7cd4>] drm_ioctl+0x1b4/0x37c [drm]
[<802ac090>] do_vfs_ioctl+0xb0/0x8b4
[<802ac8d8>] SyS_ioctl+0x44/0x6c
[<80108140>] ret_fast_syscall+0x0/0x28
[] 0xffffffff

Let me know if additional information is required to track this issue down.

rpi-4.14.y-kernel_configuration_file.gz

Saw another issue still open here for memory leak, but what I report happens with pretty high CMA, 256MB, so not related to low CMA setting. Also the trace is different.

Thank you,
-Nikolay

[stschake]

I think it's leaking the dma_fences, certainly the "01" refcount in the hexdump looks suspiciously like them. Could you try the attached patch?

0001-drm-vc4-Release-fence-after-signalling.txt

(Sorry, filename is off because GitHub, ridiculously so, does not allow attaching .patch files)

[nkichukov]

Hello Stefan,
thank you for the quick response! It fixed the huge memleak issue! Thank you very much!

However, another memleak appeared(it may already have been there before, I just did not see it), see below:

unreferenced object 0x842aa340 (size 128):
comm "X", pid 2009, jiffies 7034334 (age 301.527s)
hex dump (first 32 bytes):
50 3f eb b6 01 00 00 00 00 00 00 00 00 00 00 00 P?..............
50 a3 2a 84 50 a3 2a 84 00 00 00 00 00 00 00 00 P..P..........
backtrace:
[<802827c8>] kmem_cache_alloc_trace+0x234/0x2f8
[<7f167ae8>] drm_atomic_helper_setup_commit+0x1cc/0x3b0 [drm_kms_helper]
[<7f1e2aa0>] vc4_atomic_commit+0x30/0x130 [vc4]
[<7f0f44a0>] drm_atomic_commit+0x5c/0x68 [drm]
[<7f0f5700>] drm_atomic_connector_commit_dpms+0xf8/0x108 [drm]
[<7f0fae64>] drm_mode_obj_set_property_ioctl+0x1bc/0x2b4 [drm]
[<7f0f9820>] drm_mode_connector_property_set_ioctl+0x48/0x50 [drm]
[<7f0e296c>] drm_ioctl_kernel+0x78/0xb8 [drm]
[<7f0e2cd4>] drm_ioctl+0x1b4/0x37c [drm]
[<802ac090>] do_vfs_ioctl+0xb0/0x8b4
[<802ac8d8>] SyS_ioctl+0x44/0x6c
[<80108140>] ret_fast_syscall+0x0/0x28
[] 0xffffffff
unreferenced object 0x83ee2340 (size 128):
comm "X", pid 2009, jiffies 7044631 (age 291.296s)
hex dump (first 32 bytes):
50 3f eb b6 01 00 00 00 00 00 00 00 00 00 00 00 P?..............
50 23 ee 83 50 23 ee 83 00 00 00 00 00 00 00 00 P#..P#..........
backtrace:
[<802827c8>] kmem_cache_alloc_trace+0x234/0x2f8
[<7f167ae8>] drm_atomic_helper_setup_commit+0x1cc/0x3b0 [drm_kms_helper]
[<7f1e2aa0>] vc4_atomic_commit+0x30/0x130 [vc4]
[<7f0f44a0>] drm_atomic_commit+0x5c/0x68 [drm]
[<7f0f5700>] drm_atomic_connector_commit_dpms+0xf8/0x108 [drm]
[<7f0fae64>] drm_mode_obj_set_property_ioctl+0x1bc/0x2b4 [drm]
[<7f0f9820>] drm_mode_connector_property_set_ioctl+0x48/0x50 [drm]
[<7f0e296c>] drm_ioctl_kernel+0x78/0xb8 [drm]
[<7f0e2cd4>] drm_ioctl+0x1b4/0x37c [drm]
[<802ac090>] do_vfs_ioctl+0xb0/0x8b4
[<802ac8d8>] SyS_ioctl+0x44/0x6c
[<80108140>] ret_fast_syscall+0x0/0x28
[] 0xffffffff
unreferenced object 0x84bbe040 (size 128):
comm "X", pid 2009, jiffies 7049751 (age 286.242s)
hex dump (first 32 bytes):
50 3f eb b6 01 00 00 00 00 00 00 00 00 00 00 00 P?..............
50 e0 bb 84 50 e0 bb 84 00 00 00 00 00 00 00 00 P...P...........
backtrace:
[<802827c8>] kmem_cache_alloc_trace+0x234/0x2f8
[<7f167ae8>] drm_atomic_helper_setup_commit+0x1cc/0x3b0 [drm_kms_helper]
[<7f1e2aa0>] vc4_atomic_commit+0x30/0x130 [vc4]
[<7f0f44a0>] drm_atomic_commit+0x5c/0x68 [drm]
[<7f0f5700>] drm_atomic_connector_commit_dpms+0xf8/0x108 [drm]
[<7f0fae64>] drm_mode_obj_set_property_ioctl+0x1bc/0x2b4 [drm]
[<7f0f9820>] drm_mode_connector_property_set_ioctl+0x48/0x50 [drm]
[<7f0e296c>] drm_ioctl_kernel+0x78/0xb8 [drm]
[<7f0e2cd4>] drm_ioctl+0x1b4/0x37c [drm]
[<802ac090>] do_vfs_ioctl+0xb0/0x8b4
[<802ac8d8>] SyS_ioctl+0x44/0x6c
[<80108140>] ret_fast_syscall+0x0/0x28
[] 0xffffffff

Thank you,
-Nik

[stschake]

That is an unrelated leak (another refcounting fail, 01 00..)

[nkichukov]

Hello Stefan,
If this is unrelated to the vc4 kernel driver, do I report this to the raspberrypi kernel developers? Or (if vc4 related) would you like to have a separate issue for this leak here(anholt) and track separately so this can be closed and the patch merged?

Thank you,
-Nik

[lategoodbye]

Since this is also vc4 related, it should be reported here in this repository and not to the raspberrypi guys.

[nkichukov]

Thank you Stefan. The originally reported issue was resolved by the provided patch. I am closing the case and will open a new one for the second memory leak.