-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
unicam: CMA Memory leak #3220
Comments
@6by9 Have you experienced any memory leaks with your testing on the unicam driver? |
https://github.com/kbingham/libcamera/ branch kbingham/rpi for the libcamera pipeline development |
I hadn't seen anything from bcm2835-unicam. All the buffer handling is within videobuf2, so that should deal with most of the housekeeping. There was a report that the kernel module refcount for videobuf2_dma_contig keeps on increasing when using GStreamer. I'd put that down to bcm2835-codec, but rereading it could be bcm2835-unicam as that is also used in their pipeline. |
I haven't had a chance to investigate that report, but my first steps were going to be checking as to whether vcm-cma was keeping references to dmabufs. |
Yes, @6by9 - I can indeed see the module use count is increasing:
So the usecount increases by 8 each iteration |
Hrm: On an idle run (No current QCam executing)
|
Thanks for running those couple of tests. I suspect something is unbalanced on the dmabuf refcounts, therefore each dmabuf is incrementing the exporters module refcount. |
Aha ! Thanks - that's just the debug I was looking for :-D I'd been grepping for CMA info and all sorts. I'm afraid that's pointing to vcsm-cma!
pi@rpi:~/libcamera $ sudo cat /sys/kernel/debug/dma_buf/bufinfo | grep "devices attached" | wc -l So 52 leaked buffers, which closely matches the current: assuming an extra refcount for each of the unicam/codec modules. |
Hmm, so vcsm-cma thinks it has released all resources it had, but the dmabufs don't. I don't know how easy it is for you to do, but I've just asked on the forum thread if we can try swinging the allocations round so that codec allocates, and unicam imports. Ignore the request if it is more than 5mins of effort as I'll hook a GStreamer pipeline together to replicate the forum setup instead. (Ooh, I see an interesting WIP commit at the top of your libcamera branch. Now I'm curious what you're up to with that!) |
Actually - those dmabufs get exported into the ISP, so it could equally be an issue with not correctly unexporting them there :-D Aha- and your next comment just popped up - yes, I'll try to swap the allocations around. I think that's easy. |
I bought a B101 to play with and experiment :-) |
Next you'll be integrating the audio side into libcamera! Nothing like a bit of feature creep ;-) |
Ok - so it seems there is no leak in this direction... Before After 2 runs: |
Thanks. That does pretty much confirm that it is dmabuf import with vcsm-cma. |
Is there some magic required to run qcam?
/dev/video0 is set to 1920x1080 raw10 BGGR Gstreamer seems not to be hitting issues when doing things like video transcode (filesrc ! matroskademux ! h264parse ! v4l2h264dec ! v4l2h264enc). That may be because it intercepts ctrl-c and does some cleanup. |
@6by9, No there "shouldn't" be ... but we are lacking any kind of format negotiation with the sensor currently. In my tests, it always just worked with a fixerd format from the IMX219. This should just ask the sensor what it can support and pick one, and then set that on the ISP. |
...
The 0x41414270 (in the error message above) is indeed V4L2_PIX_FMT_SBGGR10P (what your /dev/video0 is using). But the pipeline handler (raspberrypi.cpp) tries to set V4L2_PIX_FMT_SRGGB10P (the hardcoded value), and returns the error ("Failed to set format on Video device...") if the resulting format is different from RGGB. The quick hack I've used was to change the V4L2_PIX_FMT_SRGGB10P to V4L2_PIX_FMT_SBGGR10P in src/libcamera/pipeline/raspberrypi.cpp. (The proper fix would be to make the pipeline handler more flexible WRT sensor pixel format) |
Thanks @andrey-konovalov - that was exactly it. |
BTW @6by9 (in relation to this issue) kbingham/libcamera@2332f1d shows how I swapped the buffer allocation around from the unicam to the ISP. If you are testing this issue on my kbingham/rpi branch, please be aware of that commit. It has a #if 0 to switch allocations. |
I was running your old TOT with manual hack to swap from RGGB to BGGR. That's reproduced the issue, so now trying to remember how this all works.... |
It looks like the cleanup code for releasing an imported dmabuf is just plain wrong. It's going to require a touch more thought, but the cause looks fairly obvious. |
I'm not sure if this is the bug you were seeing, but I've noted that we're unbalanced in calling
Before this I was seeing dmabufs left described in /sys/kernel/debug/dma_buf/bufinfo with no attachments but a refcount roughly equal to the number of times the buffer had been presented to the component. I can't say for definite that that is the leak you're seeing, but it would generally follow. PR incoming. |
Sounds like it's along the right lines. I'll be back in the office next week so I can test things for you then. |
I've tried this change on RPi0 with recent rpi/rpi-4.19.y (when it was at commit a287d34 "rpi-wm8804-soundcard: Fixed MCLKDIV for Allo Digione"). After I ran qcam 5 times, the amount of free CMA memory didn't change ('cat /proc/meminfo |grep Cma'), the used count of videobuf2_dma_contig didn't increase ('lsmod | grep ^videobuf2_dma'), and 'sudo cat /sys/kernel/debug/dma_buf/bufinfo' showed "Total 0 objects, 0 bytes". So this change fixes the CMA memory leak in the unicam driver. Guess this issue can be resolved after #3293 is merged. |
Thanks for the ping here, Yes I have confirmed with @6by9 that the updates here have fixed the issues we saw. Closing this issue. |
Describe the bug
Capturing through the unicam driver leaks CMA memory after the device is closed.
To reproduce
Expected behaviour
There should be no considerable consumption of CMA when an application closes and releases buffers.
(even if there is a bug in libcamera, the driver should have released all resources on close)
Actual behaviour
Each iteration and launch of qcam (from libcamera) consumes about 10mB from CMA. I doubled the allocation on the unicam from 4 buffers of 1080p to 8 buffers of 1080p, and the memory leak doubled to ~20mB per iteration.
System
Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW
Which OS and version (
cat /etc/rpi-issue
)?Which firmware version (
vcgencmd version
)?uname -a
)?The text was updated successfully, but these errors were encountered: