vc4_bo_create -> Failed to allocate from CMA.

[DeviceIoControl]

Bug Description:
Usage of any type of Hardware accelerated application (VLC media player, Kodi, Google Chrome, etc.) for an extended period of time (between 30 mins to 1 hour), causes CMA to fail to allocate memory (out of memory) for the vc4-fkms-v3d driver. This causes all of the Hardware accelerated applications running at that point in time to freeze (turn black) or produce serious visual artifacting.

The only fix that I have found, is to disable the vc4-fkms-v3d driver in the /boot/config.txt file (or by switching the GL Driver to "Legacy" mode in raspi-config), but unfortunately, this results in the loss of Hardware acceleration.

To reproduce:
This typically occurs when watching a video for more than 30 mins in the VLC media player application. Eventually the video will freeze and any attempt to interact with the application will cause the application to freeze or produce visual artifacting.

Expected behaviour:
Expected to use Hardware accelerated applications for an extended period of time, without causing applications freezing or produce visual artifacting.

Actual behaviour:
See "Bug Description" section above.

System:
Device Model -> Raspberry Pi 4 Model B (4GB)
OS -> Raspbian GNU/Linux 10 (buster) armv7l
Firmware version -> a51b488198a8c0360b93351682e7432d89d70411
Kernel version -> 4.19.66-v7l+

Logs:
[3912.566190] [drm:vc4_bo_create [vc4]] ERROR Failed to allocate from CMA:
[3912.573209] [drm] kernel: 5120kb BOs (1)
[3912.579505] [drm] V3D: 15116kb BOs (15)
[3912.585862] [drm] V3D shader: 120kb BOs (29)
[3912.592241] [drm] dumb: 48kb BOs (3)

[popcornmix]

Does increasing cma help? Add cma=512M to config.txt and test again.
EDIT: cmdline.txt not config.txt

[DeviceIoControl]

OK, I've adjusted my settings, and I am testing now. Thanks.
I'll comment again if the problem still persists!

[DeviceIoControl]

Does increasing cma help? Add cma=512M to config.txt and test again.

Well, it seems like the problem is still there, but its a different message this time. (still seems to be a CMA allocation error though.)

Here are the logs from dmesg.

dmesg:
[ 1374.350936] cma: cma_alloc: alloc failed, req-size: 765 pages, ret: -16
[ 1374.350943] [vc_sm_cma_ioctl_alloc]: dma_alloc_coherent alloc of 3133440 bytes failed
[ 1374.350947] [vc_sm_cma_ioctl_alloc]: something failed - cleanup. ret -12
[ 1375.019739] cma: cma_alloc: alloc failed, req-size: 765 pages, ret: -16
[ 1375.019750] [vc_sm_cma_ioctl_alloc]: dma_alloc_coherent alloc of 3133440 bytes failed
[ 1375.019754] [vc_sm_cma_ioctl_alloc]: something failed - cleanup. ret -12

[popcornmix]

What's the simplest way to reproduce?
You say you can see the problem in 30 minutes in VLC.
Is that just playing a single file?

e.g. with a clean raspbian buster image if you run vlc (and nothing else) and play a single video longer than 30m would you see this issue? Is there a freely available video file that exhibits this problem?

(this doesn't seem to be my experience - I've left VLC or chrome playing youtube videos overnight a number of times and it still seems happy in the morning - but perhaps there is something related to format a videos, or some other difference in our setups).

[DeviceIoControl]

In my case, it occurs more often when I've switched in and out of Fullscreen mode a couple of times in VLC Media player. (3 - 5 times).

And Google chrome seems to have this problem when viewing websites that use a lot of transitions and effects like: apple.com. (But most times Google chrome seems to "Aww, Snap" (crash) when using Hardware "intensive" websites).

Here is my config if needed.

**/boot/config.txt:**
# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan
disable_overscan=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
hdmi_group=1
hdmi_mode=16

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
dtparam=i2c_arm=on
#dtparam=i2s=on
dtparam=spi=on

# Uncomment this to enable the lirc-rpi module
#dtoverlay=lirc-rpi

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
dtoverlay=vc4-fkms-v3d
max_framebuffers=1
cma=512m

[all]
# NOOBS Auto-generated Settings:
hdmi_force_hotplug=1
gpu_mem=256
#hdmi_enable_4kp60=1
start_x=1

[popcornmix]

Ah sorry - cma=512m should be added to end of cmdline.txt, not config.txt.

[DeviceIoControl]

Alright, I've adjusted my settings again and I am testing now. Thanks.
I'll report back if the issues persist.

[DeviceIoControl]

Ah sorry - cma=512m should be added to end of cmdline.txt, not config.txt.

The issue still persists (though it did take much longer to occur this time), the same CMA allocation error as mentioned in the previous comment.

It happened in VLC Media player, while watching a 1080p60 video (mp4). Specifically when trying to rewind 30 seconds back in the video.

[JamesH65]

So, if you don't wind back, then the problem does not occur? Quite an important bit of information that.

[popcornmix]

Let us know if you find a sequence of operations that makes the issue happen repeatably and ideally quickly.

We can get the vlc/chromium guy to investigate this, but specific instructions to reproduce make it a lot more likely he'll be able to find and fix the problem.

What is your display resolution? Default skin on VLC?

[DeviceIoControl]

So, if you don't wind back, then the problem does not occur? Quite an important bit of information that.

It can still freeze even if you don't rewind VLC Media player.

Let us know if you find a sequence of operations that makes the issue happen repeatably and ideally quickly.

Will do! I will make sure to let you know as soon as possible, when I can do so.

EDIT: LOL, Found it! So, literally scrubbing the time-line on VLC Media player while the video is playing can cause this error. I am currently playing a 1920x1080p 60fps video (mp4). (My config has not changed since the last time I mentioned it btw.)

EDIT 2: So the "after-shock" of this error seems to cause other applications to crash when continuing to use them after that error has been produced.

In my case Chromium seems to "Aww, Snap" (crash) the tabs that are using Hardware accelerated rendering such as:

• Tabs that are playing video (YouTube).
• Tabs that use high-fidelity graphics / images (Apple.com)

NOTE: It doesn't crash it instantly, it only crashes when you try to interact with the website (after the error has been produced.) and will only stop crashing when you reboot the system (Restarting Chromium won't fix it).

EDIT 3: Making the Chromium tabs crash, seem to push more of the same error out when taking a look at the Kernel logs using "dmesg".

And I forgot to mention that VLC Media player becomes unresponsive when trying to close it.
Closing the window will make it "minimise" to system tray but, attempts to close it completely by right-clicking the VLC Media player icon and clicking "Quit" doesn't work. (Even attempting to close it using the "kill -9 " command in the terminal doesn't work either.)

What is your display resolution? Default skin on VLC?

Raspberry Pi is running @ 1920x1080p 60fps

VLC is using the "default skin" but, I am running the LXDE Desktop environment not the default RPD desktop environment. (but this issue has occurred multiple times before I changed my Desktop environment).

[sahib]

I seem to have a similar issue, but the context is a bit different. If this counts as hijacking: Sorry, please tell me to open a new issue in this case.

Context

We have a custom build UI that runs under wayland, or to be more exact under weston in fullscreen. The display has only a resolution of 720x480. The user interface renders its content using cairo (using the cairo-gl backend), pango (for font rendering) and librsvg (for displaying some vector graphics).

Issue

On some screens the UI suddenly freezes with very similar issues described by @DeviceIoControl.
The application log just tells me Draw call returned invalid argument. expect corruption. The screen will simply not update anymore, even a restart of weston won't fix the problem. Only thing that helps is a hard reboot. It is also not 100% when the issue happens, screens that are more memory intensive (with some SVG graphics) seem to trigger it more often. Here are two dmesg logs of the problem:

• https://pastebin.com/Pk7PQxHJ (using cma=128M@128M)
• https://pastebin.com/y4aSrn0n (using cma=256M@256M)
• (I'm currently testing with cma=512M@256M to see if it makes a difference)

Both logs contain also the kernel command line, plenty of stack traces and similar lines like:

Failed to allocate memory for tile binning: -12. You may need to enable CMA or give it more memory

and:

[drm:validate_tile_binning_config [vc4]] *ERROR* Failed to allocate binner memory: -12

System

• OS: Custom Linux built with yocto (Linux raspberrypi-cm3 4.14.112 #1 SMP Fri Aug 9 13:13:52 UTC 2019 armv7l GNU/Linux)
• Device Model: Compute Module 3+

Please tell me what other information I can provide to help to debug this.

[6by9]

@sahib Your issue is very different.
You would appear to be using the full vc4 3D driver (vc4-kms-v3d). The Pi4 has a different 3D block (v3d), and currently only supports the hybrid driver (vc4-fkms-v3d - note the "f").
Both cases would appear to be out of memory scenarios, but are going to be for different reasons.

4.14 is now very out of date, particularly with regard 3D and DRM/KMS driver changes. I would strongly recommend you update to 4.19, if not later.
3D also makes use of a userside library called Mesa, and I'd suggest you ensure that is relatively up to date (19.2.0 is now released and being used by Raspbian, although that is more for v3d fixes than vc4).

[sahib]

Thank you very much @6by9, I'm still trying to wrap my head around all of this.

Your issue is very different.

Sorry then about hijacking this issue. I will open a new issue once I have some new information.

Both cases would appear to be out of memory scenarios, but are going to be for different reasons.

Is increasing the amount of space for the CMA supposed to help? Also, can this be an issue in the application itself (i.e. using insane amounts of memory for unknown reasons) or is this more of a driver issue? I suppose the answer is "both"...

4.14 is now very out of date, particularly with regard 3D and DRM/KMS driver changes. I would strongly recommend you update to 4.19, if not later.

I can try and update once I get to it. Should be easily possible.

3D also makes use of a userside library called Mesa, and I'd suggest you ensure that is relatively up to date (19.2.0 is now released and being used by Raspbian, although that is more for v3d fixes than vc4).

This seems to be at 19.0.8. I can also try to update this.

[sahib]

Update on this: I updated the kernel to 4.19.71 and set the CMA memory to 256M (might try lower later). I did not update mesa. So far I have not been able to reproduce the bug. Thanks @6by9 👍

If this crops up again, I will open a new issue.