Poor GPU performance with 6.1.32

[dmanlfc]

Describe the bug

We are seeing major graphical performance issues with Kernel 6.1.32.

dmesg is full of v3d errors like so - v3d fec00000.gpu: MMU error from client CLE (4) at 0x9841000, pte invalid

Both OpenGL & Vulkan is severely impacted by ~50%

Rolling back to Kernel 6.1.25 fixes the issue.

Steps to reproduce the behaviour

Kernel 6.1.32
Firmware 6.1.32
Mesa 23.1.1

Start a graphically intensive game. My test were with Duckstation emulator with OpenGL & Vulkan.

Device (s)

Raspberry Pi 4 Mod. B

System

Batocera Linux (v37 in development)

Kernel 6.1.32
Firmware 6.1.32
Mesa 23.1.1

Logs

No response

Additional context

Updated the bootloader, no difference.

[pelwell]

How much RAM does your test system have? If 8GB, does adding total_mem=4096 to config.txt get rid of the errors?

[pelwell]

Also:

1. Is this with dtoverlay=vc4-kms-v3d in config.txt?
2. What applications are active when you see the errors?

[dmanlfc]

1. yes - dtoverlay=vc4-kms-v3d-pi4
2. EmulationStation and then Duckstation is part of the tests I did

Earlier question - 8GB. I will need to re-test as I reverted.

[pelwell]

3. When you say "full of", what's the approximate rate? There has always been the possibility of a few of these, attributed to some over-eager pre-reading by V3D hardware. At one point I submitted a patch to mesa to add some additional run-off padding, but clearly a bit more is needed.

[pelwell]

Using Raspberry Pi OS, I'm not seeing any obvious difference in performance between 6.1.25 and 6.1.32 on an 8GB Pi 4.

Do make sure you are updating the DTB files as well as the kernel image and modules.

[dmanlfc]

3. When you say "full of", what's the approximate rate? There has always been the possibility of a few of these, attributed to some over-eager pre-reading by V3D hardware. At one point I submitted a patch to mesa to add some additional run-off padding, but clearly a bit more is needed.

Every 10 seconds or so, I would have to run it back up to give you something more precise.

[dmanlfc]

Using Raspberry Pi OS, I'm not seeing any obvious difference in performance between 6.1.25 and 6.1.32 on an 8GB Pi 4.

Do make sure you are updating the DTB files as well as the kernel image and modules.

Yes all modules & DTB's are updated.

How are you stressing / measuring?
I can see a big difference with Mangohud showing the fps in the duckstation example.

[pelwell]

How are you stressing / measuring?

Chromium and glxgears.

I can see a big difference with Mangohud showing the fps in the duckstation example.

What's the easiest way for me to get that installed?

[dmanlfc]

No idea, we build our own OS & compile our own packages via Buildroot.
I would assume you'd be able to get debian packages of some sort...

[pelwell]

Do you see a variation in the performance of:

$ vblank_mode=0 mangohud glxheads

? I get around 680fps with both kernels.

[dmanlfc]

@pelwell please find some videos on this google share below showing the problem

https://drive.google.com/drive/folders/1uyexKCZkg8zqQ5MzN88CHXDK_5CtFVNP?usp=sharing

Also in that share is the version of the OS built with the 6.1.32 kernel to test for yourself.
Note: Game roms & BIOS you will have to source yourself.

On a positive note, the move to Mesa 23.1.2 has removed the dmesg spam.

[pelwell]

Thanks, Martin. I've grabbed the downloads and watched the movies - the difference is really apparent. However, the installed Batocera doesn't include many games and I've not seen any evidence of DuckStation support - perhaps it's waiting for PSX ROMs before displaying that section. My attempt at installing a test ISO and some BIOS ROMs has so far been unsuccessful.

[pelwell]

Did you get anywhere with total_mem=4096? The working theory is that your display buffers are ending up in the upper 4GB, resulting in each one being copied before being displayed. If we can establish that that is the problem we can focus on understanding how the kernel changes have caused it and whether we can improve things.

[dmanlfc]

There is a lovely wiki here - https://wiki.batocera.org/
But essentially, get an ISO, drop it into the /userdata/roms/psx folder
Put the PS1 bios in the /userdata/bios folder

From the ES menu, update the Games List - you'll then see the Playstation menu item & associated options.