[Beta Support]: Performance issues on Meteor Lake with 0.18-beta #24083
Replies: 4 comments 24 replies
|
For starters it would be good to see comparison on 0.17, but the stats seem to clearly show that the majority of GPU and CPU usage is on object detection. |
|
Screen shots from 0.17.2: 
|
|
See my explanation, I appreciate you providing the information and I am looking to see what these changes might due to help on the object detection side. We have to work with objective data, not just jump to conclusions based on some general findings / only specific parts of the data. Even if we find it is not just object detection, we have to find out where the issue actually is to be able to make any changes. So far the GPU usage does not look to be meaningfully different, and without object detection the GPU is doing basically nothing (which is expected as decoding is very low effort). Despite the fact that the Panther lake GPU runs on 0.18, it may very well not be optimized as the driver version we pull in is still targeted at Intel B70 as the latest hardware, so there are likely many optimizations that would not be put into the driver yet by Intel. |
|
Thanks Nick. I'm dropping back to my production 0.17.2 container that just hums along barely noticed on my system. If there's something you want to try or test, please do let me know and I'd be happy to do so. I can easily spin the .18 containers back up on both boxes to test any changes. Cheers! |
|
It looks like there is a knob to improve this behavior in 0.18
|
|
Did want to clarify a couple of things in case you didn't look closely at my config - On my system, the GPU is being used for video decoding as well as downscaling the main streams down to 1280x720 for detection duty. Additionally, 2 of my 7 streams are fed to detection at full resolution as these particular cameras, due to either placement, crappy compression, or what I need them to detect, don't perform well enough at lower detection resolutions to pick up what I need. It took a couple of months of tweaking to dial this in, and it's been this way since 0.17.0. It really shouldn't matter as it's still the same config, but wanted to point that out in case you didn't pick up those details on your own. Regarding Panther Lake, using GPU-based object detection, it actually performed a little better than the Meteor Lake box using GPU-based object detection. This isn't surprising given the Panther Lake platform is considerably more powerful, but either way, the bottom status bar is painted red pretty much the entire time, they consume more power than neccessary, and the system fans are constantly running. It is great that inference times are a little lower though. I'm not sure how I can tweak those knobs you posted, but I'm happy to try them out! |
@mabnz - Way off topic, but how did you get your GPU stats to show up in Prometheus? I never could quite figure it out. |
|
More evidence using intel-gpu-top. Given that I've switched to NPU-based object detection, the GPU is just tasked with video processing, so utilization stays fairly steady state, only varying by 1-2%. VideoEnhance in particular is orders of magnitude higher. 0.17.2 0.18.0-beta3 |
|
Thanks for sharing, but this isn't quite the right reading of these numbers. The engine % is time residency, meaning how much of the sampling window the engine was busy at whatever frequency the GPU was running. At low clocks the same work occupies a much larger % of the bar. In this case, the GPU itself is drawing less on 0.18 (0.38 W vs 0.52 W), so the engines aren't doing several times more work; the work is being spread over more time at lower clocks. |
|
Not completely following. I ignore that bars, I was just looking at the percentages. If I add Video+VideoEnhance, the numbers match very closely with the running GPU Usage stat reported on Frigate. I watched these graphs for a solid 10 minutes on both .17 and .18. |
|
The bar is just showing the % visually, they represent the same thing. The % is total time active, it represents how long that engine was working, not how much work it was doing. In this case the graph shows VideoEnhance was active 50% of the time, but it was doing less work overall compared to 0.17 (as evidenced by lower GPU power usage, and slightly lower clock). Practically, this is saying in both cases the GPU is doing very little work overall. |
|
@NickM-27 I have a new important finding to share (I think it's important anyway :-) ). I reset all my streams to use VAAPI instead of QuickSync, and low and behold, and GPU utilization immediately dropped to below what I was eeing with 0.17. This was replicated on both Meteor Lake and Panther Lake. Additionally, power consumption dropped to roughly where it was with 0.17, the bottom Frigate status bar is no longer showing red GPU utilization, and the system fans are no longer constantly on. I won't go into forming a hypothesis for why changing to VAAPI helped so much, but I did want to report the finding for you to ponder. I always assumed QuickSync was "better", and "VAAPI" was old news so I standardized on it a good while back after a bunch of testing. This discovery now makes 0.18 a very viable candidate for me on either Panther Lake or Meteor Lake. Anyways, wanted to let you know. Thanks! |
|
interesting, yes generally quicksync has been better for newer gen Intel, most so with discrete GPUs but perhaps for iGPUs it is not as simple. Thanks for updating with that info. |
|
Hey all, update time. Using VAAPI instead of QuickSync has really changed the way the Meteor Lake system is behaving for the better. I'm now seeing similar performance to what I had in 0.17.2, and power consumption has dropped to what it was before. Additionally, I'm also running 0.18.-beta3 on Panther Lake, and as long as it's pure VAAPI, it's well-behaved there as well. This is Meteor Lake with NPU-based detection (my production 0.17.2 box). I ran QuickSync without issue on 0.17.2, but had to switch to VAAPI for 0.18 : This is Panter Lake with GPU-based detection and VAAPI: Seeing those kind of inference times on Panther Lake is very promising indeed. Given it's doing GPU-based detection, I'm still seeing occasional spikes to 100% GPU, but it's leaps and bound better than where it ran with QuickSync. I'm hoping Nick is still considering optimizing GPU-based detection to see where this ultimeately lands, and of course, given the massive positive impact NPU-based detection has on system resources, I'm chomping at the bit to get Panther Lake NPU support. It will make a fantastic Frigate host once it's all dailed in. |
|
We did merge the change, so the CPU usage should be reduce when using OpenVINO GPU detection |
|
Awesome! How about you get me a pre-release of Panther Lake NPU support?? ;-) I did dig into the 0.18.0 and 0.17.2 containers, and could see that the QuickSync drivers are completely different. Additional observation is that it's missing a symlink in 0.18 to libmfx-gen.so. I have no way of knowing if this is intentional, but wanted to bring it up anyway. `bob@ojochal:/opt/homeassistant/frigate-beta$ sudo docker run --rm --entrypoint sh ghcr.io/blakeblackshear/frigate:0.17.2 -c "ls -la /usr/lib/x86_64-linux-gnu/libmfx*" /usr/lib/x86_64-linux-gnu/libmfx-gen: /usr/lib/x86_64-linux-gnu/libmfx-gen: |
|
The NPU will take a lot more work as it requires a lot of other versions to move with it, that is something that we will have to see if it will be possible in 0.19, it may have to wait until we update the base image to Trixie which now looks like it is not ready to be done in 0.19 |
|
I'm curious about the massive drop in the size of the enctools.so driver (yeah, I know I have too much time on my hands). From what I've been able to find, that driver is directly responsible for QSV optimizations. I saw the discussion here #22866, which I assume is what prompted the QSV driver upgrade. Did Intel just massively refactor the QSV support for Battlemage? Thanks for indulging my curiousity :-). |
|
Another update - Given that I'm an OCD dog with a bone with too much time on his hands, I found the following PR #24007 and decided to build my own custom container of the dev branch + this PR. Low and behold: 
It's been running perfect for 24 hours now. Insanely low inference times, and has heavy detection activity. Seems Intel's December 2025 NPU driver update to include Panther Lake works pretty well. I'm going to leave it in "production" and monitor it for any weirdness, but so far so good! |
Uh oh!
There was an error while loading. Please reload this page.
Describe the problem you are having
I migrated my Core Ultra 5 frigate install today from 0.17.2 to 0.18.0-beta3. I immediately noticed that performance was much MUCH worse on 0.18 than on 0.17.
Long story short, I think something changed in .18 that has greatly impacted overall performance on my system. Running identical configs on both 0.17 and 0.18, and I'm routinely pegging GPU utilization, and CPU is higher now as well. I tried reconfiguring to use ffmpeg 7 to rule out ffmpeg 8, and it made little difference. The performance degradation with .18 is very apparent, not just with GPU, but CPU as well (monitoring with htop and intel_gpu_top). Another observation that illustrates the kind of performance hit is that power consumption on the Meteor Lake box nearly DOUBLED vs 0.17.2. Not great obviously. On 0.17.2, I rarely went over 5% CPU and 10% GPU, this is orders of magnitude higher.
Note that this same exact config on the same exact hardware with 0.17.2 has been running flawlessly since 0.17.0 came out. I haven't changed anything on my host machine, on my network, or on my cameras. Everything is 100% identical. I'm happy to test further if there are specific things that you all want me to try. I have separate containers for .17 and .18, so it's fairly easy for me to flip between the 2.
Note 2: In my "production" 0.17.2 install, I use the NPU for object detection rather than the GPU. For this comparison test, I'm using GPU object detection for both .17 and .18 as it better illustrates the issues with GPU and CPU utilization. I'm happy to flip both to NPU, but I'm also trying to migrate this to a Panther Lake platform which won't have NPU support until 0.19. Obviously, it's a lot more work for me to run the same extensive testing for both GPU and NPU detection, so I want to focus on GPU for now. I'm seeing similar behavior when I run this on my Panther Lake box (0.17.2 doesn't run on Panther Lake so can't compare to .17 there), but don't want to get into that right now.
Host system is Debian Trixie with all the latest patches applied.
Beta Version
0.18.0-beta3
Issue Category
Performance / Stability
Frigate config file
Relevant Frigate log output
Relevant go2rtc log output (if applicable)
No response
Install method
Docker Compose
docker-compose file or Docker CLI command
Operating system
Debian
CPU / GPU / Hardware
Intel Core Ultra 5 125H
Screenshots
Steps to reproduce
No response
Any other information that may be helpful
No response
All reactions