optimize output_frame_resize #6917

skrashevich · 2023-06-26T02:32:55Z

…threads used by OpenBLAS library in Dockerfile

netlify · 2023-06-26T02:32:59Z

✅ Deploy Preview for frigate-docs canceled.

Name	Link
🔨 Latest commit	`ebcdb63`
🔍 Latest deploy log	https://app.netlify.com/sites/frigate-docs/deploys/6563d8d74fee5d0008eed1f6

…mage.Resampling.NEAREST in copy_yuv_to_position function

skrashevich · 2023-06-26T03:30:22Z

I have ~x2 performance improvement with this modifications

NickM-27 · 2023-06-26T03:31:44Z

I have ~x2 performance improvement with this modifications

to be clear, that means ~ one half CPU usage? I'll try this in the morning and see how it goes in my setup

skrashevich · 2023-06-26T03:33:00Z

to be clear, that means ~ one half CPU usage? I'll try this in the morning and see how it goes in my setup

that means ~25% CPU utilisation by frigate.output process before, and ~12% after

in the morning

so relative... :-)

NickM-27 · 2023-06-26T03:43:26Z

So I ended up trying this now, wanted to clarify the above because I think #6853 got steered into the direction of the motion detection CPU usage which is separate from frigate.output. My motion detection CPU usage is unaffected by this PR.

In any case, this PR brought the frigate.output CPU usage in my setup from 20% of to 13% in top. I am just wondering if the openblas changes have affect that at all or if it is just the other changes, in which case maybe the openblas changes aren't needed?

skrashevich · 2023-06-26T04:35:41Z

My motion detection CPU usage is unaffected by this PR.

test within latest commit (7ee53df)

skrashevich · 2023-06-26T06:13:56Z

maybe the openblas changes aren't needed?

Given in mind my mistake in pushed Dockerfile (openblas and env vars were installed only in wheels step, but not in the final step) and your feedback on performance improvements - apparently, openblas does not significantly affect the result :)

NickM-27 · 2023-06-26T12:22:20Z

My motion detection CPU usage is unaffected by this PR.

test within latest commit (7ee53df)

This has caused my CPU usage to go up for each camera detect process by ~ 2 - 3 %

NickM-27 · 2023-06-26T12:23:40Z

Might be better for this PR to keep things focused on the frame_resize improvements which reduced CPU usage by frigate.output and potentially work on other changes in a separate PR

skrashevich · 2023-06-26T12:59:52Z

Might be better for this PR to keep things focused on the frame_resize improvements which reduced CPU usage by frigate.output and potentially work on other changes in a separate PR

You suggest rollback this PR to 4f9cf22, and move other commits to separate PR?

NickM-27 · 2023-06-26T13:01:25Z

Might be better for this PR to keep things focused on the frame_resize improvements which reduced CPU usage by frigate.output and potentially work on other changes in a separate PR

You suggest rollback this PR to 4f9cf22, and move other commits to separate PR?

Based on previous conversation I would also suggest removing openblas, but yeah I think it makes sense to keep it separate since the frigate.output is separate from the motion detector

…mber of threads used by OpenBLAS library in Dockerfile" This reverts commit 2bc977a.

skrashevich · 2023-06-26T13:20:57Z

Might be better for this PR to keep things focused on the frame_resize improvements which reduced CPU usage by frigate.output and potentially work on other changes in a separate PR

You suggest rollback this PR to 4f9cf22, and move other commits to separate PR?

Based on previous conversation I would also suggest removing openblas, but yeah I think it makes sense to keep it separate since the frigate.output is separate from the motion detector

What do you think about create configuration option for interpolation algorithm, used in internal calculations?

NickM-27 · 2023-06-26T13:22:50Z

I think it is too advanced for most users to understand / our docs to explain which one to use. I'm also not sure in which cases a user would want to change that

ccutrer · 2023-06-26T17:23:44Z

this decreased my output process CPU usage by at least half its previous usage. good work!

blakeblackshear · 2023-06-28T10:43:56Z

frigate/util.py

+        position,
+        resize_dim,
+        offset,
+        interpolation=Image.Resampling.NEAREST,


In my testing, this interpolation for a resize is faster, but the quality is really bad. I would be curious to see if there is still a performance improvement if using Image.Resampling.BILINEAR instead. I believe that is the equivalent to opencv's INTER_LINEAR which was used previously.

So this is specifically for birdseye, no? Running this PR, when I only have a single camera active (which is rare), I can notice a quality degradation. Though that's possibly because my birdseye resolution (1920x1080) is bigger than my detect resolution (1280x720) that drives it. When I have multiple cameras, I can't notice poor quality at all. Perhaps if we're resizing larger, we use bilinear, but if we're resizing smaller the faster/simpler algorithm can be used?

In my testing, this interpolation for a resize is faster, but the quality is really bad

describe your testing configuration

blakeblackshear · 2023-07-01T12:51:00Z

frigate/util.py

+    def assign_resized_frame(
+        destination_slice, source_slice, resize_dim, interpolation
+    ):
+        source_img = Image.fromarray(source_slice)


I think its likely the performance gain here has nothing to do with using PIL. Using opencv's cv2.INTER_NEAREST probably gets the same performance gain without casting back and forth from a numpy array.

some optimisations about casting numpy array made in a2b30d0

I would still prefer to use opencv's resize unless there is a good reason to switch since we use that in almost every other part of the code base.

but it give performance improvement... Especially with enhanced multiprocessing (#6986 and #6936)
It's allows to get rid of unnecessary context switching

Running with the current dev branch this PR provides higher CPU usage for me, opencv INTER_NEAREST provides a slight reduction in CPU usage

Current dev branch:

Current dev branch and using opencv INTER_NEAREST

This PR:

can you repeat this test with #7053 merged in? This can significantly affect the result.And will be wonderful to see py-spy output

I will try it, but if that PR made a difference it would lower the CPU usage for all cases not just the PIL case

of course. This PR provides 2 optimisations. Maybe, in some cases, an error in the second optimization neutralizes the benefits of the first one. need to check

… performance

skrashevich · 2023-07-06T13:04:53Z

@blakeblackshear this PR is related to #6986
I didn't test them separately

…ut-frame-resize

skrashevich · 2023-11-27T01:55:05Z

up?

skrashevich added 2 commits June 26, 2023 05:26

copy_yuv_to_position refactor

6e89b74

Add OPENBLAS_NUM_THREADS environment variable to limit the number of …

2bc977a

…threads used by OpenBLAS library in Dockerfile

Change interpolation method from OpenCV's INTER_NEAREST to Pillow's I…

4f9cf22

…mage.Resampling.NEAREST in copy_yuv_to_position function

skrashevich mentioned this pull request Jun 26, 2023

Performance optimisation: refactoring and interpolation method update #6891

Closed

skrashevich marked this pull request as ready for review June 26, 2023 03:29

skrashevich mentioned this pull request Jun 26, 2023

Performance improvement: add small delay to reduce CPU usage on empty queues #6892

Closed

skrashevich changed the title ~~openblas & optimize output_frame_resize~~ openblas & optimize output_frame_resize & motion detect & process_frames Jun 26, 2023

Revert "Add OPENBLAS_NUM_THREADS environment variable to limit the nu…

9de8f3f

…mber of threads used by OpenBLAS library in Dockerfile" This reverts commit 2bc977a.

skrashevich force-pushed the 230626-optimize-output-frame-resize branch from 7ee53df to 9de8f3f Compare June 26, 2023 13:04

skrashevich changed the title ~~openblas & optimize output_frame_resize & motion detect & process_frames~~ optimize output_frame_resize Jun 26, 2023

NickM-27 approved these changes Jun 26, 2023

View reviewed changes

blakeblackshear reviewed Jun 28, 2023

View reviewed changes

blakeblackshear reviewed Jul 1, 2023

View reviewed changes

Refactor copy_yuv_to_position function to use np.asarray for improved…

a2b30d0

… performance

skrashevich mentioned this pull request Jul 2, 2023

Performance: multiprocessing improvement: step 2 #6986

Merged

Merge branch 'dev' into 230626-optimize-output-frame-resize

a97c0c7

skrashevich added 2 commits July 6, 2023 16:09

isort

48d72b7

Merge remote-tracking branch 'upstream/dev' into 230626-optimize-outp…

2d00bd4

…ut-frame-resize

skrashevich force-pushed the 230626-optimize-output-frame-resize branch from 954e6c6 to 2d00bd4 Compare July 6, 2023 15:12

isort

a1c4918

github-actions bot added the stale label Aug 6, 2023

blakeblackshear added pinned and removed stale labels Aug 6, 2023

Merge branch 'dev' into 230626-optimize-output-frame-resize

ebcdb63

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

optimize output_frame_resize #6917

optimize output_frame_resize #6917

skrashevich commented Jun 26, 2023

netlify bot commented Jun 26, 2023 •

edited

skrashevich commented Jun 26, 2023

NickM-27 commented Jun 26, 2023

skrashevich commented Jun 26, 2023 •

edited

NickM-27 commented Jun 26, 2023

skrashevich commented Jun 26, 2023 •

edited

skrashevich commented Jun 26, 2023 •

edited

NickM-27 commented Jun 26, 2023

NickM-27 commented Jun 26, 2023

skrashevich commented Jun 26, 2023 •

edited

NickM-27 commented Jun 26, 2023

skrashevich commented Jun 26, 2023

NickM-27 commented Jun 26, 2023

ccutrer commented Jun 26, 2023

blakeblackshear Jun 28, 2023

ccutrer Jun 28, 2023

skrashevich Jul 2, 2023

blakeblackshear Jul 1, 2023

skrashevich Jul 2, 2023

blakeblackshear Jul 6, 2023

skrashevich Jul 6, 2023 •

edited

NickM-27 Jul 6, 2023 •

edited

skrashevich Jul 6, 2023

NickM-27 Jul 6, 2023

skrashevich Jul 6, 2023

skrashevich commented Jul 6, 2023

skrashevich commented Nov 27, 2023

optimize output_frame_resize #6917

Are you sure you want to change the base?

optimize output_frame_resize #6917

Conversation

skrashevich commented Jun 26, 2023

netlify bot commented Jun 26, 2023 • edited

✅ Deploy Preview for frigate-docs canceled.

skrashevich commented Jun 26, 2023

NickM-27 commented Jun 26, 2023

skrashevich commented Jun 26, 2023 • edited

NickM-27 commented Jun 26, 2023

skrashevich commented Jun 26, 2023 • edited

skrashevich commented Jun 26, 2023 • edited

NickM-27 commented Jun 26, 2023

NickM-27 commented Jun 26, 2023

skrashevich commented Jun 26, 2023 • edited

NickM-27 commented Jun 26, 2023

skrashevich commented Jun 26, 2023

NickM-27 commented Jun 26, 2023

ccutrer commented Jun 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skrashevich Jul 6, 2023 • edited

Choose a reason for hiding this comment

NickM-27 Jul 6, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skrashevich commented Jul 6, 2023

skrashevich commented Nov 27, 2023

netlify bot commented Jun 26, 2023 •

edited

skrashevich commented Jun 26, 2023 •

edited

skrashevich commented Jun 26, 2023 •

edited

skrashevich commented Jun 26, 2023 •

edited

skrashevich commented Jun 26, 2023 •

edited

skrashevich Jul 6, 2023 •

edited

NickM-27 Jul 6, 2023 •

edited