Improved scene detection #2710

master-of-zen · 2021-03-31T12:02:49Z

Goals and motivation

Current fast scene detection in rav1e relatively slow to compared methods of scene detection, and have proclivity to show false results or don't detect scene changes where they appear

This pr reworks fast scene detection algorithm, making it faster, better, and more accurate

Achieved goals are:

Faster decision making ( Both less and more efficient computations )
More accurate scene detection, by adjusting threshold based on previous frames
Frame downscale for faster decisions.

Example of adaptive threshold not cutting high motion segment which default static threshold method would cut

?  [SC-Detect] Frame 519: T=25.0 P=9.5 No cut
?  [SC-Detect] Frame 520: T=25.0 P=10.2 No cut
?  [SC-Detect] Frame 521: T=25.0 P=17.2 No cut

?  [SC-Detect] P: 29.0 [0.0, 1.8, 9.5, 10.2, 17.2] Cut: false
?  [SC-Detect] Frame 522: T=25.0 P=29.0 No cut

?  [SC-Detect] P: 33.6 [1.8, 9.5, 10.2, 17.2, 29.0] Cut: false
?  [SC-Detect] Frame 523: T=25.0 P=33.6 No cut

?  [SC-Detect] P: 34.0 [9.5, 10.2, 17.2, 29.0, 33.6] Cut: false
?  [SC-Detect] Frame 524: T=25.0 P=34.0 No cut

?  [SC-Detect] P: 26.4 [10.2, 17.2, 29.0, 33.6, 34.0] Cut: false
?  [SC-Detect] Frame 525: T=25.0 P=26.4 No cut

?  [SC-Detect] Frame 526: T=25.0 P=22.8 No cut
?  [SC-Detect] Frame 527: T=25.0 P=15.2 No cut
?  [SC-Detect] Frame 528: T=25.0 P=0.4 No cut

Speed comparison

2160p

new	old

720p

new	old

vibhoothi

Hi,

Thanks for the patch, 57 commits for this is nto easy to review, please squash the relevant commits, so reviewers can look it in better way

lu-zero

Please squash and rebase on top of the current tree.

shssoichiro · 2021-03-31T12:50:49Z

My primary concern is that this seems to remove the scene flash detection. This was used for both fast and normal (cost-based) detection and is based on what x264 does to try and avoid an extra keyframe for a very short (5-frame-or-less) scene, preferring to just put one keyframe after the flash. In the past, it also helped to alleviate some false positives with pans being detected as scenecuts using the old fast method--maybe that issue has been resolved with the new fast algorithm?

master-of-zen · 2021-03-31T13:49:43Z

@shssoichiro
I overlooked the flash detection for new algorithm and improvements should be made.

At this moment, new algorithm should handle pans just as fine as it use dequeue of scores to adjust threshold, which should also handle high motion / fade-in / fade-out

Making score dequeue bidirectional (compare score for current frame to scores before and after) should fix the issue flash detection, and should give free speed boost for all speeds as it reduce amount of computation

lu-zero · 2021-05-03T09:33:29Z

Can you please rebase this? :)

master-of-zen · 2021-05-03T18:54:45Z

It's wip) it will crash)

vibhoothi

I did not check the code in-depth, was seeing the general implementation, I have some suggestions before landing

The documentation for various things are good, but I felt the linebreaks are not consistent, sometimes happening after 20/30words, sometimes at 40/60, would be nice to be more consistent
The commit logs, would require a cleanup of commits to see by splitting into two/three, and giving a gist of the changes int eh commit messages, maybe could see the latest PR from @shssoichiro where he was giving a good explanation of things either in the commit log or the PR.

Apart from that It is good in shape generally. Thanks for the work.

@lu-zero would be nice if you can take a close-look in the rust part of APIs :)

vibhoothi · 2021-05-26T15:27:05Z

src/scenechange/mod.rs

-    )
-  }
+    // Initially fill score deque with forward frames
+    // ititiallization is different depending on frame set length


Suggested change

// ititiallization is different depending on frame set length

// Initialization is different depending on frame set length

Need to fix the typo+ maybe a rewording is a good idea, like merging both sentences?
Initially fill score deque with forward frames based on frame_set length

Fixed typos

vibhoothi · 2021-05-26T15:29:37Z

src/scenechange/mod.rs

+      debug!("[SC-score-deque]{:.0?}", self.score_deque);
+      self.score_deque.clear();
+    } else {
+      // Keep score deque 5 + lookahead_size frames


I wonder how did we reach 5+ lookahead_size, would be nice to add in commit message or here, whichever is better.

On seeing further code, could see the mention of 5 logic, would be better if it is mentioned here than later.

IIRC 5 was chosen because it's the size of a frame pyramid. I think at one point there was at least one comment mentioning that, although I haven't touched this code in a while so my memory is fuzzy. +1 for making sure we still have a comment mentioning that.

Improved comment

vibhoothi · 2021-05-26T15:39:19Z

src/scenechange/mod.rs

-        has_scenecut: delta >= threshold,
-      }
+  ) -> ScenecutData {
+    let frame2_ref2 = Arc::clone(&frame2);


Is there frame2_ref1 or frame2_ref ?
It maybe slightly confusing if we have 2 at the end if there is no frame2_ref or frame2_ref1. Could simply keep ref itself,
IIRC it was there earlier, itself, maybe @lu-zero could say if it is good idea to keep frame2_ref itself or not.

I don't know why it's done the way it is, I just refactored the code)

coveralls · 2021-05-27T21:05:13Z

Coverage decreased (-0.4%) to 83.521% when pulling ecfd502 on master-of-zen:scene-detection into bdee3b9 on xiph:master.

src/scenechange/mod.rs

vibhoothi

Further to add here to the PR from the #daala IRC discussion,
On checking the BD-Rate1, there seems regression of 8-14% BD-Rates on Speed-10, and also 1 frame displacement in the new scene-change mechanism2, on lower speed levels tested, the BD-Rate remains neutral, while giving a boost in encoding/decoding time generally.

The subsequent frames in the decoded output have +1 change throughout, this happens in the case of a scene-change.
Things to try out,

Examine the Output of new scene-change with different decoders, and extracting frame-by-frame(maybe with ffmpeg), and see the frames which shown the issue.
I did not check the code properly to find the +1 change of frame, it is happening to some clips which have scene-change at some levels, need to test at different levels/disabling some encoding features and seeing if it is an offset of some other encoding function,

master-of-zen · 2021-06-02T12:14:34Z

@vibhoothi I changed fast scenecut threshold back to what it is on master, can you rerun awcy s10?

lu-zero · 2021-07-09T21:28:15Z

Please rebase the whole thing.

master-of-zen · 2021-07-14T16:14:22Z

AWCY vimeo 10s corpus

speed 6

AWCY

PSNR Y	PSNR Cb	PSNR Cr	CIEDE2000	SSIM	MS-SSIM	PSNR-HVS Y	PSNR-HVS Cb	PSNR-HVS Cr	PSNR-HVS	VMAF	VMAF-NEG
0.1524	N/A	N/A	N/A	0.0814	0.1332	N/A	N/A	N/A	N/A	0.1599	0.1187

speed 9

AWCY

PSNR Y	PSNR Cb	PSNR Cr	CIEDE2000	SSIM	MS-SSIM	PSNR-HVS Y	PSNR-HVS Cb	PSNR-HVS Cr	PSNR-HVS	VMAF	VMAF-NEG
0.1763	N/A	N/A	N/A	0.1007	0.1433	N/A	N/A	N/A	N/A	0.2342	0.1804

speed 10

AWCY

PSNR Y	PSNR Cb	PSNR Cr	CIEDE2000	SSIM	MS-SSIM	PSNR-HVS Y	PSNR-HVS Cb	PSNR-HVS Cr	PSNR-HVS	VMAF	VMAF-NEG
0.2841	N/A	N/A	N/A	0.3555	0.3468	N/A	N/A	N/A	N/A	0.2160	0.2319

speed 10 with default scene detection and new algo

AWCY

PSNR Y	PSNR Cb	PSNR Cr	CIEDE2000	SSIM	MS-SSIM	PSNR-HVS Y	PSNR-HVS Cb	PSNR-HVS Cr	PSNR-HVS	VMAF	VMAF-NEG
-0.6573	N/A	N/A	N/A	-0.4079	-0.6386	N/A	N/A	N/A	N/A	-1.3326	-1.3189

lu-zero

It is fine for me if it is fine for @shssoichiro .

shssoichiro · 2021-07-14T17:31:39Z

It is good for me as well

Requested changes were made; we got an updated AWCY run as requested via IRC

Current fast scene detection in rav1e relatively slow to compared methods of scene detection, and have proclivity to show false results or don't detect scene changes where they appear This pr reworks fast scene detection algorithm, making it faster, better, and more accurate Achieved goals are: Faster decision making ( Both less and more efficient computations ) More accurate scene detection, by adjusting threshold based on previous frames Frame downscale for faster decisions.

vibhoothi previously requested changes Mar 31, 2021

View reviewed changes

lu-zero requested changes Mar 31, 2021

View reviewed changes

master-of-zen force-pushed the scene-detection branch 3 times, most recently from 0b0308b to 88d185a Compare April 6, 2021 19:32

master-of-zen force-pushed the scene-detection branch from 88d185a to 4b23924 Compare April 20, 2021 19:45

master-of-zen force-pushed the scene-detection branch from 629292f to 0828d1f Compare May 3, 2021 18:53

master-of-zen force-pushed the scene-detection branch from 50759ec to a653e4c Compare May 7, 2021 20:55

master-of-zen changed the title ~~Improved fast scene detection~~ Improved scene detection May 14, 2021

master-of-zen force-pushed the scene-detection branch 3 times, most recently from 4d1b7de to fbff89b Compare May 25, 2021 07:54

master-of-zen force-pushed the scene-detection branch 2 times, most recently from 809fea8 to 3984fa8 Compare May 26, 2021 15:16

barrbrain requested review from lu-zero and vibhoothi May 26, 2021 15:18

vibhoothi reviewed May 26, 2021

View reviewed changes

master-of-zen force-pushed the scene-detection branch 2 times, most recently from aa10a8c to 3097ce2 Compare May 27, 2021 20:03

master-of-zen force-pushed the scene-detection branch from 3097ce2 to 9594180 Compare May 28, 2021 18:06

barrbrain reviewed May 28, 2021

View reviewed changes

src/scenechange/mod.rs Outdated Show resolved Hide resolved

lu-zero reviewed May 29, 2021

View reviewed changes

src/scenechange/mod.rs Show resolved Hide resolved

master-of-zen force-pushed the scene-detection branch from 9594180 to 9354fef Compare May 29, 2021 17:54

barrbrain reviewed May 31, 2021

View reviewed changes

src/scenechange/mod.rs Outdated Show resolved Hide resolved

vibhoothi reviewed Jun 1, 2021

View reviewed changes

master-of-zen force-pushed the scene-detection branch 2 times, most recently from 183e56a to 2745c52 Compare June 2, 2021 12:13

master-of-zen force-pushed the scene-detection branch from 2745c52 to 4fca2a0 Compare June 5, 2021 14:33

shssoichiro mentioned this pull request Jul 9, 2021

[Meta] Parity with aomenc #2759

Open

19 tasks

master-of-zen force-pushed the scene-detection branch from 44bfce4 to b4d5b01 Compare July 10, 2021 18:15

Improve scene detection

ecfd502

master-of-zen force-pushed the scene-detection branch from b4d5b01 to ecfd502 Compare July 14, 2021 16:02

lu-zero approved these changes Jul 14, 2021

View reviewed changes

shssoichiro merged commit 7970d35 into xiph:master Jul 14, 2021

master-of-zen deleted the scene-detection branch July 15, 2021 00:11

vibhoothi mentioned this pull request Aug 26, 2021

Non-deterministic encodes on Speed 10 on fast-scene detection #2781

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved scene detection #2710

Improved scene detection #2710

master-of-zen commented Mar 31, 2021

vibhoothi left a comment

lu-zero left a comment

shssoichiro commented Mar 31, 2021 •

edited

Loading

master-of-zen commented Mar 31, 2021

lu-zero commented May 3, 2021

master-of-zen commented May 3, 2021

vibhoothi left a comment

vibhoothi May 26, 2021

master-of-zen May 29, 2021

vibhoothi May 26, 2021

shssoichiro May 26, 2021 •

edited

Loading

master-of-zen May 29, 2021

vibhoothi May 26, 2021

master-of-zen May 29, 2021

coveralls commented May 27, 2021 •

edited

Loading

vibhoothi left a comment

master-of-zen commented Jun 2, 2021

lu-zero commented Jul 9, 2021

master-of-zen commented Jul 14, 2021 •

edited by vibhoothi

Loading

lu-zero left a comment

shssoichiro commented Jul 14, 2021

	// ititiallization is different depending on frame set length
	// Initialization is different depending on frame set length

Improved scene detection #2710

Improved scene detection #2710

Conversation

master-of-zen commented Mar 31, 2021

Goals and motivation

Example of adaptive threshold not cutting high motion segment which default static threshold method would cut

Speed comparison

2160p

720p

vibhoothi left a comment

Choose a reason for hiding this comment

lu-zero left a comment

Choose a reason for hiding this comment

shssoichiro commented Mar 31, 2021 • edited Loading

master-of-zen commented Mar 31, 2021

lu-zero commented May 3, 2021

master-of-zen commented May 3, 2021

vibhoothi left a comment

Choose a reason for hiding this comment

vibhoothi May 26, 2021

Choose a reason for hiding this comment

master-of-zen May 29, 2021

Choose a reason for hiding this comment

vibhoothi May 26, 2021

Choose a reason for hiding this comment

shssoichiro May 26, 2021 • edited Loading

Choose a reason for hiding this comment

master-of-zen May 29, 2021

Choose a reason for hiding this comment

vibhoothi May 26, 2021

Choose a reason for hiding this comment

master-of-zen May 29, 2021

Choose a reason for hiding this comment

coveralls commented May 27, 2021 • edited Loading

vibhoothi left a comment

Choose a reason for hiding this comment

master-of-zen commented Jun 2, 2021

lu-zero commented Jul 9, 2021

master-of-zen commented Jul 14, 2021 • edited by vibhoothi Loading

AWCY vimeo 10s corpus

speed 6

speed 9

speed 10

speed 10 with default scene detection and new algo

lu-zero left a comment

Choose a reason for hiding this comment

shssoichiro commented Jul 14, 2021

shssoichiro commented Mar 31, 2021 •

edited

Loading

shssoichiro May 26, 2021 •

edited

Loading

coveralls commented May 27, 2021 •

edited

Loading

master-of-zen commented Jul 14, 2021 •

edited by vibhoothi

Loading