Flicker Reduction

[beosliege]

Let's discuss what the 'Flicker Reduction' option in the GUI will do.

Current values are: 0.99, 0.9, 0.75, 0.50
What does this option do?
When should it be used?
Is this a feature of another project I could read about, or is it something built-in and unique to iw3?

Thanks Everyone!

[nagadomi]

Flicker Reduction is --ema-normalize option on CLI.
A simple method to stabilize min-max normalization. Not an image content-based anti-flicker.

Background: iw3 applies min-max normalization to depth as a preprocessing step.

Min-max normalization: the minimum value of that depth gets transformed into a 0.0, the maximum value gets transformed into a 1.0, and every other value gets transformed into a decimal between 0.0 and 1.0.

min_value = depth.min()
max_value = depth.max()
normalized_depth = (depth - min_value) / (max_value - min_value)

In video processing, in addition to model accuracy, this min-max normalization can cause depth flicker.
Because the maximum and minimum depth values for each video frame are particularly unstable, so normalizing with them will make everything unstable and cause flickering.

With --ema-normalize option, instead of using the maximum and minimum values directly, use their moving averages (smoothed value).

min_value = depth.min()
max_value = depth.max()
ema_min_value = 0.9 * ema_min_value + (1.0 - 0.9) * min_value
ema_max_value = 0.9 * ema_max_value + (1.0 - 0.9) * max_value
normalized_depth = (depth - ema_min_value) / (ema_max_value - ema_min_value)
normalized_depth = clamp(normalized_depth, min=0, max=1)

where 0.9 is the smoothing factor.
Here is a comparison, left: without EMA min-max normalization, right: with EMA min-max normalization.
In the left image, the sky can be seen flickering.

comp.mp4

(src: https://www.pexels.com/video/a-woman-standing-on-the-boat-8515879/
I had a hard time finding good examples. Since this problems do not occur often)

Here is a plot of the maximum value of the frame sequence. raw_max is the raw value (without --ema-normalize).

0.99 is the smoothest, but shows a large deviation from the actual value.
Also, 0.9 shows 10-20 frames delay. 0.99, few seconds delay will occur.
If the setting value is too large, visual transitions of 3D depth may occur when changing scenes.

Personally, I do not see much effect of this option on the SBS video. So I don't really recommend it.
You can use this option if you are concerned about depth flicker.

[beosliege]

Thank you - This is very informative!

I found that when using ZoeD_NK, enabling flicker reduction would cause the video to "jump" left or right a few pixels just for a split-second. It is quite noticeable (even though it happens very quickly). It could even possibly cause the user some nausea. I did not notice it happening with other models on the same video clip. If you would like to see this, I will prepare 10-second clips to demonstrate it. Just let me know.

I agree with you that I see little reason to use flicker reduction. At least so far, my results are very good without it, and my eye has not noticed any depth changes frame-to-frame in the same scene.

[nagadomi]

I added this feature when iw3 only had ZoeDepth, so this feature is intended for ZoeDepth output.
ZoeDepth output sometimes very flickering on depth-only videos, but it is not very visible on SBS Video.

Depth flicker is disparity(parallax) flicker on SBS video that causes the image to vibrate to the left or right.
If Flicker Reduction causes a problem, it could be a case of,
when the flicker is too fast, it is not visible, but when it is smoothed out (i.e., slowed down) by Flicker Reduction, it may become visible.

Flicker is less visible when the video is moving, but is more visible in the video from static camera.
In the case of anime and CG, flicker is more visible because they sometimes motion with a low FPS than the video FPS.

Also I have not tested ZoeD_K, ZoeD_NK, ZoeD_Any_K much.
They are made available only for fairness, but I may remove it from options on GUI in the future.