Add Temporal Coherence to Single Image AI Upscaling Models in VapourSynth

Also known as temporal consistency, line wiggle fix, stabilization, deshimmering, temporal denoising, or temporal fix.
This runs on the CPU in parallel to the upscaling on the GPU. Intended for 2D animation.

Check out hddvddegogo's comparisons here and here.

Requirements

• fftw3.3 (required by mvtools)
  Windows: download and put .dll files in plugin folder
  Linux: apt install libfftw3-dev or compile from source
• mvtools (release v24 or newer)
• mvtools-sf (optional, only for tr > 6)
• motionmask
• fillborders
• zsmooth (release v0.14 or newer)
• retinex
• tcanny (release r14 or newer)
• vszip (optional, slight speed boost)

Setup

Put the vs_temporalfix.py file into your vapoursynth scripts folder.
Or install via pip: pip install -U git+https://github.com/pifroggi/vs_temporalfix.git

Usage

from vs_temporalfix import vs_temporalfix
clip = vs_temporalfix(clip, strength=400, tr=6, denoise=False, exclude=None, debug=False)

clip
Temporally unstable upscaled clip. Should have no black borders.

strength
Suppression strength of temporal inconsistencies. Higher means more aggressive. 400-700 works great in most cases.
The best way to finetune is to find a static scene and adjust till lines and details are stable.
Reduce if you get blending/ghosting on small movements, especially in dark or hazy scenes.

tr
Temporal radius sets the number of frames to average over.
Higher means more stable, especially on slow pans and zooms, but is slower. 6 works great in most cases.
The best way to finetune is to find a slow pan or zoom and adjust till lines and details are stable.

denoise (optional)
Removes grain and low frequency noise/flicker left over by the main processing step. Only enable if these issues actually exist! It risks to remove some details like every denoiser, but is useful if you're planning to denoise anyway and has the benefit of almost no performance impact compared to using an additional denoising filter.

exclude (optional)
Optionally exclude scenes with intended temporal inconsistencies, or in case this causes unexpected issues.
Example setting 3 scenes: exclude="[10 20] [600 900] [2000 2500]"
First number in the brackets is the first frame of the scene, the second number is the last frame (inclusive).

debug (optional)
Shows areas that will be left untouched in pink. This includes areas with high motion, scene changes and previously excluded scenes. May help while tuning parameters to see if the area is even affected.

Caution

• If vs_temporalfix is very slow for you, try adding core.max_cache_size = 15000 (15GB) to increase vapoursynth's frame cache. High tr and resolution or large filter scripts may need even more.

Tip

• Crop any black borders on the input clip, as those may cause ghosting on bright frames.
• There is a big drop in performance for tr > 6, due to switching from mvtools to mvtools-sf, which is slower.
• The plugin mvtools-sf release r9 and the r10 pre-release are both supported, but r9 is slightly faster for me.
• The plugin zsmooth requires a CPU with AVX2 support (roughly post 2014). If your CPU does not have support, remove zsmooth and replace it with the slightly slower fallbacks temporalmedian, ctmf, and rgvs.

Benchmarks

Hardware	Resolution	TR	Average FPS
Ryzen 5900X	1440x1080	6	~8 fps
Ryzen 5900X	2880x2160	6	~5 fps

Alternative Usage Options

Several projects integrated this script to simplify usage without the need for VapourSynth knowledge.

• Vapourkit (Windows only)
  Video filter and upscaling program with a nice and easy GUI. Can upscale and do the temporal fix at the same time, as well as other filters. This is the easiest way to use it, just switch to advanced mode and add the filter from the dropdown.
• py_temporalfix (Windows only)
  Simple portable command line tool to just do the temporal fix. Easy if you are okay with command line.
• Hybrid (Windows and Linux)
  Video filter toolbox with a GUI. Can be a bit overwhelming due to the amount of features, but can upscale and do the temporal fix at the same time, as well as many many more filters.
• VSGAN-tensorrt-docker (Windows and Linux)
  Command line AI upscale and interpolation toolbox. Rudimentary knowledge of Docker and VapourSynth is recommended, but the readme also explains it. Can upscale and do the temporal fix at the same time.