LoViRife

LoViRife stands for Long Video frame interpolation using RIFE NCNN.

Usecase / Goals

This is a simple practical approach, made for a specific environment, i.e.

• having some input video of 1-3 hours
• limited resources for temporary disk storage, RAM size, etc., electric power is costly
• the end result is supposed to be played on a PC (i.e. no trade-offs WRT console/TV/etc. capabilities wanted)
• few dependencies, i.e. local tool versions shall be used where possible (no installer silo, etc.)
• plain console usage (no UI requirement), i.e. easy to use on remote PC

Software Requirements

• ffmpeg
• rife-ncnn-vulkan
• Python 3.x

ffmpeg can be probably installed from any modern Linux distribution, like with apt-get install ffmpeg for Debian and Debian-based distros. rife-ncnn-vulkan should be runnable from the $PATH, e.g. if downloaded, it may be added to personal ~/bin folder as a symlink.

Hardware Requirements

• a modern CPU which can handle fast H264 encoding on many cores, like AMD Ryzen (3rd gen) or newer
• 16GB of RAM (more or less, depending on where temporary data is stored, see -t and -T options)
• a recent Vulkan compatible graphics card, see hardware requirements of rife-ncnn-vulkan
• an SSD for temporary data storage, with free space for roughly 5 times input size (depending on various options)

Usage

Call python3 lovirife.py -h for further information.

Implementation notes

The script converts in multiple steps. The input video is split into multiple segments (including uncompressed audio), the video data is converted to images, the temporary dump is converted with rife-ncnn-vulkan and its output is converted to H264/AVC video. The result chunks are later concatenated, the audio streams are recompressed with OPUS, and subtitle streams and other metadata from the original input are attached.

NOTE: The conversion steps requires dumping of (weakly compressed) frame images into the filesystem, which requires a lot of space for each chunk. Therefore, the slice length should typically be short, and the local temp data directory (i.e. /tmp or defined by TMPDIR env. var) should be fast, preferably in RAM (tmpfs).

For the input/output chunks, uncompressed audio (wave/PCM) is used, to get around audio/video drift issues. This is later compressed into OPUS audio (because who wants to mess around with ancient codecs?!).

Known issues

For some videos:

• the glue points of the chunks are showing stuttering or audio-skipping effects
• subtitles get shifted, apparently by the amount of seconds before the first subtitle appears

Future Work

• stop using the precompiled rife-ncnn-vulkan tool because the data handling is cumbersome and space consuming. The chunking approach can be probably replaced with video streams from and into ffmpeg processes. Instead, use internal processing, maybe with help of the rife-ncnn-vulkan-python bindings (once the build issues there are resolved)
• support other audio codecs and/or pass-through