Use keyframes as scrubbing thumbnails for Trickplay, providing a ~110x speed boost for Trickplay image generation

[n00mkrad]

Currently, trickplay uses the ffmpeg fps filter to extract scrubbing thumbnails.

This is slow as it needs to decode the entire video.
Simply extracting all keyframes instead is roughly 110 times faster (244 vs 2.2 FPS, tested on Win11, Ryzen 7950X3D Eco Mode)

Current ffmpeg command for 10s interval:

ffmpeg [...] -i [INPUT] -an -sn -vf "fps=0.10000000149011612,setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,240)/2)*2:trunc(ow/a/2)*2,format=yuv420p" -threads 8 -c:v mjpeg -qscale:v 2 -f image2 "[...]/%08d.jpg"

My proposal, 110x faster:

ffmpeg [...] -skip_frame nokey -i [INPUT] -an -sn -vf "fps=0.10000000149011612,setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*a)\,240)/2)*2:trunc(ow/a/2)*2,format=yuv420p" -threads 8 -c:v mjpeg -qscale:v 2 -vsync passthrough -f image2 "[...]/%08d.jpg"

Note the -skip_frame nokey before the input, which tells the decoder to only decode keyframes, and -vsync passthrough before the output, which avoids duplicated frames in the output as ffmpeg would otherwise duplicate frames to match the input frame rate.

I'm currently looking at the code and trying to get it to work, though I'm not familiar with the codebase and how the timestamps are stored, if at all.

[jellyfin-bot]

Hi, it seems like your issue report has the following item(s) that need to be addressed:

• This bug report was not filed using the issue template.

This is an automated message, currently under testing. Please file an issue here if you encounter any problems.

[gnattu]

You will need much more work for this to align to our current behaviors.

The key frame interval in videos is not a constant value, and we do support the arbitrary trickplay thumbnail interval settings. When this interval is smaller than the key frame interval then you need to align the key frame to the closet timing. If the key frame interval is larger than the user-specified interval you may need to duplicate this frame to match the required thumbnail count. In either case, this will generate slightly different thumbnails to our current behavior.

I’m personally fine with this as long as the generated thumbnails have correct timing

[nyanmisaka]

Only software decoders and hardware accelerators may benefit from this. QSV and CUVID cannot, as these decoders do not support skip_frame option.

https://github.com/FFmpeg/FFmpeg/blob/7e59c4f90885a9ffffb0a3f1d385b4eae3530529/libavcodec/h264_slice.c#L2128

[n00mkrad]

Only software decoders and hardware accelerators may benefit from this. QSV and CUVID cannot, as these decoders do not support skip_frame option.

https://github.com/FFmpeg/FFmpeg/blob/7e59c4f90885a9ffffb0a3f1d385b4eae3530529/libavcodec/h264_slice.c#L2128

I honestly fail how this is relevant as the speedup from doing software decoding on just keyframes is still much much faster than HW decoding all frames.
Ultimately it could just be optional.

[n00mkrad]

You will need much more work for this to align to our current behaviors.

The key frame interval in videos is not a constant value, and we do support the arbitrary trickplay thumbnail interval settings. When this interval is smaller than the key frame interval then you need to align the key frame to the closet timing. If the key frame interval is larger than the user-specified interval you may need to duplicate this frame to match the required thumbnail count. In either case, this will generate slightly different thumbnails to our current behavior.

I’m personally fine with this as long as the generated thumbnails have correct timing

Well, how does it work currently?
My proposal would certainly require timestamps to be stored, but I'm a bit confused what the current method is - Is the interval simply loaded from the settings? Because that'd mean that all Trickplay data would become worthless as soon as the option is changed.

[nyanmisaka]

Only software decoders and hardware accelerators may benefit from this. QSV and CUVID cannot, as these decoders do not support skip_frame option.

https://github.com/FFmpeg/FFmpeg/blob/7e59c4f90885a9ffffb0a3f1d385b4eae3530529/libavcodec/h264_slice.c#L2128

I honestly fail how this is relevant as the speedup from doing software decoding on just keyframes is still much much faster than HW decoding all frames.

Ultimately it could just be optional.

Since decoding keyframes is cheap, P and B frames will be ignored and thus save a lot of CPU cycles. Just like few people care about hardware decoding MJPEG. Also, the FPS filters don't need to wait for several frames before emitting a frame to the encoder.

[gnattu]

Well, how does it work currently? My proposal would certainly require timestamps to be stored,

Sorry each file doesn't have stored timestamp and probably hard to change it to unless you want to rewrite how trickplay works in a whole.

but I'm a bit confused what the current method is - Is the interval simply loaded from the settings? Because that'd mean that all Trickplay data would become worthless as soon as the option is changed.

It will create a database entry upon generation, so it is not loaded from the settings when reading images. It calculates the defined interval, puts multiple small images into a larger one, and then generates an HLS for those image sequences. Each of these larger images is called a tile, and a tile includes 100 thumbnails by default. The player would expect the interval or frame time between these 100 images to be equal, and for each tile, the server will calculate a tile duration using the thumbnail interval multiplied by the number of thumbnails in that tile.

Now you will know why it is a big problem that the B-frame interval is not a constant and you will need more works. You can save timestamp for the b-frame images and do you calculation to provide as close as possible alternate frame for the required timestamp(calculated using interval * image number) to make it work.

[n00mkrad]

I just tested something - We can just leave the fps filter in the command, that way it reliably outputs the correct timestamps.
Only downside is the potentially lower accuracy as the thumbnail might be a duped/dropped frame, though this should not be an issue unless intervals of <10s are used.

[gnattu]

unless intervals of <10s are used.

This is what's my main concern and this is a supported use case.

This should be an opt-in feature and have a configuration panel. Also we may need to make this option to be mutually exclusive with certain hardware decoders.

[n00mkrad]

I tried what I mentioned in the previous comment (adding -skip_frame nokey without removing the fps filter) and it works reliably. Blu-ray has a max 1s keyframe interval, streaming services around 2-4s, so I think accuracy is good enough.
But yes it should be an option considering it does have minor downsides.