Videos: Improve preview image generation depending on duration

[electricpollution]

Video thumbnails are currently created from the first frame hardcoded with the code -ss 00:00:00.001 -vframes 1. Any video that has a black first frame or anything not meaningful is not useful for previews.

Suggest defaulting to the first frame or having a parameter to customize how many seconds into the video a preview is created.

---

IssueHunt Summary

Backers (Total: $5.00)

• electricisfun ($5.00)

Become a backer now!

Or submit a pull request to get the deposits!

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

[graciousgrey]

Related discussion: #1235

[issuehunt-oss]

@electricisfun has funded $5.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[G2G2G2G]

Get video full length:
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 videofile.mkv

Get video black bars and remove them. Useful if someone has.. black bars encoded into the video. (python running a bash script 10 times and averaging it, so in Go you'd just do the same thing):

croplist = []
for x in range(10):
    times = str(random.randint(3,59))
    cropcheckcmd = "ffmpeg -ss 00:" + times + ":00 -i \"" + filename + "\" -t 1 -vf cropdetect=17:4:3000 -f null - 2>&1 | awk '/crop/ { print $NF }' | tail -1"
    cropcheck = subprocess.run([cropcheckcmd], stdout=subprocess.PIPE, shell=True).stdout.decode('utf-8').rstrip()
    print (cropcheck)
    croplist.append(cropcheck)
print (croplist)
cropmode = mode(croplist)

This "croplist" should work directly into ffmpeg as -vf "CROPLISTVARIABLEHERE" (including the quotes)
Note my 00:" + times + ":00 uses a random int 3-59 (cuz I use this for stripping out black bars in legal videos I acquire) but for you guys, you'd need to use the time you get in the previous FFPROBE command (first one I posted)
You can also run this for a set amount of time:
https://superuser.com/questions/810471/remove-mp4-video-top-and-bottom-black-bars-using-ffmpeg
However it does not work well for dark movies / videos. Some scenes are very dark and some are bright. SO I found randomly skipping around the video and looking at a frame is far better.
I've also been doing this for over a decade so it is well tested.

Last all that is needed is to pick a random time from 0 to the end of the original "duration" of the video.

Looks like what @graciousgrey guy linked they were already looking at a thumbnailer program though. That is a good project it's used in almost all of the linux file managers. I know he said he was trying to not use C++ stuff though so my above implementations you can cut out black bars + get full duration + pick a random timestamp to make a thumbnail out of while only calling FFMPEG. you can replace AWK with something in Go to parse the output instead.

[G2G2G2G]

Actually ignore that crap above you can extract ACTION frames (frames that have pixel movement, thus not black!) purely in ffmpeg
https://ffmpeg.org/ffmpeg-filters.html#select_002c-aselect

value between 0 and 1 to indicate a new scene; a low value reflects a low probability for the current frame to introduce a new scene, while a higher value means the current frame is more likely to be one (see the example below)

This is all you need for every video:
ffmpeg -i VIDEOFILE.mkv -vf "select=gt(scene\,0.2)" -frames:v 1 -vsync vfr thumbnail.jpg

the magic is here: scene\,0.2 this is generous here, a video file that does not have a lot of pixel shifts is going to get a thumbnail here. However the thumbnail may be in the beginning of the video and not a very good thumbnail.
If you want a great thumbnail, maybe make this a setting so people can say "yes I want better thumbnails" or whatever.
But you check less generously and if it fails, have a fallback to a more generous check, like this:

//great action shot
ffmpeg -i VIDEOFILE.mkv -vf "select=gt(scene\,0.8)" -frames:v 1 -vsync vfr thumbnail.jpg
//if it succeeded you are DONE!
ELSE IF IT FAILED:
//medium action shot
ffmpeg -i VIDEOFILE.mkv -vf "select=gt(scene\,0.5)" -frames:v 1 -vsync vfr thumbnail.jpg
//if it succeeded you are DONE!
ELSE IF IT FAILED:
//low action shot
ffmpeg -i VIDEOFILE.mkv -vf "select=gt(scene\,0.1)" -frames:v 1 -vsync vfr thumbnail.jpg
DONE

or something like that, this is what I'd personally do. However that is the only command you need. You can copy paste the above command and replace your thumbnail right now and it'll already work better!

[lastzero]

@G2G2G2G Any idea what the performance impact might be, if any?

[G2G2G2G]

@lastzero how fast is it now? I tested on a "hollywood movie" the 0.1 and 0.2 are almost instant
All of my videos are h264 as well so it's very fast decode. and it is ryzen 3700X processor.
Video is 1280x720 resolution too so pretty fast.

From my test I think yours (for me) would be around "902.35 millis" and vs the 0.2 which is "1.02 secs" I think that's well worth it.
(assuming millis in the linux time command is milliseconds so 902 is almost 1 second, so they basically take the same time)

THIS IS TAKING THE FIRST FRAME OF VIDEO, NO PROCESSING AT ALL BASICALLY, AND NO FILTER FLAGS! 
IT IS A "CONTROL" 
Executed in  902.35 millis    fish           external
   usr time   75.65 millis  344.00 micros   75.31 millis
   sys time   35.99 millis  147.00 micros   35.84 millis



0.2

Executed in    1.02 secs      fish           external
   usr time  361.55 millis  391.00 micros  361.16 millis
   sys time   41.46 millis  118.00 micros   41.34 millis


0.4

Executed in    1.43 secs    fish           external
   usr time    4.60 secs  379.00 micros    4.59 secs
   sys time    0.10 secs  112.00 micros    0.10 secs


0.6

Executed in    1.64 secs    fish           external
   usr time    8.74 secs  361.00 micros    8.73 secs
   sys time    0.21 secs  115.00 micros    0.21 secs


0.8

Executed in    3.88 secs    fish           external
   usr time   28.50 secs  365.00 micros   28.50 secs
   sys time    0.57 secs  129.00 micros    0.57 secs

For me, personally, 0.8 is worth it taking 4 seconds to process the frame it finds is only like 20 seconds into the video.. So if you have like a long video and it never finds one with a lot of pixels changing then it's gonna take a LONG time since it'd go through the whole video and fail.. and then try a lower number.

[G2G2G2G]

Technically there's no guarantee that even a 0.1 threshold would work if your video is extremely dark / black almost the entire thing. Somewhere I have videos of us exploring a cave that are terrible videos, if I can find them I'll test it but I guess we need a fallback to just grab the first frame as you do now, in the very worst case?

[lastzero]

Bottom line is that a fallback must be implemented and tested, so it's a bit more complicated than just replacing the current ffmpeg command?

[G2G2G2G]

Since someone can upload a video that is just all 1 color and never changes, yes. If we guaranteed people would upload actual videos then we can guarantee 0.1 would get a thumbnail. But we can't guarantee what people upload.

[G2G2G2G]

What needs to be done to add this? @lastzero I feel bad for @StianOby and @axllent #648 lol

I can write this in like 15 minutes. Where is a video thumb created which function? https://github.com/photoprism/photoprism/tree/develop/internal/thumb I cannot find it

I've tested scene,0.8 on thousands of random videos on imgur.com and none failed. My fear is a lightning video like: https://www.youtube.com/watch?v=Q8j5T3Qn5G0 will only work on 0.6:

ffmpeg -hide_banner -y -loglevel error -i A\ Lightning\ Strike\ at\ Night-Q8j5T3Qn5G0.mp4 -vf "select=gt(scene\,0.6)" -frames:v 1 -vsync vfr thumbnail.jpg

However terrible videos (like the one I uploaded that is entirely white the entire duration) won't find any movement.. obviously

test.mp4

Videos like this also don't really trigger any thumbnails due to how slow they move
https://www.youtube.com/watch?v=2Uj1A9AguFs
some do, some don't. It's based on if a bar jumps in 1 frame or not. Sometimes on ones like the "international exports" video, when the USSR split into russia etc, russia exported a ton of extra crap to other countries driving their bar from non existent to in the top 5 for 5 frames of the video, which triggers scene\,0.1) to detect it, but normally nothing will get any thumbnail thus needing a "first frame" fallback (which is easy to do, but just an example of why it's needed)