Videos: Improve preview image generation depending on duration

[electricpollution]

Video thumbnails are currently created from the first frame hardcoded with the code -ss 00:00:00.001 -vframes 1. Any video that has a black first frame or anything not meaningful is not useful for previews.

Suggest defaulting to the first frame or having a parameter to customize how many seconds into the video a preview is created.

---

IssueHunt Summary

Backers (Total: $5.00)

• electricisfun ($5.00)

Become a backer now!

Or submit a pull request to get the deposits!

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

[graciousgrey]

Related discussion: #1235

[issuehunt-oss]

@electricisfun has funded $5.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[G2G2G2G]

Get video full length:
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 videofile.mkv

Get video black bars and remove them. Useful if someone has.. black bars encoded into the video. (python running a bash script 10 times and averaging it, so in Go you'd just do the same thing):

croplist = []
for x in range(10):
    times = str(random.randint(3,59))
    cropcheckcmd = "ffmpeg -ss 00:" + times + ":00 -i \"" + filename + "\" -t 1 -vf cropdetect=17:4:3000 -f null - 2>&1 | awk '/crop/ { print $NF }' | tail -1"
    cropcheck = subprocess.run([cropcheckcmd], stdout=subprocess.PIPE, shell=True).stdout.decode('utf-8').rstrip()
    print (cropcheck)
    croplist.append(cropcheck)
print (croplist)
cropmode = mode(croplist)

This "croplist" should work directly into ffmpeg as -vf "CROPLISTVARIABLEHERE" (including the quotes)
Note my 00:" + times + ":00 uses a random int 3-59 (cuz I use this for stripping out black bars in legal videos I acquire) but for you guys, you'd need to use the time you get in the previous FFPROBE command (first one I posted)
You can also run this for a set amount of time:
https://superuser.com/questions/810471/remove-mp4-video-top-and-bottom-black-bars-using-ffmpeg
However it does not work well for dark movies / videos. Some scenes are very dark and some are bright. SO I found randomly skipping around the video and looking at a frame is far better.
I've also been doing this for over a decade so it is well tested.

Last all that is needed is to pick a random time from 0 to the end of the original "duration" of the video.

Looks like what @graciousgrey guy linked they were already looking at a thumbnailer program though. That is a good project it's used in almost all of the linux file managers. I know he said he was trying to not use C++ stuff though so my above implementations you can cut out black bars + get full duration + pick a random timestamp to make a thumbnail out of while only calling FFMPEG. you can replace AWK with something in Go to parse the output instead.

[G2G2G2G]

Actually ignore that crap above you can extract ACTION frames (frames that have pixel movement, thus not black!) purely in ffmpeg
https://ffmpeg.org/ffmpeg-filters.html#select_002c-aselect

value between 0 and 1 to indicate a new scene; a low value reflects a low probability for the current frame to introduce a new scene, while a higher value means the current frame is more likely to be one (see the example below)

This is all you need for every video:
ffmpeg -i VIDEOFILE.mkv -vf "select=gt(scene\,0.2)" -frames:v 1 -vsync vfr thumbnail.jpg

the magic is here: scene\,0.2 this is generous here, a video file that does not have a lot of pixel shifts is going to get a thumbnail here. However the thumbnail may be in the beginning of the video and not a very good thumbnail.
If you want a great thumbnail, maybe make this a setting so people can say "yes I want better thumbnails" or whatever.
But you check less generously and if it fails, have a fallback to a more generous check, like this:

//great action shot
ffmpeg -i VIDEOFILE.mkv -vf "select=gt(scene\,0.8)" -frames:v 1 -vsync vfr thumbnail.jpg
//if it succeeded you are DONE!
ELSE IF IT FAILED:
//medium action shot
ffmpeg -i VIDEOFILE.mkv -vf "select=gt(scene\,0.5)" -frames:v 1 -vsync vfr thumbnail.jpg
//if it succeeded you are DONE!
ELSE IF IT FAILED:
//low action shot
ffmpeg -i VIDEOFILE.mkv -vf "select=gt(scene\,0.1)" -frames:v 1 -vsync vfr thumbnail.jpg
DONE

or something like that, this is what I'd personally do. However that is the only command you need. You can copy paste the above command and replace your thumbnail right now and it'll already work better!

[lastzero]

@G2G2G2G Any idea what the performance impact might be, if any?

[G2G2G2G]

@lastzero how fast is it now? I tested on a "hollywood movie" the 0.1 and 0.2 are almost instant
All of my videos are h264 as well so it's very fast decode. and it is ryzen 3700X processor.
Video is 1280x720 resolution too so pretty fast.

From my test I think yours (for me) would be around "902.35 millis" and vs the 0.2 which is "1.02 secs" I think that's well worth it.
(assuming millis in the linux time command is milliseconds so 902 is almost 1 second, so they basically take the same time)

THIS IS TAKING THE FIRST FRAME OF VIDEO, NO PROCESSING AT ALL BASICALLY, AND NO FILTER FLAGS! 
IT IS A "CONTROL" 
Executed in  902.35 millis    fish           external
   usr time   75.65 millis  344.00 micros   75.31 millis
   sys time   35.99 millis  147.00 micros   35.84 millis



0.2

Executed in    1.02 secs      fish           external
   usr time  361.55 millis  391.00 micros  361.16 millis
   sys time   41.46 millis  118.00 micros   41.34 millis


0.4

Executed in    1.43 secs    fish           external
   usr time    4.60 secs  379.00 micros    4.59 secs
   sys time    0.10 secs  112.00 micros    0.10 secs


0.6

Executed in    1.64 secs    fish           external
   usr time    8.74 secs  361.00 micros    8.73 secs
   sys time    0.21 secs  115.00 micros    0.21 secs


0.8

Executed in    3.88 secs    fish           external
   usr time   28.50 secs  365.00 micros   28.50 secs
   sys time    0.57 secs  129.00 micros    0.57 secs

For me, personally, 0.8 is worth it taking 4 seconds to process the frame it finds is only like 20 seconds into the video.. So if you have like a long video and it never finds one with a lot of pixels changing then it's gonna take a LONG time since it'd go through the whole video and fail.. and then try a lower number.

[G2G2G2G]

Technically there's no guarantee that even a 0.1 threshold would work if your video is extremely dark / black almost the entire thing. Somewhere I have videos of us exploring a cave that are terrible videos, if I can find them I'll test it but I guess we need a fallback to just grab the first frame as you do now, in the very worst case?

[lastzero]

Bottom line is that a fallback must be implemented and tested, so it's a bit more complicated than just replacing the current ffmpeg command?

[G2G2G2G]

Since someone can upload a video that is just all 1 color and never changes, yes. If we guaranteed people would upload actual videos then we can guarantee 0.1 would get a thumbnail. But we can't guarantee what people upload.

[G2G2G2G]

What needs to be done to add this? @lastzero I feel bad for @StianOby and @axllent #648 lol

I can write this in like 15 minutes. Where is a video thumb created which function? https://github.com/photoprism/photoprism/tree/develop/internal/thumb I cannot find it

I've tested scene,0.8 on thousands of random videos on imgur.com and none failed. My fear is a lightning video like: https://www.youtube.com/watch?v=Q8j5T3Qn5G0 will only work on 0.6:

ffmpeg -hide_banner -y -loglevel error -i A\ Lightning\ Strike\ at\ Night-Q8j5T3Qn5G0.mp4 -vf "select=gt(scene\,0.6)" -frames:v 1 -vsync vfr thumbnail.jpg

However terrible videos (like the one I uploaded that is entirely white the entire duration) won't find any movement.. obviously

test.mp4

Videos like this also don't really trigger any thumbnails due to how slow they move
https://www.youtube.com/watch?v=2Uj1A9AguFs
some do, some don't. It's based on if a bar jumps in 1 frame or not. Sometimes on ones like the "international exports" video, when the USSR split into russia etc, russia exported a ton of extra crap to other countries driving their bar from non existent to in the top 5 for 5 frames of the video, which triggers scene\,0.1) to detect it, but normally nothing will get any thumbnail thus needing a "first frame" fallback (which is easy to do, but just an example of why it's needed)

[Jarvid]

Hello,

my 2c as I just stumbled accross the black thumbnail (black screen in 1st second) problem:

Customization of the thumbnail generation is hardly worth the effort.
Every setting will work one some, but fail on other videos.

Either the thumbnail is custom, where the user can select a specific frame of the video in the GUI or upload a specific picture,
or
simple create a collage thumbnail.

Did it with a 2x2 layout to have big enough pictures even on a mobile.
Took the duration(video playback time) from the Mediafile metadata to get 4 frames evenly distributed on the source videos.
Only 4 lines in covert.go.
But Indexing is slower now and scales with the playback time of the video. Took 25 seconds for a 60 minutes video with 2giga bytes file size while the original approach was only 1 second.

I'm happy with the result though. Unless you happen to have a lot of collage pictures in your library, this change makes videos also more distinguishable from pictures aside from the little arrow icon.

[lastzero]

We need the full, first frame so it can be used as still image for Live Photos. Otherwise, the UI would have to load the entire video just to display the first frame. Note that Live Photos will only play when you hover over them with the mouse, for example in search results.

[Jarvid]

Excuse my way to write the same content in a new reply again, but I as orignally answered via E-Mail github does not allow to use markdown in reply.

I see, is this still limited to videos up to 3 seconds or intended to be used for all kind of videos in the Future?

I additionally now check for isLive(). If true it takes the first picture as before. A collage for a ~3 sec video is unlikely to produce any benefit anyways.
If false it creates the 2x2 tile thumbnail.
Not sure if this logic is to everyones liking or if this few lines even justify a pull request.

-edit-
Ok, isLive() is always false im my 2 seconds hvc test example, even tough it is correctly typed as live by photoprism.
Is this data populated later?
But aside from this thing:
I don't see any network traffic difference indicating it is loading the entire video for the first frame.
It behaves just as in an unmodified photoprism instance. Well with the current difference, the thumbnail is a collage and once you mouseover the clip starts just normal.
I've attached a video to show the current way it looks.

Desktop.2022.02.05.-.18.45.52.04.mp4

[lastzero]

Is this still limited to videos up to 3 seconds or intended to be used for all kind of videos in the Future?

The type is determined automatically based on the duration (and possibly other properties like the codec, see public source code), but it is possible to change it manually. So to avoid unnecessary complexity, it would be best to always have the same behavior - at least until we can pay an armada of developers to implement bells and whistles.

[lastzero]

With this commit, still images from live photos remain unchanged, while other videos may get better thumbnails, especially if the first few frames are just black or white. It's still not perfect, but hopefully an improvement.

[spyfly]

Commit 78151a7 is a great improvement overall, but for videos with longer intros this can be problematic.

Could you make this value configurable within the convert command?

[lastzero]

Generally yes, but I don't have time to add a config for it. What delay would be good instead of the 9 seconds? Note that this is dynamic and depends on the length of the video.

[spyfly]

I think going for something like 30 seconds to a few minutes would be a good idea

[lastzero]

How about this?

[spyfly]

How about this?

Thanks, looks good, will test when it is in the preview branch 👍