Multi-row output (grid of thumbnails)

[aleksejrs]

nordlicht should be able to wrap its output into multiple rows, to show a grid of thumbnails (or a slitscan that is much more likely to show real frames on a long movie).

Main options:

• output image resolution;
• number of rows or columns.

When multiple styles are combined, there are two ways:

• wrap the combined result;
• wrap each style separately, giving each its own half of the picture.

Thumbnails should optionally have timestamps.

Some existing software (like Totem) also includes the file name, the resolution and the duration at the top of the output image. Maybe a formatting string could be used for that.

Maybe also (not sure it's possible without significantly more generation time):

• for short and simple videos or a really static picture, nordlicht may be allowed to collapse some frames and reduce the amount of rows/columns? Or specify the maximum amount of thumbnails per second, or frames per thumbnail. Then a smaller grid with a similar aspect ratio could be generated, with a lower total resolution or bigger thumbnails.

Existing examples

People have tried doing it using FFmpeg, but the commands do not generate just enough thumbnails to fit the given space like nordlicht does:

• https://ffmpeg.org/ffmpeg-filters.html#Examples-87 "produce 8x8 PNG tiles of all keyframes (-skip_frame nokey) in a movie"
• https://ffmpeg.org/pipermail/ffmpeg-user/2012-November/011101.html

Totem can "Create Screenshot Gallery", but the settings are "Screenshot width (in pixels)" and "Number of screenshots" (can be calculated automatically). The grid size is chosen by Totem.

There is also a shareware program that does it.

There is a Greasemonkey script for the old-style YouTube that puts the thumbnails (that do show times) into a grid you can click to seek.

[blinry]

That's a really cool suggestion! I appreciate the thought you already put into this! I also think that "wrapping" the tracks will be useful for the tumbnail and slitscan styles in long videos. I'd probably like some "automatic" mode which will try to determine a reasonable amount of rows based on the video's duration.

When a thumbnail is cut off halfway because the row ends, should it be continued on the next row? Or should a new thumbnail begin there? What's the more expected thing? Probably the second option, right?

As for wrapping two styles together or seperately, we'll just have to try that. I guess wrapping them together is probably more useful.

[aleksejrs]

When a thumbnail is cut off halfway because the row ends, should it be continued on the next row?

IMO, the original track should be calculated for each row to fit an integer number of thumbnails, so no thumbnail is cut off. Since all thumbnails are (I suppose) of the same size, it would be against readability to cut them or arrange in less than a perfect grid (not sure about this one though, if there is extra space).

[blinry]

As a first step, I added a --rows <n> option to the tool (and a matching function to the API). It will wrap the tracks as if they were cut into n parts and glued on top of each other. This seems the most natural extension to me. Here's an example:

And here is an extreme example, where the width is smaller than half of the thumbnail width, meaning the thumbnail has to be wrapped two times:

Here's my rationale for wrapping the thumbnails around like that: In the slitscan style, I wouldn't expect to reset the position of the scanline at the beginning of each row, so it makes sense to keep the thumbnails in the same format. An example with both styles:

nordlicht -r 3 -s thumbnails+slitscan -w 1000 -h 500 video.3gp

Or does it make more sense to reset both at the beginning of each row?

As specification of the images height, width, and number of rows determines the height of each individual row, I don't see how we can fit an integer number of thumbnails into the row yet. We could maybe cut off parts of the thumbnails' left and right parts?

What would your ideal command line interface look like to create a thumbnail grid?

This changes introduces a regression: If the nordlicht's height is not a multiple of rows*styles, it will a black part at the bottom. Handling this properly requires more care than before, but I'll look into this.

[aleksejrs]

I was inspired by the shareware program, which I won't name yet because it might prevent thinking. I'll only say that it has options for frames and spacing between thumbnails, and so I imagined separate thumbnails. I tried combined styles in nordlicht only yesterday.

Or does it make more sense to reset both at the beginning of each row?

I can't understand what the problem is.

• The thumbnail mode produces thumbnails of the normal aspect ratio, where each thumbnail shows only a single frame. It makes sense to put spacing between such thumbnails.
• Slitscan mode does the same (only the end is different: slitscan cannot show a full "screen" there), but its "thumbnails" contain vertical lines from many different frames. It makes less sense to put spacing between such thumbnails when not along the thumbnail mode, but on your picture, both modes have the left and right screen edges at the same points.

[aleksejrs]

As specification of the images height, width, and number of rows determines the height of each individual row, I don't see how we can fit an integer number of thumbnails into the row yet. We could maybe cut off parts of the thumbnails' left and right parts?

Shorten the track, so that there are fewer thumbnails. There will be margins on the sides.

This changes introduces a regression: If the nordlicht's height is not a multiple of rows*styles, it will a black part at the bottom. Handling this properly requires more care than before, but I'll look into this.

A black part at the bottom is not a problem: it is balanced out by the black part for the text at the top.

[aleksejrs]

What would your ideal command line interface look like to create a thumbnail grid?

Short options are easier to edit, but this is not something to be run once, so let us stick to the long forms for now.

• --grid-resolution=1024x768 # I could accidentally type 1024×768. This is the total image resolution, which includes what will be spent on the caption, margins and spacing between thumbnails and rows. So it should be separate from --height, because --height is the height of a track. Also, --width probably makes you think of the track length, and redefining it might make the code confusing.
• --caption="Name: %f\nDuration: %{duration}\nResolution: %{resolution}\nFilesize: %{size}\n" # print file name, video duration, resolution and file size (probably formatted; "find" has %s for "file size in bytes") on the top margin, each on its own line. I guess it's not specific to the grid form.
• --caption-font-size=32px # unless there is a good way to specify it in the format string. There are probably ways to specify the font in the same option.
• --grid-timestamps # enables timestamps on the thumbnails. Maybe should accept values to control whether they should be under/above the picture, partly on the picture, or where on the picture. Maybe also not specific to a grid.

Available space also depends on:

• font/size of the timestamps, if not inside the pictures.
• spacing between thumbnails.
• margin restrictions, if any.

Timestamps need a color suitable for the pictures. Maybe, if there are enough pixels, something universal could be used, like a border (works even with the small text Totem uses) or a shadow. Or they could just be in the space between rows.

[aleksejrs]

Media Player Classic (now MPC-HC), a GPL player for Windows, has supported it for many years.

• UI screenshot from 9 months ago — the settings are JPEG Quality, Image width (total, pixels), Rows, Columns.
• Output: https://yopiardinal.files.wordpress.com/2016/03/4.jpg
• Output (basically the same, different settings and video colors): https://youtu.be/Kc91pwVlSbM?t=51