Thanks @dHannasch for this idea and great work! I think it would make a great addition to the examples section as well. Maybe we can build out some visualization components for scikit-video that at least includes this gui to demonstrate quality assessment tools.

As for the diff, we need to be general enough to allow more than 1 type of diff, as is the case with spatio-temporal metrics like STRRED and MOVIE. Alternatively, maybe we can specialize the visualization for each quality assessment method. I agree that we should keep the diff separate from the video to reduce confusion.

Several years ago, I put together an instructional package of videos that we could use as further examples. Here is a link to that package if you (or anyone else is interested): https://live.ece.utexas.edu/research/VIP/VIP_materials.html

What is MOVIE? I don't see that in http://www.scikit-video.org/stable/measure.html.

http://www.scikit-video.org/stable/modules/generated/skvideo.measure.strred.html#skvideo.measure.strred is tricky. We could still have a plot on the bottom and just map each frame index (from 0 to T-1) to an appropriate-seeming index in the ST-RRED array, possibly just dividing by 2. It obviously doesn't make complete sense to match a single frame to one of the ST-RRED results, but it seems like the overall idea of how a video is more or less distorted at different points in time still applies. (Though if the user paused the video, they might find the the "current" score didn't appear to be justified by the actual distortion in that particular frame...we'd need to be careful to make clear what the plot was actually showing.) But I don't know whether we'd want to show the spatial score or the temporal score...we could show multiple scores, but it seems like the display would get too "busy".

Another complicating factor is that I think it would be best to avoid referencing specific differencing methods in the code of the display function itself (checking specifically for strred, and so forth). That seems like it would be unsustainable in the long term.

The videos you linked aren't currently in skvideo.datasets, are they? Should they be?

MOVIE is full-reference video quality metric that requires a large amount of compute, but seems to perform best on a number of datasets. A significant improvement on this method is FS-MOVIE, but it requires even more compute. It's not implemented in skvideo yet due to the amount of compute required (large number and size of convolutions). I can link the relevant papers if you are curious.

That seems like it would be unsustainable in the long term.

I agree! Another feature that'd be nice is multiple quality lines on the same plot. For example, measurements for a collection of distorted videos. Maybe we should create some wrappers for matplotlib that make video plots in a grid layout easier to make. This could easily get complicated fast of course.

The videos you linked aren't currently in skvideo.datasets, are they? Should they be?

The videos I linked are educational videos showing how methods work. The videos provided in skvideo.datasets allow repeatable real-world test cases. I'd like to avoid adding more, since that makes the package size larger.

Not sure what to do about multiple quality-plots just yet, but there's a minimal feature set at #131 --- just skipping over the plot if we have nothing for it, as in the case of ST-RRED.

Do you think it would be worth merging in with just the single plot as shown?
Or would you want the function to take a list of image-differencing functions rather than only one, in case we want to have multiple quality-plots later? (Honestly I'm skeptical that someone would want to clutter their window with multiple different measurements rather than pick whichever one they happen to like, but I can certainly see that if we might eventually want that then it would be good to set the function signature to take multiple image-differencing functions now.)