Incorrect Detection of VP9 Encoded Video

[wjs018]

Bug/Issue Description:

I have found that vp9 encoded videos are not analyzed correctly. Scenes are not detected at the right times and a portion of the video is not detected at all.

Required Information:
Provide the following information to assist with reporting the bug:

1. Provide a full copy of the command line options you are using:

I have made a proof-of-concept script to reproduce the result on my system. It downloads and analyzes the same video from youtube in two different formats. One copy has vp9 encoded video and the other has avc encoded video.

Note: youtube_dl is needed to run the script.

The relevant output is as follows:

.mkv file with vp9 codec:
fps: 23.976023976023978
frames read: 4311
Number of scenes: 169
.mkv file with avc codec:
fps: 23.976023976023978
frames read: 4557
Number of scenes: 172

2. Add -v debug -l BUG_REPORT.txt to the beginning of the command, then re-run PySceneDetect and attach the generated BUG_REPORT.txt file.

I am using the python interface, not the cli.

Expected Behavior:
A clear and concise description of what you expected to happen.

I expected the two copies of the same video with different encodings to result in the same detected scene breaks. Also, that the same number of frames are analyzed as both versions of the video are the same framerate.

Computing Environment:

• OS: Ubuntu 18.04
• Python Version: 3.6.6
• OpenCV Version: 3.4.2

Additional Information:
Add any other information you feel might be relevant to the bug/issue report but was not covered in one of the previous categories.

I am not sure whether this is a scenedetect issue or a larger OpenCV issue. Feel free to close this if it is out of your control. I have just been banging my head against this problem for a while until I finally figured out why some videos would work and some wouldn't.

[Breakthrough]

Hmm at first glance I don't think there's much I can do about this, using OpenCV currently for all video I/O. That being said, it may be worth trying to compile OpenCV yourself manually, but using a newer version of ffmpeg, and seeing if you get any different results after that.

If anyone else runs into this issue, feel free to share your experiences here as well. And of course, if anyone has any suggestions on how to approach this issue, any ideas are most welcome.

Thanks for the report, @wjs018.

[Breakthrough]

Added to the backlog for now. I'm especially interested to see if I can replicate what you're seeing, and if I can get that behavior to change with a different decoder or version (e.g. GStreamer vs ffmpeg).

Thanks for the test script as well, that will indeed come in handy!

[Breakthrough]

TODO: youtube-dl defaults to VP9 encoded videos occasionally, so bumping the priority of this issue. Need to re-test locally to try and replicate @wjs018's findings and begin root cause analysis.

[Breakthrough]

I'm hoping that #213 will help to resolve this by allowing more video backends for input. That being said, I'll make sure to at least reproduce locally and make sure to add a script/test case to the repo, but this probably won't make the v0.5.6 release. Will at least post an update before bumping the target release though.

[Breakthrough]

Am able to reproduce locally, thanks for the script. One thing that I noticed is the returned value of CAP_PROP_FOURCC is zero for some reason, which should allow for printing a warning. Very strange, however, that it can't read all the frames using the same backend.

Indeed this appears to be an OpenCV bug, however I wasn't able to find any bug reports for this in the OpenCV repo. Since OpenCV can't detect the right FourCC code, it makes me wonder if it might be due to how the opencv-python package is being compiled. I'll try building from source manually with VP9 support enabled and see if that helps.

While #213 will likely resolve this issue for most users, further investigation is still warranted to improve the error message. In the meantime, however, I'll add a warning whenever the FourCC code is detected to be zero, and indicate that the video may not be processed correctly. Thanks for the report!

[Breakthrough]

Added improved error messaging in this case before detection begins as per the commit referenced above. Thanks @wjs018!

[Breakthrough]

Marking as known issue, re-opening, and moving to v1.0, as this should also be resolved once #213 is completed.

[Breakthrough]

With the latest version of OpenCV, this seems to work ok now, but not with the one I have for the Windows build yet. With the latest v0.6 release, this should work at least by specifying --backend pyav. Will keep this for v1.0 when the warning might be removed.