Seeking through long videos is slow

[briandwagner]

Hello! First of all, thank you so much for all of your work on this. This program has saved my tail multiple times. I've recently run into a problem and I don't know if it would be fixable or not.

I'm working with some large (multiple hours), lossless, FFV1 AVI files, and I'm wanting to detect some scenes only at the end of the clip. Currently, I'm using the time --start command to start PySceneDetect at the last couple minutes of the clip, and I'm letting it run from there. However, PySceneDetect takes a long time just to find the time code within my file before it actually starts detecting scenes. When I run it with -v debug, it hangs on DEBUG: context.process_input(): Processing input... for several minutes. Once it finishes, it moves on to detect the scenes as normal.

I understand that seeking specific time code in a massive file is going to take a long time no matter what, but, I think in cases where a user only wants to detect scenes at the end of a file, this can be drastically sped up by reading the video file in reverse.

This is what I envision:

Lets say I want to look for scenes only in the last 5 minutes of a 4 hour lossless FFV1 clip and the clip is 200 GB. Currently, if I use the time --start 03:55:00 command, PySceneDetect reads through the whole file until it finds that time code, and then it starts running. This means it's reading almost the full 200 GB from my drive. This takes a very long time. Instead, I would like to be able to use a command such as time --reverse --duration 00:05:00. The --reverse function would cause PySceneDetect to jump directly to the end of the video stream and start playback backwards and then detect scenes as normal. In this example, it would start at 04:00:00, then go to 03:59:59, etc.

As for the results, it could display them in multiple ways. Let's say that there is a scene that goes from 03:59:50 until 04:00:00. Since it's running in reverse, PySceneDetect would find this scene as starting at 04:00:00 and ending at 03:59:50. When it lists scenes out, it of course could display it this way, or I could use list-scenes --reversecorrection to flip it back to starting at 03:59:50 and ending at 04:00:00 exactly as PySceneDetect does now if I run it with the normal time --start command.

Media Examples:
Unfortunately, I can't just share a massive lossless video file, but any file can be transcoded to FFV1 with ffmpeg by running ffmpeg -i videofile.mp4 -c:v ffv1 -level 3 -g 1 -threads THREADCOUNT -c:a pcm_s16le output.avi. Obviously transcoding a lossy file to lossless would be pointless in the real world, but it should be sufficient for testing.

Alternative Solutions:

Currently, running with time --start 03:55:00 achieves the same output, but the time it takes PySceneDetect to read through the file is too long.

Starting with smaller video files should help speed things along from a drive reading stand point, but in the video archival world (where I work) or in the broadcast world, starting from a non-lossless source isn't always an option. I am using FFV1 as my codec because it losslessly compresses better than any other lossless video codec. My files are as small as they can get.

I tried using ffmpeg to reverse the file and export it as a lossy h264 file to act as a proxy for PySceneDetect to read from, but even with NVENC and the fastest settings it allows for, ffmpeg still has to read the whole lossless file to transcode it.

I'm terribly sorry if I'm asking for the impossible. I know very little about programming as I am a video engineer, not a computer engineer. Thank you so much again for all of your work on this project. The passion of the open source community just makes my heart sing.

[Breakthrough]

Hi @briandwagner;

Unfortunately you can't decode video files in reverse, due to the way video compression works. That being said, I do acknowledge that video seeking in PySceneDetect is very inefficient currently. The underlying video seeking method provided by OpenCV does not reliably work in all instances, thus a slow seek method was introduced.

The first solution I can see is allowing use of both fast and slow seeking with a command line switch, noting that the fast seek may be inaccurate. This is something I can add to the following release of PySceneDetect.

The second solution involves much more work to achieve fast and accurate seeking, but is where this project is eventually headed - I want to re-implement much of the core PySceneDetect video processing functionality in C++, since much the functionality is also shared with another Python project of mine, DVR-Scan. Essentially it would be a low-level scene detection library in C++ that PySceneDetect would use, to allow significantly faster processing speeds, and possibly using ffmpeg directly for frame decoding to allow for both fast and accurate seeking.

Thanks for the report.

P.S. I've also updated the title to better reflect how to implement a solution to solve this issue, please feel free to modify it further if you disagree with the change.

[briandwagner]

Hello! Sorry for not responding sooner. I don't use GitHub too often.

The fast vs slow seeking thing reminds me a lot of FFmpeg's input and output seeking methods. Input seeking is instant, but results in all time code used in the filtergraph to be relative to the the point in which the video was seeked to. IE, if you seek to 10 seconds in via input seeking, the 15 second point of the source video becomes the 5 second point; the 60 second point would become the 50 second point, etc. By contrast, there is output seeking, which decodes the whole video, but keeps time code "true". It sounds like PySceneDetect is using something similar to FFmpeg's output seeking for decoding.

Using FFmpeg for decoding would only be faster if you are using it's ability to input seek. I've never had accuracy issues with this, but I suppose it's possible. I think if you seek to a point between two I-frames ffmpeg will just back up to the last I-frame and then decode frame by frame to your desired point and then delete everything it decoded up until the desired frame itself. That's a kind of a complicated process, so I can see how that might be inaccurate from time to time, depending on how large the file's GOP is. If you are going to use FFmpeg input seeking to decode, Pyscenedetect would need to then calculate the actual, proper time codes by adding them to the time code given to FFmpeg for input seeking. To go with the example above, giving ffmpeg the time code 00:00:10 to seek to would result in ffmpeg treating 00:00:15 as 00:00:05. Pyscenedetect would need to add 00:00:10 to all of it's output to give actual meaningful timecodes.

Using FFmpeg to decode would open up a lot of possibilities for hardware acceleration, which really excites me. I routinely use NVDEC for extremely rapid decoding of H.264. Input seeking, and then decoding with NVDEC and then using ffmpeg to stream copy to the new, divided video files would result in extremely fast, non transcoded H.264 out the other end. Amazing.