Reference of paper for the methods used.

[leonwyang]

Hi Brandon,

Thanks this is a great work! Could you please list the paper(if there is any) that inspired the detection algorithms? It would be nice if I can see the the paper and decide whether this is the right algorithm or how to change it to meet my own application.

Thanks.

[Breakthrough]

Hello @leonwyang;

You may find the following papers useful:
https://ieeexplore.ieee.org/document/7040826/
https://ieeexplore.ieee.org/document/7079058/

You might also learn about some other algorithms which are more advanced by searching "scene change detection" on IEEEXplore. Currently there is a threshold (pixel value) based algorithm, content (changes in HSV colour space) detection algorithm, and motion-detection via background subtraction.

I am also investigating building a histogram instead of computing whole-frame delta scores, although the number of frequency bins would probably be small in order to have it not take up too much room in the statistics file, as well as have it easy for end users to determine optimal parameters.

Another big thing I want to look into is adaptive threshold values, or automatically detecting the optimal value by analyzing the peak values for a given video.

If you have any questions regarding your own application feel free to share them here and I will answer them to the best of my ability. Thank you for your interest.

[leonwyang]

Hello @Breakthrough

Thanks Brandon! I'll look into the literature you pointed to.

The type of video I am dealing with is surgery video, where basically the entire video consists of scenes that surgeons doing surgery inside human body, for example, in this video. The content based algorithm works pretty well for this type but makes false predictions when there is for example a reflection light came out from the equipment or there is a hand with white glove blocks part of the video.

It seems that the content based method does not work too well for gradual change, there are cases where
part of the surgery video is cut out of a whole procedure, but it has the gradual cut between who scenes that does not differ that much, and I suppose the dissolve detector maybe better in this case? Do you know if there is any scene detection algorithm that is robust to both abrupt and gradual change?

And It would be great if you can point to any package that can do dissolve scene change detection.

Thanks!

[Breakthrough]

Hi @leonwyang;

One feature I have been looking into adding to PySceneDetect is a dissolve detector, hopefully once the v0.5 release is done I will have some more time to work on it. Other than these programs, I'm not really familiar with any other software that can accomplish what this does.

That being said, there is already an open issue (#35) regarding camera flashes. After reviewing the linked video, it appears that the reflection or white glove blocking issues may also be resolved once I determine an algorithm to deal with the flashing or momentary interruptions of videos.

That being said, there are some other parts of the video that aren't encompassed in the camera flash issue, so I will leave this issue open for further discussion/suggestions in regards to development of these features. I will also create a separate issue when development starts on the dissolve detector to track progress.

Thanks for providing some sample material to experiment with. If you yourself have any suggestions in regards to how some of these features should be implemented (esp. with respect to the algorithm choices), please feel free to share your opinion on the matter.

[leonwyang]

Hi Brandon,

Thanks for the detailed response regarding all the issues.

I am just wondering at this point do you have any algorithm for dissolve detector in mind(for example, any paper that may be helpful)? I could go look it up and implement it myself.

Thanks!

[Breakthrough]

For dissolve, I was going to look at cases where in HSL colourspace, there is a consistent drop in luma (either linear or exp/log), and use that to signal a scene cut/break.

Another option would be to use histograms (see the notebook in #53) and detect a shift in the mean downwards, although I'm not sure if binning pixels is required for dissolve detection.

[Breakthrough]

I've added a new page to the PySceneDetect site which aims to track uses of the project in literature, which will now allow tracking of proposed and better scene detection algorithms in the future:

https://pyscenedetect.readthedocs.io/en/latest/other/resources/

I have seen a few interesting papers comparing the baseline content-detection algorithm against other alternatives and new designs (usually NN-based, some hybrids) which are definitely worth considering going forwards. If anyone has any other pieces of literature they think would be relevant to be added to that page, please feel free to let me know.

Eventually I would also like that page to contain some of the sources contained in this issue as a way of providing some insight into the current detection algorithms as well as other common implementations.

[Breakthrough]

Closing this issue to cleanup the issue tracker. If anyone wishes to continue this discussion, however, feel free to either leave a comment here, reopen this issue, or create a new one.

Ideally, I'd like to eventually create a Wiki page that everyone can edit that contains references to PySceneDetect in literature, as well as references to other scene detection methodologies (if anyone would like to go for it, please feel free to do so!)

Thank you all!