Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reference of paper for the methods used. #62

Open
leonwyang opened this issue Jul 18, 2018 · 6 comments
Open

Reference of paper for the methods used. #62

leonwyang opened this issue Jul 18, 2018 · 6 comments
Labels

Comments

@leonwyang
Copy link

@leonwyang leonwyang commented Jul 18, 2018

Hi Brandon,

Thanks this is a great work! Could you please list the paper(if there is any) that inspired the detection algorithms? It would be nice if I can see the the paper and decide whether this is the right algorithm or how to change it to meet my own application.

Thanks.

@Breakthrough

This comment has been minimized.

Copy link
Owner

@Breakthrough Breakthrough commented Jul 19, 2018

Hello @leonwyang;

You may find the following papers useful:
https://ieeexplore.ieee.org/document/7040826/
https://ieeexplore.ieee.org/document/7079058/

You might also learn about some other algorithms which are more advanced by searching "scene change detection" on IEEEXplore. Currently there is a threshold (pixel value) based algorithm, content (changes in HSV colour space) detection algorithm, and motion-detection via background subtraction.

I am also investigating building a histogram instead of computing whole-frame delta scores, although the number of frequency bins would probably be small in order to have it not take up too much room in the statistics file, as well as have it easy for end users to determine optimal parameters.

Another big thing I want to look into is adaptive threshold values, or automatically detecting the optimal value by analyzing the peak values for a given video.

If you have any questions regarding your own application feel free to share them here and I will answer them to the best of my ability. Thank you for your interest.

@leonwyang

This comment has been minimized.

Copy link
Author

@leonwyang leonwyang commented Jul 20, 2018

Hello @Breakthrough

Thanks Brandon! I'll look into the literature you pointed to.

The type of video I am dealing with is surgery video, where basically the entire video consists of scenes that surgeons doing surgery inside human body, for example, in this video. The content based algorithm works pretty well for this type but makes false predictions when there is for example a reflection light came out from the equipment or there is a hand with white glove blocks part of the video.

It seems that the content based method does not work too well for gradual change, there are cases where
part of the surgery video is cut out of a whole procedure, but it has the gradual cut between who scenes that does not differ that much, and I suppose the dissolve detector maybe better in this case? Do you know if there is any scene detection algorithm that is robust to both abrupt and gradual change?

And It would be great if you can point to any package that can do dissolve scene change detection.

Thanks!

@Breakthrough

This comment has been minimized.

Copy link
Owner

@Breakthrough Breakthrough commented Jul 22, 2018

Hi @leonwyang;

One feature I have been looking into adding to PySceneDetect is a dissolve detector, hopefully once the v0.5 release is done I will have some more time to work on it. Other than these programs, I'm not really familiar with any other software that can accomplish what this does.

That being said, there is already an open issue (#35) regarding camera flashes. After reviewing the linked video, it appears that the reflection or white glove blocking issues may also be resolved once I determine an algorithm to deal with the flashing or momentary interruptions of videos.

That being said, there are some other parts of the video that aren't encompassed in the camera flash issue, so I will leave this issue open for further discussion/suggestions in regards to development of these features. I will also create a separate issue when development starts on the dissolve detector to track progress.

Thanks for providing some sample material to experiment with. If you yourself have any suggestions in regards to how some of these features should be implemented (esp. with respect to the algorithm choices), please feel free to share your opinion on the matter.

@leonwyang

This comment has been minimized.

Copy link
Author

@leonwyang leonwyang commented Jul 23, 2018

Hi Brandon,

Thanks for the detailed response regarding all the issues.

I am just wondering at this point do you have any algorithm for dissolve detector in mind(for example, any paper that may be helpful)? I could go look it up and implement it myself.

Thanks!

@Breakthrough

This comment has been minimized.

Copy link
Owner

@Breakthrough Breakthrough commented Jul 26, 2018

For dissolve, I was going to look at cases where in HSL colourspace, there is a consistent drop in luma (either linear or exp/log), and use that to signal a scene cut/break.

Another option would be to use histograms (see the notebook in #53) and detect a shift in the mean downwards, although I'm not sure if binning pixels is required for dissolve detection.

@Breakthrough

This comment has been minimized.

Copy link
Owner

@Breakthrough Breakthrough commented Aug 5, 2019

I've added a new page to the PySceneDetect site which aims to track uses of the project in literature, which will now allow tracking of proposed and better scene detection algorithms in the future:

https://pyscenedetect.readthedocs.io/en/latest/other/resources/

I have seen a few interesting papers comparing the baseline content-detection algorithm against other alternatives and new designs (usually NN-based, some hybrids) which are definitely worth considering going forwards. If anyone has any other pieces of literature they think would be relevant to be added to that page, please feel free to let me know.

Eventually I would also like that page to contain some of the sources contained in this issue as a way of providing some insight into the current detection algorithms as well as other common implementations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.