New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Camera flash question #35

Open
annishaa88 opened this Issue Jun 13, 2017 · 6 comments

Comments

Projects
None yet
3 participants
@annishaa88

annishaa88 commented Jun 13, 2017

There are some videos that have camera flashes in them

for example:
http://assetsprod2-a.akamaihd.net/tag_reuters_com_2017_newsml_ov6kua1nj_reuters_ingest/tag_reuters_com_2017_newsml_ov6kua1nj_reuters_ingest_LOWRES.mp4
this video gives me the following result: ['00:00:09.766', '00:01:23.266', '00:01:35.066']

Can I do something about it?

@Breakthrough

This comment has been minimized.

Owner

Breakthrough commented Sep 6, 2017

Hello @annishaa88;

Which detector algorithm are you using in this case, threshold, or content? Currently there is only support to ignore subsequent camera flashes within a certain window (using the minimum-scene-length argument), but the initial flash in the window will still be detected due to the design of the detection algorithms.

That being said, I can see this being a relatively common issue, so after the following release of PySceneDetect (v0.5, where the focus is major changes to the Python API), I will look into adding support for ignoring camera flash in certain videos. There are a few ideas I have that can solve this issue, but they will require some modification to the existing detection algorithms (as well as some additional command-line parameters), but are definitely possible, and should take care of almost all instances of flicker/camera flash.

If you have any suggestions regarding how the implementation should be, by all means, your comments would be most welcome. I will keep you posted as to my progress in this regard, and should hopefully have something for you to test after the next major release. Lastly, thank you very much for providing a sample video - this will be quite handy when the time for testing finally rolls around.

@Breakthrough

This comment has been minimized.

Owner

Breakthrough commented Jul 7, 2018

My apologies for the lack of update in regards to progress. Unfortunately my development efforts have been focused on the release of the new v0.5 API/CLI, and not so much on enhancements/features. I hope to have the new version released by the end of the month, at which time I can attempt to tackle this issue.

Just to confirm again @annishaa88, what command were you using to generate the results? The specific detection algorithm/thresholds being used would be very useful information.

@Breakthrough

This comment has been minimized.

Owner

Breakthrough commented Jul 11, 2018

Also one idea I just had to solve this is to add the ability to specify an intensity value which if the frame exceeds it, scene detection will be disabled. This might be rather easy to implement, so I will look into squeezing this in for the upcoming release.

@wjs018

This comment has been minimized.

wjs018 commented Sep 2, 2018

Hi Breakthrough,

I have an example video that I have analyzed with bright flashes that cause erroneous scene breaks being detected. The music video for Growl by EXO is done in one continuous shot, but there are strobe lights that flash in the background. I analyzed the video using the content aware detector using the settings of threshold=30 and min_scene_len=10 and ended up with a total of 83 scenes being detected (link). I would expect at least a couple due to transitions to and from title cards, but the strobes account for the vast majority.

On a side note, I have updated to the newest version, and have been liking it so far. The new API has been working great.

@Breakthrough

This comment has been minimized.

Owner

Breakthrough commented Sep 2, 2018

Hi @wjs018;

Thank you very much for the extensive example - very well put together as well, might I add. Also thanks for your comments regarding the API, means a lot to me - if you have any improvements you want to suggest, feel free to bring them forwards.

I read through some of your work, and agree that edge detection is definitely a viable solution to the strobing issue. I'm looking into how I can create a new EdgeDetector class to detect scenes purely using edge detection, or possibly a more robust detector (RobustDetector?) that combines all features of the detectors (including slow fades and what not).

Also left a few suggestions for performance in one of your repos for how you might be able to improve your runtime using PySceneDetect - sorry the documentation is still under works, I need to add more examples of different usage styles, the current api_test.py is geared towards multiple calls to the function from starting/stopping the program entirely.

@wjs018

This comment has been minimized.

wjs018 commented Nov 24, 2018

@Breakthrough
I have been doing some work on this problem recently and implemented a working example of an EdgeDetector (code here). I tried to style it after the existing detectors in PySceneDetect and it seems to be plug and play with my existing programs. I basically took the same approach that skvideo does in their scenedet function (docs) (github). This function is just an implementation of this paper (pdf). It adds a dependency on skvideo for their motion estimation code, and a dependency on scipy for some binary image morphology operations.

Some notes about edge detection:

  • I tried a bunch of different methods to automatically generate the low and high threshold for the Canny edge filter, but none of them performed as well as the method used by skvideo, so I ended up using the same method.
  • The motion estimation parameter r_dist is the radius (in pixels) over which the detector will look for motion in the frame. However, this is pixels in the scaled down image if downscale_factor is not 1 in the VideoManager object.
  • The performance is much slower than the other detectors. For some numbers to compare, I get ~20 fps detecting scenes in a 1080p video with ContentDetector with no downscaling. Using EdgeDetector, I need to downscale by 4x in order to match that 20 fps.

Results:

  • Music video for Growl by EXO (one continuous shot) analyzed by ContentDetector:
    • video
    • 62 detected scenes (threshold=30, min_scene_len=10)
  • Same video analyzed by EdgeDetector:
    • video
    • 10 detected scenes (threshold=0.4, min_scene_len=10, r_dist=6)

Overall, I am happy with it for my purposes, but different videos are going to require parameter tuning to get better accuracy. There are certain videos for which I have found the ContentDetector seems to perform better, while others perform better with the EdgeDetector. I am going to experiment a bit with using both in combination. Perhaps to do something akin to the RobustDetector you mentioned, would it make sense to add the ability to not add anything to a cut list unless all detectors added detect a cut for that frame? Currently, I believe a cut is added to the list if any of the detectors is triggered (which works great in some cases).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment