Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Light Flash/Strobe Suppression #35

Open
annishaa88 opened this issue Jun 13, 2017 · 12 comments
Open

Light Flash/Strobe Suppression #35

annishaa88 opened this issue Jun 13, 2017 · 12 comments

Comments

@annishaa88
Copy link

@annishaa88 annishaa88 commented Jun 13, 2017

There are some videos that have camera flashes in them

for example:
http://assetsprod2-a.akamaihd.net/tag_reuters_com_2017_newsml_ov6kua1nj_reuters_ingest/tag_reuters_com_2017_newsml_ov6kua1nj_reuters_ingest_LOWRES.mp4
this video gives me the following result: ['00:00:09.766', '00:01:23.266', '00:01:35.066']

Can I do something about it?

@Breakthrough
Copy link
Owner

@Breakthrough Breakthrough commented Sep 6, 2017

Hello @annishaa88;

Which detector algorithm are you using in this case, threshold, or content? Currently there is only support to ignore subsequent camera flashes within a certain window (using the minimum-scene-length argument), but the initial flash in the window will still be detected due to the design of the detection algorithms.

That being said, I can see this being a relatively common issue, so after the following release of PySceneDetect (v0.5, where the focus is major changes to the Python API), I will look into adding support for ignoring camera flash in certain videos. There are a few ideas I have that can solve this issue, but they will require some modification to the existing detection algorithms (as well as some additional command-line parameters), but are definitely possible, and should take care of almost all instances of flicker/camera flash.

If you have any suggestions regarding how the implementation should be, by all means, your comments would be most welcome. I will keep you posted as to my progress in this regard, and should hopefully have something for you to test after the next major release. Lastly, thank you very much for providing a sample video - this will be quite handy when the time for testing finally rolls around.

@Breakthrough
Copy link
Owner

@Breakthrough Breakthrough commented Jul 7, 2018

My apologies for the lack of update in regards to progress. Unfortunately my development efforts have been focused on the release of the new v0.5 API/CLI, and not so much on enhancements/features. I hope to have the new version released by the end of the month, at which time I can attempt to tackle this issue.

Just to confirm again @annishaa88, what command were you using to generate the results? The specific detection algorithm/thresholds being used would be very useful information.

@Breakthrough
Copy link
Owner

@Breakthrough Breakthrough commented Jul 11, 2018

Also one idea I just had to solve this is to add the ability to specify an intensity value which if the frame exceeds it, scene detection will be disabled. This might be rather easy to implement, so I will look into squeezing this in for the upcoming release.

@wjs018
Copy link
Contributor

@wjs018 wjs018 commented Sep 2, 2018

Hi Breakthrough,

I have an example video that I have analyzed with bright flashes that cause erroneous scene breaks being detected. The music video for Growl by EXO is done in one continuous shot, but there are strobe lights that flash in the background. I analyzed the video using the content aware detector using the settings of threshold=30 and min_scene_len=10 and ended up with a total of 83 scenes being detected (link). I would expect at least a couple due to transitions to and from title cards, but the strobes account for the vast majority.

On a side note, I have updated to the newest version, and have been liking it so far. The new API has been working great.

@Breakthrough
Copy link
Owner

@Breakthrough Breakthrough commented Sep 2, 2018

Hi @wjs018;

Thank you very much for the extensive example - very well put together as well, might I add. Also thanks for your comments regarding the API, means a lot to me - if you have any improvements you want to suggest, feel free to bring them forwards.

I read through some of your work, and agree that edge detection is definitely a viable solution to the strobing issue. I'm looking into how I can create a new EdgeDetector class to detect scenes purely using edge detection, or possibly a more robust detector (RobustDetector?) that combines all features of the detectors (including slow fades and what not).

Also left a few suggestions for performance in one of your repos for how you might be able to improve your runtime using PySceneDetect - sorry the documentation is still under works, I need to add more examples of different usage styles, the current api_test.py is geared towards multiple calls to the function from starting/stopping the program entirely.

@wjs018
Copy link
Contributor

@wjs018 wjs018 commented Nov 24, 2018

@Breakthrough
I have been doing some work on this problem recently and implemented a working example of an EdgeDetector (code here). I tried to style it after the existing detectors in PySceneDetect and it seems to be plug and play with my existing programs. I basically took the same approach that skvideo does in their scenedet function (docs) (github). This function is just an implementation of this paper (pdf). It adds a dependency on skvideo for their motion estimation code, and a dependency on scipy for some binary image morphology operations.

Some notes about edge detection:

  • I tried a bunch of different methods to automatically generate the low and high threshold for the Canny edge filter, but none of them performed as well as the method used by skvideo, so I ended up using the same method.
  • The motion estimation parameter r_dist is the radius (in pixels) over which the detector will look for motion in the frame. However, this is pixels in the scaled down image if downscale_factor is not 1 in the VideoManager object.
  • The performance is much slower than the other detectors. For some numbers to compare, I get ~20 fps detecting scenes in a 1080p video with ContentDetector with no downscaling. Using EdgeDetector, I need to downscale by 4x in order to match that 20 fps.

Results:

  • Music video for Growl by EXO (one continuous shot) analyzed by ContentDetector:
    • video
    • 62 detected scenes (threshold=30, min_scene_len=10)
  • Same video analyzed by EdgeDetector:
    • video
    • 10 detected scenes (threshold=0.4, min_scene_len=10, r_dist=6)

Overall, I am happy with it for my purposes, but different videos are going to require parameter tuning to get better accuracy. There are certain videos for which I have found the ContentDetector seems to perform better, while others perform better with the EdgeDetector. I am going to experiment a bit with using both in combination. Perhaps to do something akin to the RobustDetector you mentioned, would it make sense to add the ability to not add anything to a cut list unless all detectors added detect a cut for that frame? Currently, I believe a cut is added to the list if any of the detectors is triggered (which works great in some cases).

@dave-epstein
Copy link

@dave-epstein dave-epstein commented May 4, 2019

Hey, is there any progress on dealing with camera flash, or does anyone know any libraries that are able to deal with this?

@Breakthrough
Copy link
Owner

@Breakthrough Breakthrough commented May 20, 2019

Hey @dave-epstein;

Sorry, no progress yet on that front, I'd like to start cleaning up the backlog before addressing any new features at the present moment in time... My apologies, haven't had much time to keep up with the project lately.

I definitely do want to integrate this with PySceneDetect though. In the meantime, any pull requests are still most welcome.

Thank you.

@Breakthrough
Copy link
Owner

@Breakthrough Breakthrough commented Jun 14, 2020

Interestingly, it appears that a pretty novel solution using a lookahead buffer was implemented in rav1e AVC encoder (which itself was based on the detect-content algorithm!):
https://github.com/xiph/rav1e/blob/master/src/scenechange/mod.rs

This indicates that an underlying design change will be required to support frame lookahead, but this seems like a viable (and awesome!) approach I never originally considered. Will definitely be looking more into how this can be integrated into PySceneDetect to allow for adding flash suppression to detect-content.

@Breakthrough
Copy link
Owner

@Breakthrough Breakthrough commented Jul 1, 2020

I'll take a bite at this one for the v0.6.x release, as it's a really nice to have feature. I think I managed to come up with a method that doesn't require a lookahead buffer, and has minimal impact on performance. It doesn't use edge detection, but again the same method as rav1e just modified to use a state machine rather than frame lookahead.

Edit: I don't want to ignore your other method either, @wjs018 - would be awesome if that could be either integrated with ContentDetector or shipped as part of PySceneDetect as another detection method. Just think I have a way to solve the most pressing use cases with minimal impact on performance (and a more "tunable" max # frames per flash setting).

@Breakthrough Breakthrough self-assigned this Jul 1, 2020
@Breakthrough Breakthrough added this to the v0.6 milestone Jul 4, 2020
@Breakthrough Breakthrough added this to In Progress in v0.6 Release Jul 4, 2020
@Breakthrough
Copy link
Owner

@Breakthrough Breakthrough commented Jul 19, 2020

I think my method may not perform as well as yours @wjs018, but will have minimal performance impact. Will likely be using yours as a source of test footage. It's probably worth shipping your edge detector with one of the next releases, as certain users will likely have the same use case as you did.

@Breakthrough Breakthrough pinned this issue Jul 19, 2020
Breakthrough added a commit that referenced this issue Jul 19, 2020
Add unit tests to validate flash suppression on existing test case video.

TODO: Hook up new API parameter to CLI.
@Breakthrough Breakthrough moved this from In Progress to To Do in v0.6 Release Jul 19, 2020
@Breakthrough Breakthrough moved this from To Do to In Progress in v0.6 Release Jul 19, 2020
@Breakthrough Breakthrough changed the title Camera flash question Light Flash/Strobe Suppression Aug 4, 2020
@Breakthrough
Copy link
Owner

@Breakthrough Breakthrough commented Aug 4, 2020

This has been implemented in the v0.6.x branch (link to download .zip), for users wishing to beta test this feature before it's official release. It is turned on by default when running detect-content with a suppression amount of 2 frames.

The suppression amount (called flicker_frames) can be changed by the call to detect-content via the -f / --flicker [N] argument, which specifies the flash suppression amount, in frames, e.g.:

scenedetect -i video.mp4 detect-content -f 3

To turn flash suppression off, set -f to 0. Any feedback is most welcome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
v0.6 Release
  
In Progress
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants