Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scenedetect fails on mp4 video with multiple audio tracks #179

Closed
tslater opened this issue Aug 15, 2020 · 19 comments
Closed

Scenedetect fails on mp4 video with multiple audio tracks #179

tslater opened this issue Aug 15, 2020 · 19 comments
Labels
bug Issues directly affecting expected behavior (e.g. application crash). known issue Known limitations or bugs in current production builds/distributions. status: completed Completed issues :)
Milestone

Comments

@tslater
Copy link

tslater commented Aug 15, 2020

Bug/Issue Description:
I get fatal errors using scenedetect with a multiaudio stream mp4. When I used ffmpeg to copy to an mkv with the same streams, or when I copy to an mp4 with only one audio stream, it works. It will not work with multiple streams in an mp4.

Required Information:
Provide the following information to assist with reporting the bug:

  1. Provide a full copy of the command line options you are using, for example:
scenedetect --input /Volumes/sources/sources/video/movie/chapter_5/landscape.mp4  --output /Volumes/sources/sources/movie/movie/chapter_5/scene_detect  --stats stats.csv detect-content list-scenes save-images            

attach the generated BUG_REPORT.txt file.

INFO: __init__.scenedetect_cli(): Version: v0.5.3
INFO: __init__.scenedetect_cli(): Info Level: debug
INFO: __init__.scenedetect_cli(): Output directory set:
  /Volumes/sources/sources/video/movie/chapter_5/scene_detect
DEBUG: context.parse_options(): Parsing program options.
DEBUG: context._init_video_manager(): Initializing VideoManager.
INFO: video_manager.__init__(): Loaded 1 video, framerate: 23.98 FPS, resolution: 886 x 480
INFO: video_manager.set_downscale_factor(): Downscale factor set to 3, effective resolution: 295 x 160
DEBUG: context.parse_options(): VideoManager initialized.
DEBUG: __init__.detect_content_command(): Detecting content, parameters:
  threshold: 30, min-scene-len: 14
INFO: context.list_scenes_command(): Scene list CSV file name format:
  $VIDEO_NAME-Scenes.csv
INFO: context.save_images_command(): Image output format set: JPEG
DEBUG: context.process_input(): Processing input...
INFO: context.process_input(): Detecting scenes...
INFO: context.process_input(): Processed 0 frames in 0.0 seconds (average 0.00 FPS).
DEBUG: context.process_input(): No frame metrics updated, skipping update of the stats file.
INFO: context.process_input(): Detected 1 scenes, average shot length 0.0 seconds.
INFO: context.process_input(): Writing scene list to CSV file:
  /Volumes/sources/sources/video/movie/chapter_5/scene_detect/landscape-Scenes.csv
INFO: context.process_input(): Scene List:
-----------------------------------------------------------------------
 | Scene # | Start Frame |  Start Time  |  End Frame  |   End Time   |
-----------------------------------------------------------------------
 |      1  |           0 | 00:00:00.000 |           0 | 00:00:00.000 |
-----------------------------------------------------------------------

INFO: video_manager.set_downscale_factor(): Downscale factor set to 1, effective resolution: 886 x 480
INFO: context._generate_images(): Generating output images (3 per scene)...
DEBUG: context.cleanup(): Cleaning up...


INFO: __init__.scenedetect_cli(): Version: v0.5.3
INFO: __init__.scenedetect_cli(): Info Level: debug
INFO: __init__.scenedetect_cli(): Output directory set:
  /Volumes/sources/sources/video/movie/chapter_5/scene_detect
DEBUG: context.parse_options(): Parsing program options.
DEBUG: context._init_video_manager(): Initializing VideoManager.
INFO: video_manager.__init__(): Loaded 1 video, framerate: 23.98 FPS, resolution: 886 x 480
INFO: video_manager.set_downscale_factor(): Downscale factor set to 3, effective resolution: 295 x 160
DEBUG: context.parse_options(): VideoManager initialized.
DEBUG: __init__.detect_content_command(): Detecting content, parameters:
  threshold: 30, min-scene-len: 14
INFO: context.list_scenes_command(): Scene list CSV file name format:
  $VIDEO_NAME-Scenes.csv
INFO: context.save_images_command(): Image output format set: JPEG
DEBUG: context.process_input(): Processing input...
INFO: context.process_input(): Detecting scenes...
INFO: context.process_input(): Processed 0 frames in 0.0 seconds (average 0.00 FPS).
DEBUG: context.process_input(): No frame metrics updated, skipping update of the stats file.
INFO: context.process_input(): Detected 1 scenes, average shot length 0.0 seconds.
INFO: context.process_input(): Writing scene list to CSV file:
  /Volumes/sources/sources/video/movie/chapter_5/scene_detect/landscape-Scenes.csv
INFO: context.process_input(): Scene List:
-----------------------------------------------------------------------
 | Scene # | Start Frame |  Start Time  |  End Frame  |   End Time   |
-----------------------------------------------------------------------
 |      1  |           0 | 00:00:00.000 |           0 | 00:00:00.000 |
-----------------------------------------------------------------------

INFO: video_manager.set_downscale_factor(): Downscale factor set to 1, effective resolution: 886 x 480
INFO: context._generate_images(): Generating output images (3 per scene)...
DEBUG: context.cleanup(): Cleaning up...


Expected Behavior:
Expect the process to run with output and no error

Computing Environment:
Dockerfile

FROM python:3.8-slim

RUN apt-get update && apt-get install -y \
  libglib2.0-0 \
  libsm6 \
  libxext6 \
  libxrender-dev \
  libgl1-mesa-glx \
  && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir scenedetect[opencv,progress_bar]==0.5.3

Additional Information:
Just my gratefulness for making this!

@tslater
Copy link
Author

tslater commented Aug 18, 2020

Here's the error omitted when I run without BUG_REPORT.txt:


[PySceneDetect] Downscale factor set to 1, effective resolution: 886 x 480
[PySceneDetect] Generating output images (3 per scene)...

  0%|          | 0/3 [00:00<?, ?images/s]Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 792, in main
    ctx.exit()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 558, in exit
    raise Exit(code)
click.exceptions.Exit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/scenedetect", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/scenedetect/__main__.py", line 58, in main
    cli.main(obj=cli_ctx)   # Parse CLI arguments with registered callbacks.
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 792, in main
    ctx.exit()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 404, in __exit__
    self.close()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 494, in close
    cb()
  File "/usr/local/lib/python3.8/site-packages/scenedetect/cli/context.py", line 395, in process_input
    self._generate_images(scene_list=scene_list, video_name=video_name,
  File "/usr/local/lib/python3.8/site-packages/scenedetect/cli/context.py", line 200, in _generate_images
    timecode_list = [
  File "/usr/local/lib/python3.8/site-packages/scenedetect/cli/context.py", line 201, in <listcomp>
    [
  File "/usr/local/lib/python3.8/site-packages/scenedetect/cli/context.py", line 202, in <listcomp>
    FrameTimecode(int(f), fps=fps) for f in [
  File "/usr/local/lib/python3.8/site-packages/scenedetect/frame_timecode.py", line 115, in __init__
    self.frame_num = self._parse_timecode_number(timecode)
  File "/usr/local/lib/python3.8/site-packages/scenedetect/frame_timecode.py", line 251, in _parse_timecode_number
    raise ValueError('Timecode frame number must be positive and greater than zero.')
ValueError: Timecode frame number must be positive and greater than zero.

  0%|          | 0/3 [00:00<?, ?images/s]
    at ChildProcess.exithandler (child_process.js:308:12)
    at ChildProcess.emit (events.js:314:20)
    at ChildProcess.EventEmitter.emit (domain.js:486:12)
    at maybeClose (internal/child_process.js:1051:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)

@Breakthrough
Copy link
Owner

Breakthrough commented Aug 19, 2020

Hi @tslater;

Are you running into this issue with all files with multiple audio tracks, or just a few? I'll try to replicate this on my end and keep you posted (may need a small sample if I'm unable to reproduce on my end). Looks like you might've found a couple bugs here, thanks for the report.

One of my concerns is I'm not sure at first glance how I would go about detecting this condition at runtime, without relying on another tool like ffprobe, but will definitely do some more investigation on my end and see if I can come up with a resolution. I haven't heard any issues like this before with OpenCV + video files, so hoping this might just be a one-off issue.

Thanks for the report!

@Breakthrough Breakthrough added the bug Issues directly affecting expected behavior (e.g. application crash). label Aug 19, 2020
@tslater
Copy link
Author

tslater commented Aug 19, 2020

You're very welcome. I'm seeing that it mkv videos with multiple audio tracks are working, but not mp4. Hopefully that helps.

@tslater
Copy link
Author

tslater commented Aug 19, 2020

I can send an offending video file, but privately because the video files I'm working with are all copyrighted. I have permission to work with the film, but distribution rights are currently in negotiation status. Just let me know an email I can send it to if that helps!

@Breakthrough
Copy link
Owner

Breakthrough commented Aug 20, 2020

@tslater If you could that would be great. Send me a message via the form on http://bcastell.com/contact/ and I'll get in touch with you. Thanks!

@tslater
Copy link
Author

tslater commented Aug 21, 2020

I think your contact form is broken? I get a 400 JS error.

@tslater

This comment has been minimized.

@Breakthrough
Copy link
Owner

Breakthrough commented Aug 22, 2020

@tslater
Copy link
Author

tslater commented Aug 22, 2020

@Breakthrough
Copy link
Owner

Breakthrough commented Aug 22, 2020

Awesome, many thanks for the sample. Now here's where things are going to get a bit tricky unfortunately - I was able to run the video successfully, as well as split it on my end. However, when playing the video back using VLC, I've noticed a few inconsistencies when playing it back on my end (it reports video length as ~15 minutes, and after I hit "play", the timestamp jumps to 30 seconds)...

So I'm wondering if it might be due to how the files are being generated/encoded (this isn't common, but has cropped up occasionally for some users). However, it's strange that PySceneDetect seems to work fine on my end (in particular, my OpenCV/ffmpeg libraries/binaries).

Could you provide some more details about the OpenCV package you're using, as well as particular OS distribution / machine? The more details you can provide the better. I'll try the same environment in a Linux VM to see if I can replicate it on my end, but the closer I can get to your setup, the better it would be to help track down the issue here.

Likewise, thank you for the material & feedback. In the meantime, is re-encoding your source material or using MKV as an input container a viable workaround for you? Just want to make sure you're not blocked by this bug while we figure out what's going on here.


Console output I get:

[PySceneDetect] Processed 253 frames in 0.1 seconds (average 3011.25 FPS).
[PySceneDetect] Detected 5 scenes, average shot length 2.1 seconds.
[PySceneDetect] Writing scene list to CSV file:
  C:\Users\Pietro\Documents\git-repos\PySceneDetect\blah\test-Scenes.csv
[PySceneDetect] Scene List:
-----------------------------------------------------------------------
 | Scene # | Start Frame |  Start Time  |  End Frame  |   End Time   |
-----------------------------------------------------------------------
 |      1  |           0 | 00:00:00.000 |          68 | 00:00:02.836 |
 |      2  |          68 | 00:00:02.836 |         173 | 00:00:07.216 |
 |      3  |         173 | 00:00:07.216 |         193 | 00:00:08.050 |
 |      4  |         193 | 00:00:08.050 |         224 | 00:00:09.343 |
 |      5  |         224 | 00:00:09.343 |         253 | 00:00:10.552 |
-----------------------------------------------------------------------

Also this is a perfect scene for testing a new feature I plan on rolling out in #35 to suppress flashes that cause false positives - I was looking for something like this for a long time (where flashlights "strobe" across the frame). I'll do my due diligence and obtain a copy legally on my end so I can extract this short segment and possibly use it as a test case for that feature (the short clip you provided me will only be used by me to triaging this specific issue, and will never be distributed as agreed). Love this movie, btw :)

@tslater
Copy link
Author

tslater commented Aug 22, 2020

Yeah the timing is off in the clip I sent you. I actually cut it down from a longer clip that was ~15 minutes because my upload speed is horrendous and the original clip is ~500MB or so...but the timing is accurate on that one. That is interesting that it is working for you. I am actually using docker (for consistency), so if you are able to use that, it should be easy to replicate. Here is my Dockerfile:

FROM python:3.8-slim

RUN apt-get update && apt-get install -y \
  libglib2.0-0 \
  libsm6 \
  libxext6 \
  libxrender-dev \
  libgl1-mesa-glx \
  && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir scenedetect[opencv,progress_bar]==0.5.3

@Breakthrough
Copy link
Owner

Breakthrough commented Aug 22, 2020

Just to confirm, you're experiencing the same issue with that cut sample you sent me? (Smaller is better for issues like this, most definitely - just hoping it's not due to the way you cut the file, so want to confirm that first)

I'm not really familiar with docker so might need to have a few colleagues get me up to speed with that. In the meantime I'll try to replicate this under a Linux environment. I was under the impression that the operating system Docker was running on is important - do you have that information by any chance? (If not, no worries - I still have a lot of triaging to do with this one on my end still, just trying to grasp for any additional details before getting into too deep of a rabbit hole haha).

@tslater
Copy link
Author

tslater commented Aug 22, 2020

Yeah, the issue exists in the parent clip and the one I sent you. I confirmed before I sent it. It might be something unique to the docker setup. I'm using Docker on Ubuntu 18.04, but that's really the point of docker is that the underlying operating system should make little to no difference. At this point, maybe it would be easier for me to start getting into PySceneDetect source code and do my own debugging than make you learn docker though...Unless you're interested.

@Breakthrough
Copy link
Owner

Breakthrough commented Aug 22, 2020

That definitely helps! Will setup an Ubuntu VM on my end and let you know if I can replicate. The issue here (from what I can see in the debug logs you provided at least) is that the underlying OpenCV/ffmpeg implementation doesn't seem to give any errors for PySceneDetect to report - it seems to fail silently, with 0 frames processed. I definitely need to add some better handling of that situation so I can report that something went wrong, but am uncertain if I can tell the user why specifically (other than just that the video is incompatible, and to try re-encoding/re-muxing/transcoding it).

If you're up for it, you can try to just use the OpenCV library from Python directly to open the video file using the VideoCapture class and try to read() a few frames from it. As mentioned though, I'll setup a VM to try and replicate this on my end, so no rush (but anything else you find in the meantime would definitely be helpful!). Just working on a few other issues for the v0.5.4 release, after which this will be the highest priority issue.

Even if I can't find a root cause for why this is happening, better error handling is most definitely required on my end here so I can report a better error message when this situation occurs (which has been occasionally reported in the past). Hopefully I can sneak that in for v0.5.4 as well. All the best, will keep you posted - and thank you again for providing a test case & the required info!

@Breakthrough Breakthrough added the status: accepted Issues accepted for inclusion in an upcoming version of PySceneDetect. label Aug 22, 2020
@tslater
Copy link
Author

tslater commented Aug 22, 2020

Thanks so much. If I have time later next week, and I haven't heard back, I might do some poking with OpenCV

Breakthrough added a commit that referenced this issue Sep 2, 2020
@Breakthrough Breakthrough added status: completed Completed issues :) and removed status: accepted Issues accepted for inclusion in an upcoming version of PySceneDetect. labels Sep 2, 2020
@Breakthrough Breakthrough added this to the v0.5.4 milestone Sep 2, 2020
@Breakthrough
Copy link
Owner

Breakthrough commented Sep 2, 2020

Hey @tslater;

Should have some improved messaging for this error now in v0.5.4, so this should be in the next release. Long term would require switching libraries to handle video I/O which isn't really feasible right now unfortunately. I did update the FAQ with two methods to get around this issue using ffmpeg or mkvmerge via the following commands:

ffmpeg -i input.mp4 -c copy -an output.mp4

mkvmerge -o output.mkv input.mp4

Thanks again for the report!

@Breakthrough Breakthrough reopened this Sep 2, 2020
@Breakthrough Breakthrough added the known issue Known limitations or bugs in current production builds/distributions. label Oct 6, 2020
@Breakthrough
Copy link
Owner

Breakthrough commented Aug 14, 2021

This may actually be possible to fix once multiple video backends are supported as per #213 - that is still a way's away, but just wanted to state that for completeness.

Edit: Have re-opening this and moving to the v1.0 release, as I expect supporting PyAV should resolve the issue.

@Breakthrough Breakthrough removed the status: completed Completed issues :) label Aug 21, 2021
@Breakthrough Breakthrough added the status: accepted Issues accepted for inclusion in an upcoming version of PySceneDetect. label Aug 21, 2021
@Breakthrough Breakthrough modified the milestones: v0.5.4, v1.0 Aug 21, 2021
@Breakthrough
Copy link
Owner

Breakthrough commented Jan 24, 2022

Interestingly enough, I'm now unable to reproduce this issue with the latest version of Python/OpenCV. If anyone runs into this issue, try updating to the latest version and see if that resolves the issue - and if not, please post an update here!

@Breakthrough Breakthrough added status: awaiting response Issues awaiting response from a community member. and removed status: accepted Issues accepted for inclusion in an upcoming version of PySceneDetect. labels Jan 24, 2022
@Breakthrough Breakthrough modified the milestones: v1.0, v0.6 Mar 11, 2022
@Breakthrough Breakthrough added status: in progress Issues either actively being worked on, or which have had some progress since being filed. and removed status: awaiting response Issues awaiting response from a community member. labels Mar 11, 2022
@Breakthrough
Copy link
Owner

Breakthrough commented Mar 11, 2022

Some users are still running into issues with this - this should be resolved in v0.6 if you install PyAV (#213). An error message will also be displayed when processing a video fails indicating to attempt using the PyAV backend as an alternative.

The v0.6 beta is out now which has this fix in place.

@Breakthrough Breakthrough added status: completed Completed issues :) and removed status: in progress Issues either actively being worked on, or which have had some progress since being filed. labels Mar 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issues directly affecting expected behavior (e.g. application crash). known issue Known limitations or bugs in current production builds/distributions. status: completed Completed issues :)
Projects
None yet
Development

No branches or pull requests

2 participants