Issue converting file, MP4Box Segfaults

[three18ti]

Hello,

First, this is awesome. Thanks for putting it together.

I am having an issue when running your script though, MP4Box is unable to create the file due to a segfault. I think the problem really is MP4Box but I had a couple questions about modifications that might make troubleshooting MP4Box easier.

1. Is it possible to show the full command being run? e.g. when MP4Box is invoked can we show the actual command line parameters?

2. Would it be possible to stop your script from cleaning up the temporary files?

I think you can see where I'm going with this, I'd like to invoke MP4Box directly and get the output from MP4Box as I'm sure it will be requested from those guys.

If you have encountered this before I'd be open to any suggestions.

I have attached the relevant logs below for the sake of completeness.

Thanks,
Jon

The error I get in /var/log/syslog is:

Nov  9 12:54:02 foo kernel: [144387.385017] MP4Box[15591]: segfault at 73746284 ip 00007fbfc3a84d75 sp 00007ffff95b4460 error 4 in libc-2.15.so[7fbfc3a06000+1b5000]

Here is the full output of my attempt to invoke xenonmkv

root@foo:~/jbillo-xenonmkv-bc4c612# ./xenonmkv.py /media/video/some.mkv --profile xbox360
2012-11-09 12:52:53,015 - xenonmkv [WARNING] An invalid number of channels was specified. Falling back to 2-channel stereo audio.
Extracting track 1 with the CodecID 'V_MPEG4/ISO/AVC' to the file 'temp_video.h264'. Container format: AVC/h.264 elementary stream
Extracting track 2 with the CodecID 'A_AC3' to the file 'temp_audio.ac3'. Container format: Dolby Digital (AC3)
Progress: 100%
MPlayer svn r34540 (Ubuntu), built with gcc-4.6 (C) 2000-2012 MPlayer Team

Playing /var/tmp/temp_audio.ac3.
libavformat version 53.21.0 (external)
Mismatching header version 53.19.0
libavformat file format detected.
[lavf] stream 0: audio (ac3), -aid 0
==========================================================================
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
libavcodec version 53.35.0 (external)
Mismatching header version 53.32.2
AUDIO: 48000 Hz, 2 ch, s16le, 384.0 kbit/25.00% (ratio: 48000->192000)
Selected audio codec: [ffac3] afm: ffmpeg (FFmpeg AC-3)
==========================================================================
[AO PCM] File: audiodump.wav (WAVE)
PCM: Samplerate: 48000Hz Channels: Stereo Format s16le
[AO PCM] Info: Faster dumping is achieved with -benchmark -vc null -vo null -ao pcm:fast
[AO PCM] Info: To write WAVE files use -ao pcm:waveheader (default).
AO: [pcm] 48000Hz 2ch s16le (2 bytes per sample)
Video: no video
Starting playback...
A:1209.6 (20:09.5) of 1209.6 (20:09.5)  0.3% 


BENCHMARKs: VC:   0.000s VO:   0.000s A:   3.779s Sys:   0.322s =    4.101s
BENCHMARK%: VC:  0.0000% VO:  0.0000% A: 92.1554% Sys:  7.8446% = 100.0000%

Exiting... (End of file)
Freeware Advanced Audio Coder
FAAC 1.28

Quantization quality: 150
Bandwidth: 22000 Hz
Object type: Low Complexity(MPEG-2) + M/S
Container format: Transport Stream (ADTS)
Encoding /var/tmp/audiodump.wav to /var/tmp/audiodump.aac
   frame          | bitrate | elapsed/estim | play/CPU | ETA
56700/56700 (100%)|  168.1  |   43.1/43.1   |   28.06x | 0.0  

2012-11-09 12:54:03,117 - xenonmkv [WARNING] An error occurred while creating an MP4 file with MP4Box; 2 retries left
2012-11-09 12:54:13,905 - xenonmkv [WARNING] An error occurred while creating an MP4 file with MP4Box; 1 retries left
2012-11-09 12:54:26,337 - xenonmkv [WARNING] An error occurred while creating an MP4 file with MP4Box; 0 retries left
2012-11-09 12:54:26,631 - xenonmkv [CRITICAL] package: MP4Box could not create file after 3 retries; giving up.
root@shepard:~/jbillo-xenonmkv-bc4c612# ls /var/tmp/
0x19e6140_isotmp  0x2329140_isotmp  0xb3e140_isotmp   ruby-uuid
0x1e86140_isotmp  0x241e140_isotmp  0xd3e140_isotmp   
root@shepard:~/jbillo-xenonmkv-bc4c612# ./xenonmkv.py /media/video/downloads/Archer.S03.720p.HDTV.x264/Archer.2009.S03E09.720p.HDTV.x264-IMMERSE.mkv --profile xbox360 -vv
2012-11-09 12:58:40,024 - xenonmkv [DEBUG] Using debug/highly verbose mode output
2012-11-09 12:58:40,024 - xenonmkv [WARNING] An invalid number of channels was specified. Falling back to 2-channel stereo audio.
2012-11-09 12:58:40,024 - xenonmkv [DEBUG] Starting XenonMKV
2012-11-09 12:58:40,024 - xenonmkv [DEBUG] Found dependent application mkvinfo in /usr/bin
2012-11-09 12:58:40,024 - xenonmkv [DEBUG] Found dependent application mediainfo in /usr/bin
2012-11-09 12:58:40,025 - xenonmkv [DEBUG] Found dependent application mkvextract in /usr/bin
2012-11-09 12:58:40,025 - xenonmkv [DEBUG] Found dependent application mplayer in /usr/bin
2012-11-09 12:58:40,025 - xenonmkv [DEBUG] Found dependent application faac in /usr/bin
2012-11-09 12:58:40,025 - xenonmkv [DEBUG] Found dependent application MP4Box in /usr/bin
2012-11-09 12:58:40,025 - xenonmkv [DEBUG] Using 'Archer.2009.S03E09.720p.HDTV.x264-IMMERSE' as final container name
2012-11-09 12:58:40,025 - xenonmkv [INFO] Loading source file /media/video/downloads/Archer.S03.720p.HDTV.x264/Archer.2009.S03E09.720p.HDTV.x264-IMMERSE.mkv 
2012-11-09 12:58:40,025 - xenonmkv [DEBUG] Executing 'mkvinfo /media/video/downloads/Archer.S03.720p.HDTV.x264/Archer.2009.S03E09.720p.HDTV.x264-IMMERSE.mkv'
2012-11-09 12:58:40,467 - xenonmkv [DEBUG] mkvinfo finished; attempting to parse output
2012-11-09 12:58:40,467 - xenonmkv [DEBUG] Audio duration detected as 1209.500 seconds
2012-11-09 12:58:40,467 - xenonmkv [DEBUG] Audio duration for MKV file may have been rounded: using 1210 seconds
2012-11-09 12:58:40,467 - xenonmkv [DEBUG] Executing 'mediainfo Video;%ID%,%Height%,%Width%,%Format_Settings_RefFrames%,%Language%,%FrameRate%,%CodecID%,%DisplayAspectRatio%,~ /media/video/downloads/Archer.S03.720p.HDTV.x264/Archer.2009.S03E09.720p.HDTV.x264-IMMERSE.mkv'
2012-11-09 12:58:40,758 - xenonmkv [DEBUG] mediainfo finished; attempting to parse output for video settings
2012-11-09 12:58:40,758 - xenonmkv [DEBUG] Executing 'mediainfo Audio;%ID%,%CodecID%,%Language%,%Channels%,~ /media/video/downloads/Archer.S03.720p.HDTV.x264/Archer.2009.S03E09.720p.HDTV.x264-IMMERSE.mkv'
2012-11-09 12:58:40,813 - xenonmkv [DEBUG] mediainfo finished; attempting to parse output for audio settings
2012-11-09 12:58:40,814 - xenonmkv [DEBUG] Detected track type 'video' from mkvinfo output
2012-11-09 12:58:40,814 - xenonmkv [DEBUG] Detected track number '1' from mkvinfo output
2012-11-09 12:58:40,814 - xenonmkv [DEBUG] Attempting to parse display aspect ratio '1.772'
2012-11-09 12:58:40,814 - xenonmkv [DEBUG] GCD of 1275 height, 1276 width is 0
2012-11-09 12:58:40,814 - xenonmkv [DEBUG] Calculated pixel aspect ratio is 561:561 (0.999875)
2012-11-09 12:58:40,814 - xenonmkv [DEBUG] Rounding pixel aspect ratio up to 1:1
2012-11-09 12:58:40,814 - xenonmkv [DEBUG] Video track 1 has dimensions 1276x720 with 5 reference frames
2012-11-09 12:58:40,814 - xenonmkv [DEBUG] Video track 1 has 23.976000 FPS and codec V_MPEG4/ISO/AVC
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] Video track 1 has display aspect ratio 1.772000
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] Video track 1 has a reasonable number of reference frames, and should be compatible with low-powered devices
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] All properties set for video track 1
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] Detected track type 'audio' from mkvinfo output
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] Detected track number '2' from mkvinfo output
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] Audio track 2 has codec A_AC3 and language en
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] Audio track 2 has 6 channel(s)
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] All properties set for audio track 2
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] All tracks detected from mkvinfo output; total number is 2
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] Source file /media/video/downloads/Archer.S03.720p.HDTV.x264/Archer.2009.S03E09.720p.HDTV.x264-IMMERSE.mkv has 1 audio and 1 video track; using these
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] Executing mkvextract on /media/video/downloads/Archer.S03.720p.HDTV.x264/Archer.2009.S03E09.720p.HDTV.x264-IMMERSE.mkv'
2012-11-09 12:58:40,815 - xenonmkv [DEBUG] Using /var/tmp/ as scratch directory for MKV extraction
2012-11-09 12:58:40,816 - xenonmkv [DEBUG] Using video track from MKV file with ID 1
2012-11-09 12:58:40,816 - xenonmkv [DEBUG] Using audio track from MKV file with ID 2
Extracting track 1 with the CodecID 'V_MPEG4/ISO/AVC' to the file 'temp_video.h264'. Container format: AVC/h.264 elementary stream
Extracting track 2 with the CodecID 'A_AC3' to the file 'temp_audio.ac3'. Container format: Dolby Digital (AC3)
Progress: 100%
2012-11-09 12:58:53,162 - xenonmkv [DEBUG] mkvextract finished; attempting to parse output
2012-11-09 12:58:53,253 - xenonmkv [DEBUG] Audio track /var/tmp/temp_audio.ac3 needs to be re-encoded
2012-11-09 12:58:53,253 - xenonmkv [DEBUG] Starting decoding file to WAV with mplayer
MPlayer svn r34540 (Ubuntu), built with gcc-4.6 (C) 2000-2012 MPlayer Team

Playing /var/tmp/temp_audio.ac3.
libavformat version 53.21.0 (external)
Mismatching header version 53.19.0
libavformat file format detected.
[lavf] stream 0: audio (ac3), -aid 0
==========================================================================
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
libavcodec version 53.35.0 (external)
Mismatching header version 53.32.2
AUDIO: 48000 Hz, 2 ch, s16le, 384.0 kbit/25.00% (ratio: 48000->192000)
Selected audio codec: [ffac3] afm: ffmpeg (FFmpeg AC-3)
==========================================================================
[AO PCM] File: audiodump.wav (WAVE)
PCM: Samplerate: 48000Hz Channels: Stereo Format s16le
[AO PCM] Info: Faster dumping is achieved with -benchmark -vc null -vo null -ao pcm:fast
[AO PCM] Info: To write WAVE files use -ao pcm:waveheader (default).
AO: [pcm] 48000Hz 2ch s16le (2 bytes per sample)
Video: no video
Starting playback...
A:1209.6 (20:09.5) of 1209.6 (20:09.5)  0.3% 


BENCHMARKs: VC:   0.000s VO:   0.000s A:   3.851s Sys:   0.323s =    4.174s
BENCHMARK%: VC:  0.0000% VO:  0.0000% A: 92.2686% Sys:  7.7314% = 100.0000%

Exiting... (End of file)
2012-11-09 12:58:58,571 - xenonmkv [DEBUG] mplayer decoding to PCM WAV file complete
2012-11-09 12:58:58,571 - xenonmkv [DEBUG] Starting AAC encoding with FAAC
Freeware Advanced Audio Coder
FAAC 1.28

Quantization quality: 150
Bandwidth: 22000 Hz
Object type: Low Complexity(MPEG-2) + M/S
Container format: Transport Stream (ADTS)
Encoding /var/tmp/audiodump.wav to /var/tmp/audiodump.aac
   frame          | bitrate | elapsed/estim | play/CPU | ETA
56700/56700 (100%)|  168.1  |   41.8/41.8   |   28.92x | 0.0  

2012-11-09 12:59:40,904 - xenonmkv [DEBUG] FAAC encoding to AAC audio file complete
2012-11-09 12:59:49,894 - xenonmkv [WARNING] An error occurred while creating an MP4 file with MP4Box; 2 retries left
2012-11-09 13:00:00,860 - xenonmkv [WARNING] An error occurred while creating an MP4 file with MP4Box; 1 retries left
2012-11-09 13:00:16,307 - xenonmkv [WARNING] An error occurred while creating an MP4 file with MP4Box; 0 retries left
2012-11-09 13:00:16,308 - xenonmkv [DEBUG] Cleaning up: deleting /var/tmp/temp_video.h264
2012-11-09 13:00:16,408 - xenonmkv [DEBUG] Cleaning up: deleting /var/tmp/temp_audio.ac3
2012-11-09 13:00:16,412 - xenonmkv [DEBUG] Cleaning up: deleting /var/tmp/audiodump.aac
2012-11-09 13:00:16,418 - xenonmkv [DEBUG] Cleaning up: deleting /var/tmp/audiodump.wav
2012-11-09 13:00:16,480 - xenonmkv [CRITICAL] package: MP4Box could not create file after 3 retries; giving up.
2012-11-09 13:00:16,481 - xenonmkv [DEBUG] Traceback (most recent call last):
  File "./xenonmkv.py", line 242, in main
    mp4box.package()
  File "/root/jbillo-xenonmkv-bc4c612/mp4box.py", line 63, in package
    raise Exception("MP4Box could not create file after %i retries; giving up." % self.args.mp4box_retries)
Exception: MP4Box could not create file after 3 retries; giving up.

[three18ti]

ok, I was able to make those modifications,

Diffs are below, essentially, I added self.log.debug(cmd) to mp4box.py and I commented out the line, cleanup_temp_files(args, log) in xenonmkv.py

this prints the full array of arguments, so I had to manually remove the ' and , (well... search and replace in a text editor.)

The cleaned up version is:

/usr/bin/MP4Box -v output.mp4 -add /var/tmp/temp_video.h264 -fps 23.976 -par 1=1:1 -add /var/tmp/audiodump.aac -tmp /var/tmp/ -itags name=foo

obviously, this is variable. e.g. fps

Interestingly, when I run the command manually, I get a core dump, then when I re run it, it is able to complete, however, the file is nearly double the original. I've opened a support request with the MP4Box guys ( https://sourceforge.net/tracker/?func=detail&aid=3585833&group_id=84101&atid=571739 )

If you have any input I'd like to hear it.

--- a/mp4box.py
+++ b/mp4box.py
@@ -29,7 +29,7 @@ class MP4Box():

                while run_attempts <= self.args.mp4box_retries:
                        cmd = [self.args.tool_paths["mp4box"], "output.mp4", "-add", self.video_path, "-fps", self.video_fps, "-par", "1=" + self.video
-
+                       self.log.debug(cmd)
                        ph = ProcessHandler(self.args)
                        process = ph.start_process(cmd)

diff --git a/xenonmkv.py b/xenonmkv.py
index d5a7796..771ea84 100755
--- a/xenonmkv.py
+++ b/xenonmkv.py
@@ -241,7 +241,7 @@ def main():
        try:
                mp4box.package()
        except Exception as e:
-               cleanup_temp_files(args, log)
+#              cleanup_temp_files(args, log)
                log_exception(log, "package", e)

[jbillo]

The problem with the second attempt at MP4Box is that it tries to append to the existing file. So for a true "run again from scratch" test, you'd have to do the following:

/usr/bin/MP4Box -v output.mp4 -add /var/tmp/temp_video.h264 -fps 23.976 -par 1=1:1 -add /var/tmp/audiodump.aac -tmp /var/tmp/ -itags name=foo
rm output.mp4
/usr/bin/MP4Box -v output.mp4 -add /var/tmp/temp_video.h264 -fps 23.976 -par 1=1:1 -add /var/tmp/audiodump.aac -tmp /var/tmp/ -itags name=foo

I would suspect the second attempt would fail as well in the same manner as the first file.

For a full output log with XenonMKV (well, at least what the utility tries to parse), try the -vv option when converting. The best way to get command line parameters might be to do something like:

self.log.debug("Executing MP4box with arguments: " + ' '.join(cmd))

I will check the code during the next week and add this as a debug option if it's not already there in the development branch.

[jbillo]

Ah, sorry, saw the -vv option during your second output trace. In Vegas at the moment and reading on a small screen. I will have to check further early next week when I return to see what else I can log/debug output. I'd be curious if dropping the "-par 1=1:1" option helped at all.

[rbouqueau]

Hi guys,

I'm a GPAC maintainer. First thank you for reporting, we're always happy to fix bugs.
PS: http://sourceforge.net/tracker/?func=detail&atid=571739&aid=3585833&group_id=84101

Be careful on Unix, the filesystem of /var/tmp and your current dir must be the same. Otherwise the final 'mv' can fail.

The problem with the second attempt at MP4Box is that it tries to append to the existing file.

We have a '-new' option to avoid this :)

On the README.md file, one wrote:

I prefer not to overwrite system libraries, so I copy the application directly into /opt/gpac/0.5.0:

We write in /usr/local/bin (and /usr/local/lib/) whereas the system installs its packets in /usr/bin. So there shouldn't be any overlap. This is a general behaviour for all software on Linux. You can also ask GPAC to install in '/opt/mydir' by calling './configure --prefix=/opt/mydir'.

You're right to compile with no javascript support. MP4Box doesn't need it.

I hope it helps. Otherwise you can come and write on our forums.

Thanks,

Romain

[jbillo]

Thanks Romain. I will make the appropriate adjustments (add -new to MP4Box, add the --prefix option to the README) and see what I can do about detecting/preemptively warning about the filesystem.

Curious, though - what about with tmpfs to ext4, for example? As in, distributions that mount /tmp as tmpfs and then a "mv" to an ext4 path could have interesting side effects.

[jbillo]

As of commit f813703, XenonMKV performs the following in the development branch. This will be pushed to the master branch, probably when I've added more Windows/OS X support and those code paths have been tested a bit better.

• MP4Box file creation command uses -new option always
• Updated information in README.md to reflect that compiling MP4Box from source will put it in /usr/local/bin (this is really the desired behaviour anyway, as on my Ubuntu system PATH starts with /usr/local/bin:/usr/bin
• Temporary files are not deleted when --preserve-temp-files is requested
• ProcessHandler always outputs the exact command line being executed in debug mode (-vv)

I'm not sure exactly what to do with the filesystem concern. XenonMKV does perform a final os.rename call to move the output .mp4 into its appropriate place, which I suspect would throw an IOError if unable to complete and I could address things then.

I'll leave this open for a few more days but I'm not certain there's anything else I can do on the XenonMKV side to resolve. We'll have to wait for the friendly GPAC folks to investigate (they may need a sample, or specific bug report, etc...)

[three18ti]

Romain, What are you considering a "problem" regarding my second attempt? As I stated in GPAC thread ( http://sourceforge.net/tracker/?func=detail&atid=571739&aid=3585833&group_id=84101 ) I was able to create the .mp4 on the second attempt, if the file already existed.

root@shepard:/var/tmp# rm output.mp4 
root@shepard:/var/tmp# /usr/bin/MP4Box output.mp4 -add /var/tmp/temp_video.h264 -fps 23.976 -par 1=1:1 -add /var/tmp/audiodump.aac -tmp /var/tmp/ -itags name=video_file -tmp /var/tmp
AVC-H264 import - frame size 1276 x 720 at 23.976 FPS
AVC Import results: 29000 samples - Slices: 313 I 9265 P 19422 B - 1 SEI - 306 IDR
Stream uses forward prediction - stream CTS offset: 2 frames
AAC import  - sample rate 48000 - MPEG-2 audio - 2 channels
Saving to output.mp4: 0.500 secs Interleaving        
Segmentation fault (core dumped)                        
root@shepard:/var/tmp# ls output.mp4 
output.mp4
root@shepard:/var/tmp# /usr/bin/MP4Box output.mp4 -add /var/tmp/temp_video.h264 -fps 23.976 -par 1=1:1 -add /var/tmp/audiodump.aac -tmp /var/tmp/ -itags name=video_file -tmp /var/tmp
AVC-H264 import - frame size 1276 x 720 at 23.976 FPS
AVC Import results: 29000 samples - Slices: 313 I 9265 P 19422 B - 1 SEI - 306 IDR
Stream uses forward prediction - stream CTS offset: 2 frames
AAC import  - sample rate 48000 - MPEG-2 audio - 2 channels
Saving output.mp4: 0.500 secs Interleaving

When I add the -new flag, MP4Box core dumps even if output.mp4 exists.

Also, I have cd'd to /var/tmp where the files currently exist, and where I assume the work is going on. I also added the "-tmp /var/tmp" flag to attempt to force MP4Box to using /var/tmp

Perhaps I'm confusing the issue since I opened tickets with maintainers of the two projects before I knew if it was a MP4Box or a xenonmkv issue? Would it be better to keep all correspondence on SourceForge?

Thanks,
Jon

PS, at this point, I'm pretty sure the issue is with MP4Box, but as a work around, is there some way to get MP4Box to "incompletely complete" such that output.mp4 is left over and a second run of MP4Box will allow the file to be created? It's a bit of a "hack" for the time being, but it would at least allow me to convert MKVs for the time being.

[three18ti]

@jbillo IDK what happened, but it started working all of a sudden. Thanks for your help on this one.