Timelapse files are corrupt - segmentation fault during avconv

[stansley]

MP4 files are being produced at the end of the night, but the files won't play on any device. Omxplayer just gives the "have a nice day" message. Running timelapse.sh manually, I see the following type of failure during the avconv, and a presumably partial MP4 is created. Thoughts?

./scripts/timelapse.sh: line 29: 6502 Segmentation fault avconv -y -f image2 -r 25 -i images/$1/%04d.$EXTENSION -vcodec libx264 -b:v 2000k -pix_fmt yuv420p images/$1/allsky-$1.mp4

Last few lines of the console output:
...
`mv: 'images/20180805/1965.jpg' and 'images/20180805/1965.jpg' are the same file
'images/20180805/startrails.jpg' -> 'images/20180805/1966.jpg'
ffmpeg version 3.2.10-1deb9u1+rpt1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 6.3.0 (Raspbian 6.3.0-18+rpi1) 20170516
configuration: --prefix=/usr --extra-version='1deb9u1+rpt1' --toolchain=hardened --libdir=/usr/lib/arm-linux-gnueabihf --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx-rpi --enable-mmal --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
libavutil 55. 34.101 / 55. 34.101
libavcodec 57. 64.101 / 57. 64.101
libavformat 57. 56.101 / 57. 56.101
libavdevice 57. 1.100 / 57. 1.100
libavfilter 6. 65.100 / 6. 65.100
libavresample 3. 1. 0 / 3. 1. 0
libswscale 4. 2.100 / 4. 2.100
libswresample 2. 3.100 / 2. 3.100
libpostproc 54. 1.100 / 54. 1.100
Input #0, image2, from 'images/20180805/%04d.jpg':
Duration: 00:01:18.64, start: 0.000000, bitrate: N/A
Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 1936x1096 [SAR 1:1 DAR 242:137], 25 fps, 25 tbr, 25 tbn, 25 tbc
[swscaler @ 0xeabe80] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 0xea1a10] using SAR=1/1
[libx264 @ 0xea1a10] using cpu capabilities: ARMv6 NEON
[libx264 @ 0xea1a10] profile High, level 4.2
[libx264 @ 0xea1a10] 264 - core 148 r2748 97eaef2 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=2000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'images/20180805/allsky-20180805.mp4':
Metadata:
encoder : Lavf57.56.101
Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1936x1096 [SAR 1:1 DAR 242:137], q=-1--1, 2000 kb/s, 25 fps, 12800 tbn, 25 tbc
Metadata:
encoder : Lavc57.64.101 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 0 vbv_delay: -1
Stream mapping:
Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
Press [q] to stop, [?] for help
frame= 53 fps=1.7 q=39.0 size= 169kB time=00:00:00.00 bitrate=17748000.0kbits/s speed=2.5frame= 54 fps=1.7 q=37.0 size= 175kB time=00:00:00.04 bitrate=35821.3kbits/s speed=0.0012frame= 56 fps=1.7 q=45.0 size= 190kB time=00:00:00.12 bitrate=12988.7kbits/s speed=0.0036frame= 58 fps=1.7 q=43.0 size= 205kB time=00:00:00.20 bitrate=8389.6kbits/s speed=0.00592frame= 59 fps=1.7 q=44.0 size= 222kB time=00:00:00.24 bitrate=7573.0kbits/s speed=0.00694frame= 60 fps=1.7 q=43.0 size= 225kB time=00:00:00.28 bitrate=6578.1kbits/s speed=0.00797frame= 62 fps=1.7 q=44.0 size= 229kB time=00:00:00.36 bitrate=5213.6kbits/s speed=0.0101xframe= 64 fps=1.8 q=40.0 size= 233kB time=00:00:00.44 bitrate=4337.4kbits/s speed=0.0122xframe= 66 fps=1.8 q=42.0 size= 238kB time=00:00:00.52 bitrate=3750.1kbits/s speed=0.0142x
./scripts/timelapse.sh: line 29: 6502 Segmentation fault avconv -y -f image2 -r 25 -i images/$1/%04d.$EXTENSION -vcodec libx264 -b:v 2000k -pix_fmt yuv420p images/$1/allsky-$1.mp4

• Timelapse was created
  pi@allsky:~/allsky $
  `

[thomasjacquin]

I think this may be the issue:

'images/20180805/startrails.jpg' -> 'images/20180805/1966.jpg'

It is including the startrails image into the video. I'm not sure that's the issue but can you try moving the startrails.jpg file somewhere else and regenerate the video?

[stansley]

Thanks Thomas -- good catch! With that file removed, I still get corrupt MP4s though. Running timelapse.sh manually on the same images as above, it gets some way into the process, but drops out on frame 317 with Illegal Instruction. I'm looking to see if there is an update to avconv, perhaps that can help.

...
`frame= 315 fps=1.2 q=33.0 size= 2451kB time=00:00:10.48 bitrate=1916.2kbits/frame= 317 fps=1.2 q=33.0 size= 2480kB time=00:00:10.56 bitrate=1923.6kbits/./timelapse.sh: line 29: 7493 Illegal instruction avconv -y -f image2 -r 25 -i images/$1/%04d.$EXTENSION -vcodec libx264 -b:v 2000k -pix_fmt yuv420p images/$1/allsky-$1.mp4

• Timelapse was created
  pi@allsky:~/allsky/scripts $
  `

[thomasjacquin]

Is it always breaking on frame 317? If so, make sure the file is not corrupted.
Also, what ASI camera model are you using? Large sensors produce big images and there's a memory limit that avconv will reach if the output video is getting too big.

[stansley]

It's not always that frame -- re-running results in failures at different frames. Sometimes the error is illegal instruction, and sometimes segmentation fault.

Camera is ZWO ASI290MM-mini, so 1936 × 1096 pixels. Files are 700-900kB or so. If those are too big, do you know a good way to downsize the files as a batch, or is it possible to alter the parameters being passed to avconv? The man page for the latter is a little overwhelming...

[thomasjacquin]

The first thing you could do is to run the top command while the timelapse is generating and watch the memory usage. That may help to see if it's failing because of memory limits.

To reduce the files on the fly (overwrite them) you can use the mogrify function of image magick.
https://www.imagemagick.org/script/mogrify.php

in scripts/timelapse.sh, on line 24, you can add something like that:
magick mogrify -resize 50% images/$1/*.$EXTENSION

I haven't tested it but it should resize your images to 50% of their width and height (4 times smaller) before trying to run them in avconv. You can of course use a different percentage.

[stansley]

Thanks so much Thomas -- very helpful! Well guess what, bizarrely, after another run remotely (I've been operating both remotely and locally through all previous attempts with the same results), with a manual ./scripts/timelapse in one PuTTY window, and top in another, and after a new error I've not seen before, IT IS NOW WORKING with no errors at all!

Top is reporting bouncing around 30kB mid-run. And steady. After never seeing it get through more than a few 10s of frames, and once maybe 150, I'm now seeing full runs through ~2000 frames of full-size 2Mpx images from the ASI290MM-mini. Great! Thanks again! Not sure what we can put this down to... I'll report back if it reoccurs.

Example result: https://1drv.ms/v/s!Av6xS2tSviSBicxVXkRBqG64eaYHDQ

[thomasjacquin]

Excellent! Keep me posted if it happens again. Nice video of the milky way. Pretty busy aerial traffic too!