Video processing fails with 0.8.0

[sebiii]

Hi,

after upgrading to 0.8.0 i found that unfortunately sigal fails while processing a video. With 0.7.0 everything just worked fine.

here the stacktrace:

.ERROR: Failed to process /home/pi/gallery/pictures/Valencia 2013/CIMG4433.AVI
ERROR: Failed to process /home/pi/gallery/mygallery/Valencia 2013/CIMG4433.webm
Traceback (most recent call last):
  File "/usr/local/bin/sigal", line 9, in <module>
    load_entry_point('sigal==0.8.0', 'console_scripts', 'sigal')()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 609, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 589, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 935, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 781, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/sigal/__init__.py", line 133, in build
    gal.build(force=force)
  File "/usr/local/lib/python2.7/dist-packages/sigal/gallery.py", line 528, in build
    processor(files)
  File "/usr/local/lib/python2.7/dist-packages/sigal/gallery.py", line 581, in process_file
    return process_video(src_path, dst_path, settings)
  File "/usr/local/lib/python2.7/dist-packages/sigal/video.py", line 152, in process_video
    fit=settings['thumb_fit'], options=settings['jpg_options'])
  File "/usr/local/lib/python2.7/dist-packages/sigal/video.py", line 130, in generate_thumbnail
    check_subprocess(cmd, source, outname)
  File "/usr/local/lib/python2.7/dist-packages/sigal/video.py", line 55, in check_subprocess
    os.remove(outname)
OSError: [Errno 2] No such file or directory: '/home/pi/gallery/mygallery/Valencia 2013/thumbnails/CIMG4433.jpg'

[sebiii]

Apparently the video was ignored in 0.7.0 (the video is in Motion JPEG format). Is there a plan to support this format? or can the video be also ignored in 0.8?

[saimn]

Hi @sebiii
Before 0.8.0, if the ffmpeg subprocess failed or was interrupted, it was failling silently, and could have created an output file which was not desirable (see #90). So I added a line of code to remove this file, without checking that the file actually exists, which is bad :s. So I will add a check for this.
Concerning the MJPEG support, I don't see why it should work but I haven't tested. Can you run sigal in debug mode (-d) and paste the output to see where it is failing ?

[sebiii]

hi @saimn ,

thank you for the answer! Unfortunately, sigal still fails when trying to open the temporary tumbnail file (see trace below). I have also attached the output of sigal in debug mode.

ERROR: Failed to process /home/pi/gallery/pictures/Valencia 2013/CIMG4433.AVI
ERROR: Failed to process /home/pi/gallery/mygallery/Valencia 2013/CIMG4433.webm
Traceback (most recent call last):
  File "/usr/local/bin/sigal", line 9, in <module>
    load_entry_point('sigal==0.8.0', 'console_scripts', 'sigal')()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 936, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/sigal/__init__.py", line 133, in build
    gal.build(force=force)
  File "/usr/local/lib/python2.7/dist-packages/sigal/gallery.py", line 528, in build
    processor(files)
  File "/usr/local/lib/python2.7/dist-packages/sigal/gallery.py", line 581, in process_file
    return process_video(src_path, dst_path, settings)
  File "/usr/local/lib/python2.7/dist-packages/sigal/video.py", line 154, in process_video
    fit=settings['thumb_fit'], options=settings['jpg_options'])
  File "/usr/local/lib/python2.7/dist-packages/sigal/video.py", line 135, in generate_thumbnail
    image.generate_thumbnail(tmpfile, outname, box, fit, options)
  File "/usr/local/lib/python2.7/dist-packages/sigal/image.py", line 114, in generate_thumbnail
    img = PILImage.open(source)
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2219, in open
    fp = builtins.open(fp, "rb")
IOError: [Errno 2] No such file or directory: u'/home/pi/gallery/mygallery/Valencia 2013/thumbnails/CIMG4433.jpg.tmp.jpg'

INFO - Processing /home/pi/gallery/pictures/Valencia 2013/CIMG4433.AVI
DEBUG - Video size: 640, 480 -> 480, 360
DEBUG - Processing video: ffmpeg -i /home/pi/gallery/pictures/Valencia 2013/CIMG4433.AVI -y -crf 10 -b:v 1.6M -qmin 4 -qmax 63 -vf scale=480:trunc(ow/a/2)*2 /home/pi/gallery/mygallery/Valencia 2013/CIMG4433.webm
ERROR - Failed to process /home/pi/gallery/pictures/Valencia 2013/CIMG4433.AVI
DEBUG - STDOUT:

DEBUG - STDERR:
 ffmpeg version 0.8.13-6:0.8.13-1+rpi1, Copyright (c) 2000-2014 the Libav developers
  built on Jul 29 2014 02:09:52 with gcc 4.6.3
*** THIS PROGRAM IS DEPRECATED ***
This program is only provided for compatibility and will be removed in a future release. Please use avconv instead.
[avi @ 0x1232660] non-interleaved AVI
Input #0, avi, from '/home/pi/gallery/pictures/Valencia 2013/CIMG4433.AVI':
  Metadata:
    creation_time   : 2013-08-29/ 23:13
    encoder         : CASIO EX-S200
  Duration: 00:00:18.61, start: 0.000000, bitrate: 3596 kb/s
    Stream #0.0: Video: mjpeg, yuvj422p, 640x480, 29.97 tbr, 29.97 tbn, 29.97 tbc
    Stream #0.1: Audio: adpcm_ima_wav, 44100 Hz, 1 channels, s16, 176 kb/s
Incompatible pixel format 'yuvj422p' for codec 'libvpx', auto-selecting format 'yuv420p'
[buffer @ 0x1234560] w:640 h:480 pixfmt:yuvj422p
[scale @ 0x1233fe0] w:640 h:480 fmt:yuvj422p -> w:480 h:360 fmt:yuv420p flags:0x4
[libvpx @ 0x1233580] v1.1.0
Output #0, webm, to '/home/pi/gallery/mygallery/Valencia 2013/CIMG4433.webm':
  Metadata:
    creation_time   : 2013-08-29/ 23:13
    encoder         : Lavf53.21.1
    Stream #0.0: Video: libvpx, yuv420p, 480x360, q=4-63, 200 kb/s, 1k tbn, 29.97 tbc
    Stream #0.1: Audio: libvorbis, 44100 Hz, 1 channels, s16
Stream mapping:
  Stream #0.0 -> #0.0
  Stream #0.1 -> #0.1
Press ctrl-c to stop encoding
[webm @ 0x1232ea0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 33 >= 33
av_interleaved_write_frame(): Invalid argument

DEBUG - Process failed, removing file /home/pi/gallery/mygallery/Valencia 2013/CIMG4433.webm
DEBUG - Create thumbnail for video: ffmpeg -i /home/pi/gallery/mygallery/Valencia 2013/CIMG4433.webm -an -r 1 -vframes 1 -y /home/pi/gallery/mygallery/Valencia 2013/thumbnails/CIMG4433.jpg.tmp.jpg
ERROR - Failed to process /home/pi/gallery/mygallery/Valencia 2013/CIMG4433.webm
DEBUG - STDOUT:

DEBUG - STDERR:
 ffmpeg version 0.8.13-6:0.8.13-1+rpi1, Copyright (c) 2000-2014 the Libav developers
  built on Jul 29 2014 02:09:52 with gcc 4.6.3
*** THIS PROGRAM IS DEPRECATED ***
This program is only provided for compatibility and will be removed in a future release. Please use avconv instead.
/home/pi/gallery/mygallery/Valencia 2013/CIMG4433.webm: No such file or directory

DEBUG - Process failed, removing file /home/pi/gallery/mygallery/Valencia 2013/thumbnails/CIMG4433.jpg
Traceback (most recent call last):
  File "/usr/local/bin/sigal", line 9, in <module>
    load_entry_point('sigal==0.8.0', 'console_scripts', 'sigal')()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 936, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/sigal/__init__.py", line 133, in build
    gal.build(force=force)
  File "/usr/local/lib/python2.7/dist-packages/sigal/gallery.py", line 528, in build
    processor(files)
  File "/usr/local/lib/python2.7/dist-packages/sigal/gallery.py", line 581, in process_file
    return process_video(src_path, dst_path, settings)
  File "/usr/local/lib/python2.7/dist-packages/sigal/video.py", line 152, in process_video
    fit=settings['thumb_fit'], options=settings['jpg_options'])
  File "/usr/local/lib/python2.7/dist-packages/sigal/video.py", line 130, in generate_thumbnail
    check_subprocess(cmd, source, outname)
  File "/usr/local/lib/python2.7/dist-packages/sigal/video.py", line 55, in check_subprocess
    os.remove(outname)
OSError: [Errno 2] No such file or directory: '/home/pi/gallery/mygallery/Valencia 2013/thumbnails/CIMG4433.jpg'

[saimn]

Are you sure you have updated the code with the master branch from github ? I can't reproduce it, but I have added an extra check in 6997209. Please check if it is better now.

[sebiii]

Dear @saimn ,

I have tested the code with the extra check and it now skips the videos just fine, but seems to fail in the end at the zip generation (i have zip_media_format = 'resized').

 Collecting files  [####################################]  100%                  
  [#######-----------------------------]  1/5ERROR: Failed to process /home/pi/gallery/pictures/Valencia 2013/CIMG4433.AVI
ERROR: Failed to process /home/pi/gallery/mygallery/Valencia 2013/CIMG4433.webm
  [##############----------------------]  2/5  00:00:08ERROR: Failed to process /home/pi/gallery/pictures/Valencia 2013/CIMG4434.AVI
ERROR: Failed to process /home/pi/gallery/mygallery/Valencia 2013/CIMG4434.webm
  [#####################---------------]  3/5  00:00:06ERROR: Failed to process /home/pi/gallery/pictures/Valencia 2013/CIMG4432.AVI
ERROR: Failed to process /home/pi/gallery/mygallery/Valencia 2013/CIMG4432.webm
  [############################--------]  4/5  00:00:03ERROR: Failed to process /home/pi/gallery/pictures/Valencia 2013/CIMG4435.AVI
ERROR: Failed to process /home/pi/gallery/mygallery/Valencia 2013/CIMG4435.webm
  [####################################]  5/5          ERROR: Failed to process /home/pi/gallery/pictures/Valencia 2013/CIMG4431.AVI
ERROR: Failed to process /home/pi/gallery/mygallery/Valencia 2013/CIMG4431.webm
  [####################################]  5/5
Traceback (most recent call last):
  File "/usr/local/bin/sigal", line 9, in <module>
    load_entry_point('sigal==0.9.0-dev', 'console_scripts', 'sigal')()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 936, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/sigal-0.9.0_dev-py2.7.egg/sigal/__init__.py", line 133, in build
    gal.build(force=force)
  File "/usr/local/lib/python2.7/dist-packages/sigal-0.9.0_dev-py2.7.egg/sigal/gallery.py", line 628, in build
    writer.write(album)
  File "/usr/local/lib/python2.7/dist-packages/sigal-0.9.0_dev-py2.7.egg/sigal/writer.py", line 110, in write
    page = self.template.render(**self.generate_context(album))
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/sigal-0.9.0_dev-py2.7.egg/sigal/themes/colorbox/templates/index.html", line 119, in top-level template code
    {% if album.zip %}
  File "/usr/local/lib/python2.7/dist-packages/jinja2/environment.py", line 397, in getattr
    return getattr(obj, attribute)
  File "/usr/local/lib/python2.7/dist-packages/sigal-0.9.0_dev-py2.7.egg/sigal/gallery.py", line 452, in zip
    archive.write(p.dst_path, os.path.split(p.dst_path)[1])
  File "/usr/lib/python2.7/zipfile.py", line 1049, in write
    st = os.stat(filename)
OSError: [Errno 2] No such file or directory: '/home/pi/gallery/mygallery/Valencia 2013/CIMG4433.webm'

[saimn]

Ah yeah, if the video have not been generated they can't be used in the zip process ... I will also add a check there. The main issue is that currently the failed images/videos are not removed from the list of files of an album, I must to look at this.

[saimn]

@sebiii : I have added the check, can you test again to see if it is ok ? Thanks.

[sebiii]

Dear @saimn ,

the build process now exits without errors, but I have the videos in my gallery as items (which obviously cannot be played).

[saimn]

Yeah, currently the solution is to remove these files from the source directory, but I will try to find a better solution. It was on my todo list but I have created an issue to track this (#112). Thanks for your help on tracking these failures !