Unicode filenames break server

[kevinlondon]

If you use Unicode characters for filenames, it causes the server not to run. Locally, I added filename = filename.decode("utf-8") at the start of the __init__ method for Media in the gallery.py file and that resolved the issue.

Traceback (most recent call last):
  File "/usr/home/kevin/.virtualenvs/thumbtest/bin/sigal", line 11, in <module>
    sys.exit(main())
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/site-packages/sigal/__init__.py", line 154, in main
    parser.dispatch()
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/site-packages/argh/helpers.py", line 53, in dispatch
    return dispatch(self, *args, **kwargs)
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/site-packages/argh/dispatching.py", line 125, in dispatch
    for line in lines:
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/site-packages/argh/dispatching.py", line 202, in _execute_command
    for line in result:
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/site-packages/argh/dispatching.py", line 185, in _call
    result = args.function(*positional, **keywords)
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/site-packages/sigal/__init__.py", line 108, in build
    gal = Gallery(settings, theme=theme, ncpu=ncpu)
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/site-packages/sigal/gallery.py", line 452, in __init__
    album = Album(relpath, settings, dirs, files, self)
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/site-packages/sigal/gallery.py", line 205, in __init__
    media = Image(f, self.path, settings)
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/site-packages/sigal/gallery.py", line 129, in __init__
    super(Image, self).__init__(filename, path, settings)
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/site-packages/sigal/gallery.py", line 74, in __init__
    self.dst_path = join(settings['destination'], path, filename)
  File "/usr/home/kevin/.virtualenvs/thumbtest/lib/python2.7/posixpath.py", line 80, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 6: ordinal not in range(128)

[saimn]

Hmm, everything here should already be unicode ... could you run in debug mode (sigal build -d) and paste the output, and maybe put a print in the __init__ to print settings['destination'], path, filename ?

edit: what OS and python version are you using ?

[braoult]

Title does not seem to accept UTF-8, anyway (but filenames do).
Ubuntu 14.04, sigal 0.8.1, Python 2.7.6.

With
title = "à"
I get:

br@lorien:~/www/w/sigal$ sigal build
-> Sample Album 2 : 9 images
-> Album 1 àvec UTF-8 : 9 images

Traceback (most recent call last):
File "/usr/local/bin/sigal", line 9, in
load_entry_point('sigal==0.8.1', 'console_scripts', 'sigal')()
File "/usr/local/lib/python2.7/dist-packages/click-3.3-py2.7.egg/click/core.py", line 610, in call
return self.main(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/click-3.3-py2.7.egg/click/core.py", line 590, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click-3.3-py2.7.egg/click/core.py", line 936, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click-3.3-py2.7.egg/click/core.py", line 782, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click-3.3-py2.7.egg/click/core.py", line 416, in invoke
return callback(_args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/sigal-0.8.1-py2.7.egg/sigal/init.py", line 133, in build
gal.build(force=force)
File "/usr/local/lib/python2.7/dist-packages/sigal-0.8.1-py2.7.egg/sigal/gallery.py", line 549, in build
writer.write(album)
File "/usr/local/lib/python2.7/dist-packages/sigal-0.8.1-py2.7.egg/sigal/writer.py", line 110, in write
page = self.template.render(**self.generate_context(album))
File "/usr/local/lib/python2.7/dist-packages/Jinja2-2.7.3-py2.7.egg/jinja2/environment.py", line 969, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/local/lib/python2.7/dist-packages/Jinja2-2.7.3-py2.7.egg/jinja2/environment.py", line 742, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/sigal-0.8.1-py2.7.egg/sigal/themes/galleria/templates/index.html", line 22, in top-level template code
[ removed html code here ]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

[saimn]

Could you run again sigal with sigal build -fd -n 1 to get a useful traceback ? Thanks.

[braoult]

Here is the log with fd -n 1. I did escape 2 lines at the bottom (did not find a way to attach the file directly).

INFO: Reading settings ...
DEBUG: Settings file: sigal.conf.py
DEBUG: Rewrite destination : forPatrick -> /export/home/br/www/w/sigal/forPatrick
DEBUG: Settings:
{'albums_sort_reverse': False,
'autorotate_images': True,
'colorbox_column_size': 4,
'copy_exif_data': False,
'destination': u'/export/home/br/www/w/sigal/forPatrick',
'files_to_copy': (),
'google_analytics': '',
'ignore_directories': [],
'ignore_files': [],
'img_processor': 'ResizeToFit',
'img_size': (800, 600),
'index_in_url': False,
'jpg_options': {'optimize': True, 'progressive': True, 'quality': 85},
'keep_orig': False,
'links': '',
'locale': '',
'make_thumbs': True,
'medias_sort_attr': 'filename',
'medias_sort_reverse': False,
'orig_dir': 'original',
'orig_link': False,
'plugin_paths': [],
'plugins': [],
'source': u'/home/br/www/t/pictures',
'theme': 'galleria',
'thumb_dir': 'thumbnails',
'thumb_fit': True,
'thumb_prefix': '',
'thumb_size': (280, 210),
'thumb_suffix': '',
'title': '\xc3\xa0',
'video_size': (480, 360),
'webm_options': ['-crf', '10', '-b:v', '1.6M', '-qmin', '4', '-qmax', '63'],
'write_html': True,
'zip_gallery': False,
'zip_media_format': 'resized'}
INFO: Source : /home/br/www/t/pictures
INFO: Destination : /export/home/br/www/w/sigal/forPatrick
INFO: Theme : galleria
DEBUG: Plugin paths: []
INFO: Using 1 cores
DEBUG: Albums:
[(path=u'Sample Album 2', title=u'Sample Album 2'), (path=u'Album 1 \xe0vec UTF-8', title=u'Album 1 \xe0vec UTF-8'), (path='.', title=u'pictures')]
-> Sample Album 2 : 9 images
INFO: Processing /home/br/www/t/pictures/Sample Album 2/Angelfish.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Sample Album 2/Angelfish.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Sample Album 2/thumbnails/Angelfish.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Sample Album 2/Big-Eye Squirrel Fish.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Sample Album 2/Big-Eye Squirrel Fish.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Sample Album 2/thumbnails/Big-Eye Squirrel Fish.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Sample Album 2/Blue-Banded Sea Perch.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Sample Album 2/Blue-Banded Sea Perch.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Sample Album 2/thumbnails/Blue-Banded Sea Perch.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Sample Album 2/Breaching, Humpback Whale.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Sample Album 2/Breaching, Humpback Whale.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Sample Album 2/thumbnails/Breaching, Humpback Whale.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Sample Album 2/Bright Idea, Bottlenose Dolphin.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Sample Album 2/Bright Idea, Bottlenose Dolphin.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Sample Album 2/thumbnails/Bright Idea, Bottlenose Dolphin.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Sample Album 2/Caribbean Blue Angelfish, Gulf Of Mexico.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Sample Album 2/Caribbean Blue Angelfish, Gulf Of Mexico.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Sample Album 2/thumbnails/Caribbean Blue Angelfish, Gulf Of Mexico.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Sample Album 2/False Clown Anemonefish, Bali, Indonesia.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Sample Album 2/False Clown Anemonefish, Bali, Indonesia.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Sample Album 2/thumbnails/False Clown Anemonefish, Bali, Indonesia.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Sample Album 2/Feeling Crabby_.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Sample Album 2/Feeling Crabby_.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Sample Album 2/thumbnails/Feeling Crabby_.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Sample Album 2/Flame Angelfish.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Sample Album 2/Flame Angelfish.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Sample Album 2/thumbnails/Flame Angelfish.jpg (JPEG)
-> Album 1 àvec UTF-8 : 9 images
INFO: Processing /home/br/www/t/pictures/Album 1 àvec UTF-8/Snow - 01.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/Snow - 01.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/thumbnails/Snow - 01.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Album 1 àvec UTF-8/Snow - 02.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/Snow - 02.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/thumbnails/Snow - 02.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Album 1 àvec UTF-8/Snow - 03.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/Snow - 03.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/thumbnails/Snow - 03.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Album 1 àvec UTF-8/Snow - 04.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/Snow - 04.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/thumbnails/Snow - 04.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Album 1 àvec UTF-8/Snow - 05.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/Snow - 05.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/thumbnails/Snow - 05.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Album 1 àvec UTF-8/Snow - 06.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/Snow - 06.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/thumbnails/Snow - 06.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Album 1 àvec UTF-8/Snow - 07.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/Snow - 07.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/thumbnails/Snow - 07.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Album 1 àvec UTF-8/Snow - 08.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/Snow - 08.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/thumbnails/Snow - 08.jpg (JPEG)
INFO: Processing /home/br/www/t/pictures/Album 1 àvec UTF-8/Snow - 09.jpg
DEBUG: Processor: ResizeToFit
DEBUG: Save resized image to /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/Snow - 09.jpg (JPEG)
DEBUG: Save thumnail image: /export/home/br/www/w/sigal/forPatrick/Album 1 àvec UTF-8/thumbnails/Snow - 09.jpg (JPEG)
INFO: Album (path='.', title=u'pictures') is empty
INFO: Theme : /usr/local/lib/python2.7/dist-packages/sigal-0.8.1-py2.7.egg/sigal/themes/galleria
INFO: Output album : (path=u'Sample Album 2', title=u'Sample Album 2')
Traceback (most recent call last):
File "/usr/local/bin/sigal", line 9, in
load_entry_point('sigal==0.8.1', 'console_scripts', 'sigal')()
File "/usr/local/lib/python2.7/dist-packages/click-3.3-py2.7.egg/click/core.py", line 610, in call
return self.main(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/click-3.3-py2.7.egg/click/core.py", line 590, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click-3.3-py2.7.egg/click/core.py", line 936, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click-3.3-py2.7.egg/click/core.py", line 782, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click-3.3-py2.7.egg/click/core.py", line 416, in invoke
return callback(_args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/sigal-0.8.1-py2.7.egg/sigal/init.py", line 133, in build
gal.build(force=force)
File "/usr/local/lib/python2.7/dist-packages/sigal-0.8.1-py2.7.egg/sigal/gallery.py", line 549, in build
writer.write(album)
File "/usr/local/lib/python2.7/dist-packages/sigal-0.8.1-py2.7.egg/sigal/writer.py", line 110, in write
page = self.template.render(**self.generate_context(album))
File "/usr/local/lib/python2.7/dist-packages/Jinja2-2.7.3-py2.7.egg/jinja2/environment.py", line 969, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/local/lib/python2.7/dist-packages/Jinja2-2.7.3-py2.7.egg/jinja2/environment.py", line 742, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/sigal-0.8.1-py2.7.egg/sigal/themes/galleria/templates/index.html", line 22, in top-level template code
<h1><a href="{{ album.index_url }}"> {{ index_title }}</a></h1>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

[saimn]

@braoult It should be ok now, but if I understand correctly, you set the title (with special characters) in the settings file ? If this is the case you should use the unicode notation (u" ... ") because the settings file is a python file.