blogofile 0.8b1 fails when files contain UTF-8

[stapelberg]

I just upgraded to blogofile 0.8b1 using pip install --upgrade blogofile.

This broke my site which formerly worked in blogofile 0.7.x (not sure about x anymore). Here is how to reproduce my problem:

midna /tmp $ mkdir clean_dir
midna /tmp $ cd clean_dir 
midna /tmp/clean_dir $ blogofile init broken
_config.py for a bare (do-it-yourself) site written to broken
If you were expecting more, please see `blogofile init -h`
midna /tmp/clean_dir $ cd broken 
midna /tmp/clean_dir/broken $ blogofile build
midna /tmp/clean_dir/broken $ mkdir _templates
midna /tmp/clean_dir/broken $ echo foo > _templates/site.mako
midna /tmp/clean_dir/broken $ blogofile build
midna /tmp/clean_dir/broken $ echo 'maßband' > index.html.mako
midna /tmp/clean_dir/broken $ blogofile build                 
ERROR:blogofile:Fatal build error occured, calling bf.config.build_exception()
Traceback (most recent call last):
  File "/usr/local/bin/blogofile", line 9, in <module>
    load_entry_point('Blogofile==0.8b1', 'console_scripts', 'blogofile')()
  File "/usr/local/lib/python2.7/dist-packages/blogofile/main.py", line 58, in main
    args.func(args)
  File "/usr/local/lib/python2.7/dist-packages/blogofile/main.py", line 388, in do_build
    writer.write_site()
  File "/usr/local/lib/python2.7/dist-packages/blogofile/writer.py", line 50, in write_site
    self.__write_files()
  File "/usr/local/lib/python2.7/dist-packages/blogofile/writer.py", line 127, in __write_files
    util.path_join(root, html_path))
  File "/usr/local/lib/python2.7/dist-packages/blogofile/template.py", line 386, in materialize_template
    template = template_engine(template_name, caller=caller, lookup=lookup)
  File "/usr/local/lib/python2.7/dist-packages/blogofile/template.py", line 116, in __init__
    lookup=self.template_lookup)
  File "/usr/local/lib/python2.7/dist-packages/mako/template.py", line 273, in __init__
    (code, module) = _compile_text(self, text, filename)
  File "/usr/local/lib/python2.7/dist-packages/mako/template.py", line 615, in _compile_text
    generate_magic_comment=template.disable_unicode)
  File "/usr/local/lib/python2.7/dist-packages/mako/template.py", line 597, in _compile
    node = lexer.parse()
  File "/usr/local/lib/python2.7/dist-packages/mako/lexer.py", line 214, in parse
    self.filename,)
  File "/usr/local/lib/python2.7/dist-packages/mako/lexer.py", line 206, in decode_raw_stream
    0, 0, filename)
mako.exceptions.CompileException: Unicode decode operation of encoding 'ascii' failed at line: 0 char: 0
midna /tmp/clean_dir/broken $ env | grep LANG
LANG=de_DE.UTF-8
midna /tmp/clean_dir/broken $ env | grep LC  
LC_MESSAGES=C
LC_TIME=en_DK.UTF-8
midna /tmp/clean_dir/broken $ blogofile --version
Blogofile 0.8b1 -- http://www.blogofile.com -- CPython 2.7.3rc2
midna /tmp/clean_dir/broken $ file index.html.mako 
index.html.mako: UTF-8 Unicode text
midna /tmp/clean_dir/broken $ hd index.html.mako 
00000000  6d 61 c3 9f 62 61 6e 64  0a                       |ma..band.|
00000009

As you can see, my locale setup is correct (and no other program is having utf-8/unicode trouble), and the file contains proper utf-8: http://www.fileformat.info/info/unicode/char/df/index.htm confirms that 0xc3 0x9f is UTF-8 for U+00DF.

I tried replacing open(self.template_name) in template.py:112 with codecs.open(self.template_name, encoding='utf-8'), which seems to fix the problem. I’m not very experienced when it comes to unicode handling in python, though, so I did not submit a pull request and would rather like you to reproduce the issue and then see if the proposed fix is appropriate. Probably the fix also needs to be applied in other places in the code.

[douglatornell]

Thanks for the concise report, Michael.

I think that this is related to #134 and this thread on the mail-list: https://groups.google.com/d/topic/blogofile-discuss/Bh9UDWuf-YM/discussion

I hope to get this resolved this week.

[stapelberg]

It definitely seems like solving #134 would fix this issue, too. I’d be happy to test and I hope that you will release a new version of blogofile so that it will be pip-installable soonish :). Thanks!

[douglatornell]

This is a big enough issue that I will probably do a 0.8b2 release as soon as it is resolved. Especially since you are testing migration of a 0.7 site to 0.8 and I'm really, really interested in hearing how that goes!

[stapelberg]

Well, the photo_gallery controller I have been using (from https://github.com/kstrauser/blogofileplugins/) breaks entirely. I managed to fix the most pressing issues, but still can’t figure out how to add descriptions to image directories without getting an exception. Anyway, that’s a separate issue. If you’re interested in how to reproduce that issue, tell me how to best reach you via email :).

Apart from the photo_gallery, I could not notice any problems.

[douglatornell]

Hmm. I can't seem to reproduce this problem:

(blogofile-dev-2.7)tom:tmp$ export LANG=de_DE.UTF-8
(blogofile-dev-2.7)tom:tmp$ env | grep LANG
LANG=de_DE.UTF-8
(blogofile-dev-2.7)tom:tmp$ blogofile init broken
_config.py for a bare (do-it-yourself) site written to broken
If you were expecting more, please see `blogofile init -h`
(blogofile-dev-2.7)tom:tmp$ cd broken/
(blogofile-dev-2.7)tom:broken$ blogofile build
(blogofile-dev-2.7)tom:broken$ mkdir _templates
(blogofile-dev-2.7)tom:broken$ echo foo > _templates/index.html.mako
(blogofile-dev-2.7)tom:broken$ blogofile build
(blogofile-dev-2.7)tom:broken$ echo 'maßband' > _templates/index.html.mako
(blogofile-dev-2.7)tom:broken$ blogofile build
(blogofile-dev-2.7)tom:broken$ blogofile --version
Blogofile 0.8b1 -- http://www.blogofile.com -- CPython 2.7.3
(blogofile-dev-2.7)tom:broken$ file _templates/index.html.mako 
_templates/index.html.mako: UTF-8 Unicode text
(blogofile-dev-2.7)tom:broken$ uname -a
Darwin tom.local 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

Tried it under Python 3.2 as well, and got the same results.

I'm not questioning that the issue is real. I just need to find a way of reproducing it so that I can explore it and confirm that the resolution is correct.

[stapelberg]

You ran:

(blogofile-dev-2.7)tom:broken$ echo 'maßband' > _templates/index.html.mako

but I ran:

echo 'maßband' > index.html.mako

To be sure, I just installed Debian squeeze in a VM and ran the following commands:

apt-get install python-pip
pip install blogofile
root@squeezevm:~# cd /tmp
root@squeezevm:/tmp# blogofile init broken
_config.py for a bare (do-it-yourself) site written to broken
If you were expecting more, please see `blogofile init -h`
root@squeezevm:/tmp# cd broken
root@squeezevm:/tmp/broken# mkdir _templates
root@squeezevm:/tmp/broken# echo foo > _templates/index.html.mako
root@squeezevm:/tmp/broken# blogofile build
root@squeezevm:/tmp/broken# echo 'maßband' > _templates/index.html.mako
root@squeezevm:/tmp/broken# blogofile build
root@squeezevm:/tmp/broken# echo foo > _templates/index.html.mako
root@squeezevm:/tmp/broken# echo 'maßband' > index.html.mako
root@squeezevm:/tmp/broken# blogofile build
ERROR:blogofile:Fatal build error occured, calling bf.config.build_exception()
Traceback (most recent call last):
  File "/usr/local/bin/blogofile", line 9, in <module>
    load_entry_point('Blogofile==0.8b1', 'console_scripts', 'blogofile')()
  File "/usr/local/lib/python2.6/dist-packages/blogofile/main.py", line 58, in main
    args.func(args)
  File "/usr/local/lib/python2.6/dist-packages/blogofile/main.py", line 388, in do_build
    writer.write_site()
  File "/usr/local/lib/python2.6/dist-packages/blogofile/writer.py", line 50, in write_site
    self.__write_files()
  File "/usr/local/lib/python2.6/dist-packages/blogofile/writer.py", line 127, in __write_files
    util.path_join(root, html_path))
  File "/usr/local/lib/python2.6/dist-packages/blogofile/template.py", line 386, in materialize_template
    template = template_engine(template_name, caller=caller, lookup=lookup)
  File "/usr/local/lib/python2.6/dist-packages/blogofile/template.py", line 116, in __init__
    lookup=self.template_lookup)
  File "/usr/local/lib/python2.6/dist-packages/mako/template.py", line 273, in __init__
    (code, module) = _compile_text(self, text, filename)
  File "/usr/local/lib/python2.6/dist-packages/mako/template.py", line 615, in _compile_text
    generate_magic_comment=template.disable_unicode)
  File "/usr/local/lib/python2.6/dist-packages/mako/template.py", line 597, in _compile
    node = lexer.parse()
  File "/usr/local/lib/python2.6/dist-packages/mako/lexer.py", line 214, in parse
    self.filename,)
  File "/usr/local/lib/python2.6/dist-packages/mako/lexer.py", line 206, in decode_raw_stream
    0, 0, filename)
mako.exceptions.CompileException: Unicode decode operation of encoding 'ascii' failed at line: 0 char: 0

[douglatornell]

You're right, I did run

echo 'maßband' > index.html.mako

and when I did I'm sure I got a different error traceback about a missing template that made me think the above was a typo, missing the _template/.

Just tried again, and successfully reproduced your traceback. Sorry for the confusion.

Now to figure why blogofile is opening the template file at all instead of leaving that to Mako as Mike says in #134.

[douglatornell]

Thanks for helping to define and resolve this issue, Michael.

Note that the test case above now fails due to the lack of a site.mako file, but that's a separate issue. Adding a touch site.mako command to the workflow allows the site with non-ASCII template content to build.