Skip to content
This repository

Encoding Problems in Jinja2 Templates #146

Open
wulfraem opened this Issue September 04, 2012 · 3 comments

3 participants

wulfraem Jorge Araya Navarro Will Sams
wulfraem

Hi,

I'm working with a combination of IronPython and jinja2 but keep getting encoding problems from time to time when outputting special characters like 'ä' or returning such from python methods into jinja context.

For example the template:
{%- set test= 'ä' %}
{{ test }}

Can produce exceptions (System.Text.DecoderFallbackException) with the message 'Unable to translate bytes [E4] at index 0 from specified code page to Unicode.'

Adding a space character to the end of the second line can bypass this in some cases but this approach can't be used in larger templaes with a structure that changes depending on inputted data.
It's also possible to output such characters by htmlescaping them via the '|e' filter but again, this cannot be applied anywhere.

I would be grateful for any hints how to solve this problem as it puts a damper on an otherwise good and useful template language...

Jorge Araya Navarro

I'm having this problem too but in Pelican, which uses Jinja2 for rendering html from markdown text. Here is the debug output:

-> writing /home/jorge/Documentos/SWEETGEEKLIFELOVEBLOG/output/2010/04/juan-el-sabio-de-la-taza.html
CRITICAL: 'utf8' codec can't decode byte 0xed in position 140: invalid continuation byte
Traceback (most recent call last):

[...]

  File "/usr/lib/python2.7/site-packages/pelican-2.8.0-py2.7.egg/pelican/generators.py", line 59, in get_template
    self._templates[name] = self._env.get_template(name + '.html')
  File "/usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg/jinja2/environment.py", line 719, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg/jinja2/environment.py", line 693, in _load_template
    template = self.loader.load(self, name, globals)
  File "/usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg/jinja2/loaders.py", line 115, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg/jinja2/loaders.py", line 374, in get_source
    return loader.get_source(environment, template)
  File "/usr/lib/python2.7/site-packages/Jinja2-2.6-py2.7.egg/jinja2/loaders.py", line 169, in get_source
    contents = f.read().decode(self.encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xed in position 140: invalid continuation byte

Maybe one solution is to bypass this error by catching the exception on line 169 in /jinja2/loaders.py because the text is already in UTF-8. However, I opened the file as in such line "contents = f.read().decode(self.encoding)" but instead I use "utf-8" and "latin-1" for the self.encoding argument and any exception was raised!

actually I'm very confusing, I expected an error with my test...

wulfraem

Hm,
I tried to add a catch block for this and even to set the encoding to 'utf-8' (hardcoded, switched self.encoding with 'utf-8') but to no prevail.
I'm still having the same error... :/

ps: Sorry about closing and reopenening the issue. I hit the wrong button while submitting my post. Can someone undo this?

wulfraem wulfraem closed this October 01, 2012
wulfraem wulfraem reopened this October 01, 2012
Will Sams

http://ironpython.codeplex.com/workitem/29505

Its an old issue related to the IronPython. As Jeff Hardy pointed out, replace to_string = unicode with to_string = lambda x: unicode(x) near line 28 of runtime.py in jinja2.

Will Sams WillSams referenced this issue from a commit October 12, 2012
Will Sams Update jinja2/runtime.py to handle IronPython unicode strings issue.
IronPython fails to correctly reference unicode strings containing non-ASCII characters.

This covers [issue #146][1]

[1]:  #146
9e333b7
Will Sams WillSams referenced this issue from a commit October 12, 2012
Will Sams Update to jinja2/runtime.py to handle IronPython unicode strings issue.
IronPython fails to correctly reference unicode strings containing non-ASCII characters.  This covers issue #146.
b9ced43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.