Fix for problems with non-ascii exception messages #43

Closed
wants to merge 2 commits into from

7 participants

@Cito

For instance, PostgreSQL creates error messages in latin-1 or utf-8 depending on how it is configured. The debug toolbar assumes that error messages are always ascii strings or unicode and crashes with a UnicodeDecodeError on such exceptions. This patch makes the debug toolbar handle and display such exceptions properly.

@oxyum

ping?

@mcdonc
Pylons Project member

Bleh sorry for the delay on this. I'm making one more release without this fix (1.0). After that I'll try to review what's going on here and come up with some suitable fix if I don't just go ahead and apply this one.

@uralbash

I have the same problem when I'm working with postgres. I have a Russian environment in the Linux Debian and Postgres gives an error in Russian lang. After that I give error: / pyramid_debugtoolbar / panels / templates / logger.mako ", line 16, in render_body
$ {record ['message']} </ td>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 12: ordinal not in range (128)

As a workaround you can fix pyramid_debugtoolbar / panels / templates / logger.mako

change $ {record ['message']} to
$ {record ['message']. decode ('utf-8')}

@jhnwsk

Same problem here with German umlauts.

I cloned pyramid_debugtoolbar and pulled Cito's fix in. No luck.

uralbash's workaround works for me in:

--- a/pyramid_debugtoolbar/panels/templates/renderings.dbtmako
+++ b/pyramid_debugtoolbar/panels/templates/renderings.dbtmako
@@ -18,7 +18,7 @@
        </thead>
        <tbody> 
                <tr class="pDebugOdd">
-                       <td colspan="2">${rendering['val']|h}</td>
+                       <td colspan="2">${rendering['val'].decode('utf-8')|h}</td>
                </tr>
        </tbody>
        <thead>

my stack trace is

Traceback (most recent call last):
  File "/virtualenv/local/lib/python2.7/site-packages/waitress-0.8.1-py2.7.egg/waitress/channel.py", line 329, in service
    task.service()
  File "/virtualenv/local/lib/python2.7/site-packages/waitress-0.8.1-py2.7.egg/waitress/task.py", line 173, in service
    self.execute()
  File "/virtualenv/local/lib/python2.7/site-packages/waitress-0.8.1-py2.7.egg/waitress/task.py", line 380, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/router.py", line 251, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/router.py", line 227, in invoke_subrequest
    response = handle_request(request)
  File "/virtualenv/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.3-py2.7.egg/pyramid_debugtoolbar/toolbar.py", line 160, in toolbar_tween
    toolbar.process_response(response)
  File "/virtualenv/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.3-py2.7.egg/pyramid_debugtoolbar/toolbar.py", line 56, in process_response
    vars, request=request)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/renderers.py", line 88, in render
    return helper.render(value, None, request=request)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/renderers.py", line 550, in render
    result = renderer(value, system_values)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/mako_templating.py", line 219, in __call__
    reraise(MakoRenderingException(errtext), None, exc_info[2])
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/mako_templating.py", line 211, in __call__
    result = template.render_unicode(**system)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/template.py", line 421, in render_unicode
    as_unicode=True)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 766, in _render
    **_kwargs_for_callable(callable_, data))
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 798, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 824, in _exec_template
    callable_(context, *args, **kwargs)
  File "pyramid_debugtoolbar_templates_toolbar_dbtmako", line 111, in render_body
  File "/virtualenv/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.3-py2.7.egg/pyramid_debugtoolbar/panels/renderings.py", line 46, in content
    }, request=self.request)
  File "/virtualenv/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.3-py2.7.egg/pyramid_debugtoolbar/panels/__init__.py", line 24, in render
    return render(template_name, vars, request=request)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/renderers.py", line 88, in render
    return helper.render(value, None, request=request)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/renderers.py", line 550, in render
    result = renderer(value, system_values)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/mako_templating.py", line 219, in __call__
    reraise(MakoRenderingException(errtext), None, exc_info[2])
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/mako_templating.py", line 211, in __call__
    result = template.render_unicode(**system)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/template.py", line 421, in render_unicode
    as_unicode=True)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 766, in _render
    **_kwargs_for_callable(callable_, data))
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 798, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 824, in _exec_template
    callable_(context, *args, **kwargs)
  File "pyramid_debugtoolbar_panels_templates_renderings_dbtmako", line 31, in render_body
MakoRenderingException: 

Traceback (most recent call last):
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/mako_templating.py", line 211, in __call__
    result = template.render_unicode(**system)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/template.py", line 421, in render_unicode
    as_unicode=True)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 766, in _render
    **_kwargs_for_callable(callable_, data))
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 798, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 824, in _exec_template
    callable_(context, *args, **kwargs)
  File "/virtualenv/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.3-py2.7.egg/pyramid_debugtoolbar/templates/toolbar.dbtmako", line 60, in render_body
    ${panel.content()|n}
  File "/virtualenv/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.3-py2.7.egg/pyramid_debugtoolbar/panels/renderings.py", line 46, in content
    }, request=self.request)
  File "/virtualenv/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.3-py2.7.egg/pyramid_debugtoolbar/panels/__init__.py", line 24, in render
    return render(template_name, vars, request=request)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/renderers.py", line 88, in render
    return helper.render(value, None, request=request)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/renderers.py", line 550, in render
    result = renderer(value, system_values)
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/mako_templating.py", line 219, in __call__
    reraise(MakoRenderingException(errtext), None, exc_info[2])
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/mako_templating.py", line 211, in __call__
    result = template.render_unicode(**system)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/template.py", line 421, in render_unicode
    as_unicode=True)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 766, in _render
    **_kwargs_for_callable(callable_, data))
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 798, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 824, in _exec_template
    callable_(context, *args, **kwargs)
  File "/virtualenv/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.3-py2.7.egg/pyramid_debugtoolbar/panels/templates/renderings.dbtmako", line 21, in render_body
    <td colspan="2">${rendering['val']|h}</td>
MakoRenderingException: 

Traceback (most recent call last):
  File "/virtualenv/local/lib/python2.7/site-packages/pyramid/mako_templating.py", line 211, in __call__
    result = template.render_unicode(**system)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/template.py", line 421, in render_unicode
    as_unicode=True)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 766, in _render
    **_kwargs_for_callable(callable_, data))
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 798, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/virtualenv/local/lib/python2.7/site-packages/mako/runtime.py", line 824, in _exec_template
    callable_(context, *args, **kwargs)
  File "/virtualenv/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.3-py2.7.egg/pyramid_debugtoolbar/panels/templates/renderings.dbtmako", line 21, in render_body
    <td colspan="2">${rendering['val']|h}</td>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 76: ordinal not in range(128)
@brodul

Hi,

Please see this pull request Pylons/pyramid_debugtoolbar#108.
This should fix the problems with utf-8 encoded exception messages. But the ones with latin-1 persist.

I recommend we introduce a new setting to the debug toolbar that will handle the decoding of database errors separately. Since we can not catch all the supported charsets.

An example of supported charsets by PostgreSQL
http://www.postgresql.org/docs/8.2/static/multibyte.html#CHARSET-TABLE

I recommend we should use utf-8 as the default.

Feedback highly appreciated.

@mmerickel
Pylons Project member

Just gonna close this. Sorry we never got to it but I'm hoping it's fixed. If not please bring the PR up to date with master and we'll get it fixed.

@mmerickel mmerickel closed this Apr 21, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment