Traceback line numbers broken on Jinja 2.10.2 and newer + Python 3.7

[akx]

The 3a86ef3 commit breaks traceback line numbers on Python 3.7. (On Django, this breaks the debug traceback view altogether.)

Refs:

• Django-Jinja issue Error with template errors with Django and debug niwinz/django-jinja#238
• Django issue https://code.djangoproject.com/ticket/30405
• Django mitigation Fixed #30405 -- Fixed source code mismatch crash in ExceptionReporter.  django/django#11886

Thanks to @tomhamiltonstubber for the original minimal example.

Expected Behavior

Tracebacks should have correct line numbers, and source lines should be retrievable.

Actual Behavior

Tracebacks within Jinja templates have incorrect line numbers, and source is not retrievable.

Python Code

import jinja2

l = jinja2.FileSystemLoader('.')
e = jinja2.Environment(loader=l)
t = e.get_template('index.jinja')
print(t.render({'foo': [8]}))

Template Code

{% block content %}
  {% for x, y in foo %}
    {{ x }}
  {% endfor %}
{% endblock %}

Full Traceback

Traceback (most recent call last):
  File "test.py", line 7, in <module>
    print(t.render({'foo': [8]}))
  File "jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "./index.jinja", line 1, in top-level template code
    {% block content %}
  File "jinja2/environment.py", line 1005, in render
    return concat(self.root_render_func(self.new_context(vars)))
  File "./index.jinja", line 10, in root
  File "./index.jinja", line 19, in block_content
TypeError: cannot unpack non-iterable int object

There are only 5 lines in index.jinja and the tracebacks point to line 10 and 19.

If one cheats Jinja to use the old hack on Python 3.7, things work:

import sys
sys.version_info = (3, 6)  # don't do this at home
import jinja2

l = jinja2.FileSystemLoader('.')
e = jinja2.Environment(loader=l)
t = e.get_template('index.jinja')

print(t.render({'foo': [8]}))

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    print(t.render({'foo': [8]}))
  File "jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "./index.jinja", line 1, in top-level template code
    {% block content %}
  File "./index.jinja", line 2, in block "content"
    {% for x, y in foo %}
TypeError: cannot unpack non-iterable int object

Your Environment

• Python version: Python 3.7.4 (default, Oct 12 2019, 18:55:28) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin
• Jinja version: 2.10.3

[davidism]

PR #1051 from @vstinner introduced the change. Looks like some of that debug init code might still need to run even if we can use 3.7's native set_next method.

[vstinner]

PR #1051 from @vstinner introduced the change. Looks like some of that debug init code might still need to run even if we can use 3.7's native set_next method.

Well, I mostly cared about the test suite not to crash :-) I was happy when the test suite passed, but I didn't check the exact traceback ;-)

[davidism]

I have absolutely no idea why this is happening. make_traceback raises the patched frames with the correct line number and location information, but then Python just prints the original frames.

[davidism]

Doing some work on this. Ditching most of jinja2.debug and only supporting 3.7+, this does the right thing:

def new_make_traceback(exc_info, source_hint=None):
    exc_type, exc_value, tb = exc_info
    tb_prev = None

    while tb is not None:
        t = tb.tb_frame.f_globals.get("__jinja_template__")
        tb_next = tb.tb_next

        if t is not None:
            lineno = t.get_corresponding_lineno(tb.tb_lineno)
            fake_tb = fake_exc_info(exc_info[:2] + (tb,), t.filename, lineno)[2]
            tb = TracebackType(tb_next, fake_tb.tb_frame, tb.tb_lasti, lineno)

            if tb_prev is not None:
                tb_prev.tb_next = tb

        tb_prev = tb
        tb = tb_next

    return exc_info

Need to support TemplateSyntaxError and <= 3.6, but it's a start. Need to compare what gets created on 3.6 to this to figure out what's going on.

Also turned out that it would have failed on 3.8 for another reason: CodeType had a new 2nd argument inserted, posonlyargcount.

[vstinner]

Cool! Thanks for the fix.