Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Traceback line numbers broken on Jinja 2.10.2 and newer + Python 3.7 #1104

Closed
akx opened this issue Nov 13, 2019 · 5 comments · Fixed by #1110
Closed

Traceback line numbers broken on Jinja 2.10.2 and newer + Python 3.7 #1104

akx opened this issue Nov 13, 2019 · 5 comments · Fixed by #1110
Milestone

Comments

@akx
Copy link

@akx akx commented Nov 13, 2019

The 3a86ef3 commit breaks traceback line numbers on Python 3.7. (On Django, this breaks the debug traceback view altogether.)

Refs:

Thanks to @tomhamiltonstubber for the original minimal example.

Expected Behavior

Tracebacks should have correct line numbers, and source lines should be retrievable.

Actual Behavior

Tracebacks within Jinja templates have incorrect line numbers, and source is not retrievable.

Python Code

import jinja2

l = jinja2.FileSystemLoader('.')
e = jinja2.Environment(loader=l)
t = e.get_template('index.jinja')
print(t.render({'foo': [8]}))

Template Code

{% block content %}
  {% for x, y in foo %}
    {{ x }}
  {% endfor %}
{% endblock %}

Full Traceback

Traceback (most recent call last):
  File "test.py", line 7, in <module>
    print(t.render({'foo': [8]}))
  File "jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "./index.jinja", line 1, in top-level template code
    {% block content %}
  File "jinja2/environment.py", line 1005, in render
    return concat(self.root_render_func(self.new_context(vars)))
  File "./index.jinja", line 10, in root
  File "./index.jinja", line 19, in block_content
TypeError: cannot unpack non-iterable int object

There are only 5 lines in index.jinja and the tracebacks point to line 10 and 19.

If one cheats Jinja to use the old hack on Python 3.7, things work:

import sys
sys.version_info = (3, 6)  # don't do this at home
import jinja2

l = jinja2.FileSystemLoader('.')
e = jinja2.Environment(loader=l)
t = e.get_template('index.jinja')

print(t.render({'foo': [8]}))
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    print(t.render({'foo': [8]}))
  File "jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "./index.jinja", line 1, in top-level template code
    {% block content %}
  File "./index.jinja", line 2, in block "content"
    {% for x, y in foo %}
TypeError: cannot unpack non-iterable int object

Your Environment

  • Python version: Python 3.7.4 (default, Oct 12 2019, 18:55:28) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin
  • Jinja version: 2.10.3
@akx akx changed the title Traceback line numbers broken on Jinja 2.10.x + Python 3.7 Traceback line numbers broken on Jinja 2.10.2 and newer + Python 3.7 Nov 13, 2019
@davidism

This comment has been minimized.

Copy link
Member

@davidism davidism commented Nov 13, 2019

PR #1051 from @vstinner introduced the change. Looks like some of that debug init code might still need to run even if we can use 3.7's native set_next method.

@vstinner

This comment has been minimized.

Copy link
Contributor

@vstinner vstinner commented Nov 14, 2019

PR #1051 from @vstinner introduced the change. Looks like some of that debug init code might still need to run even if we can use 3.7's native set_next method.

Well, I mostly cared about the test suite not to crash :-) I was happy when the test suite passed, but I didn't check the exact traceback ;-)

@davidism davidism added this to the 2.11.0 milestone Nov 20, 2019
@davidism

This comment has been minimized.

Copy link
Member

@davidism davidism commented Nov 22, 2019

I have absolutely no idea why this is happening. make_traceback raises the patched frames with the correct line number and location information, but then Python just prints the original frames.

@davidism

This comment has been minimized.

Copy link
Member

@davidism davidism commented Nov 23, 2019

Doing some work on this. Ditching most of jinja2.debug and only supporting 3.7+, this does the right thing:

def new_make_traceback(exc_info, source_hint=None):
    exc_type, exc_value, tb = exc_info
    tb_prev = None

    while tb is not None:
        t = tb.tb_frame.f_globals.get("__jinja_template__")
        tb_next = tb.tb_next

        if t is not None:
            lineno = t.get_corresponding_lineno(tb.tb_lineno)
            fake_tb = fake_exc_info(exc_info[:2] + (tb,), t.filename, lineno)[2]
            tb = TracebackType(tb_next, fake_tb.tb_frame, tb.tb_lasti, lineno)

            if tb_prev is not None:
                tb_prev.tb_next = tb

        tb_prev = tb
        tb = tb_next

    return exc_info

Need to support TemplateSyntaxError and <= 3.6, but it's a start. Need to compare what gets created on 3.6 to this to figure out what's going on.

Also turned out that it would have failed on 3.8 for another reason: CodeType had a new 2nd argument inserted, posonlyargcount.

@vstinner

This comment has been minimized.

Copy link
Contributor

@vstinner vstinner commented Dec 3, 2019

Cool! Thanks for the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.