White space U+000C FORM FEED in template not preserved

[bignose-debian]

Expected Behavior

A template containing the white space character U+000C FORM FEED should render with that white space character preserved in the rendered output.

>>> import pprint
>>> import textwrap
>>> template_content = textwrap.dedent("""\
...     Lorem ipsum, dolor sit amet, consectetur adipiscing elit.
...     \N{FORM FEED}
...     Pellentesque maximus a ligula ut vehicula.
...     """)
>>> pprint.pprint(template_content)
('Lorem ipsum, dolor sit amet, consectetur adipiscing elit.\n'
 '\x0c'
 '\n'
 'Pellentesque maximus a ligula ut vehicula.\n')
>>> import jinja2
>>> jinja2.__version__
'2.10'
>>> template = jinja2.Template(template_content)
>>> pprint.pprint(template.render())
('Lorem ipsum, dolor sit amet, consectetur adipiscing elit.\n'
 '\x0c'
 '\n'
 'Pellentesque maximus a ligula ut vehicula.')

Actual Behavior

The U+000C character is removed and replaced with U+000A LINE FEED.

>>> import pprint
>>> import textwrap
>>> template_content = textwrap.dedent("""\
...     Lorem ipsum, dolor sit amet, consectetur adipiscing elit.
...     \N{FORM FEED}
...     Pellentesque maximus a ligula ut vehicula.
...     """)
>>> pprint.pprint(template_content)
('Lorem ipsum, dolor sit amet, consectetur adipiscing elit.\n'
 '\x0c'
 '\n'
 'Pellentesque maximus a ligula ut vehicula.\n')
>>> import jinja2
>>> jinja2.__version__
'2.9.6'
>>> template = jinja2.Template(template_content)
>>> pprint.pprint(template.render())
('Lorem ipsum, dolor sit amet, consectetur adipiscing elit.\n'
 '\n'
 '\n'
 'Pellentesque maximus a ligula ut vehicula.')

Environment

• Python version: Python 3.5.4
• Jinja version: 2.9.6

[bignose-debian]

This problem still occurs with Jinja 2.10:

>>> pprint.pprint(template_content)
('Lorem ipsum, dolor sit amet, consectetur adipiscing elit.\n'
 '\x0c'
 '\n'
 'Pellentesque maximus a ligula ut vehicula.\n')
>>> import jinja2
>>> jinja2.__version__
'2.10'
>>> template = jinja2.Template(template_content)
>>> pprint.pprint(template.render())
('Lorem ipsum, dolor sit amet, consectetur adipiscing elit.\n'
 '\n'
 '\n'
 'Pellentesque maximus a ligula ut vehicula.')

[davidism]

If this affects you and you can track it down, it would be helpful to see a patch or at least where the behavior is happening. I can't really say if it's intended or not without that, and I'd have to dig through the code myself to see.

[bignose-debian]

If this affects you and you can track it down […]

My attempts to track it down in a debugger session lead to code I can't inspect. The ‘Template.root_render_func’ turns out to not exist in the Jinja source code; maybe it's compiled from text at run time? I don't know how to track it further than that.

[davidism]

Yes, you need to look at the parser/compiler, not the template code. You can ask Jinja to render the Python module instead of the final output and see what's going on.