Jinja2 fails to iterate models in 0.8.0RC2/3

[jbiason]

In 0.7.9, we could pass a QuerySet directly to Jinja2 and it would render without a problem; with 0.8.0, it seems checking the length resets the cursor, duplicating the displaying data.

For example, with the following model:

from mongoengine import Document
from mongoengine import StringField


class RandomData(Document):
    title = StringField()
    description = StringField()

And the following data:

from mongoengine import connect

from model import RandomData

connect('mongoengine_iterable')

examples = [('This is title', 'This is description'),
            ('Title', 'Description'),
            ('Not description', 'Not title')]

for (title, description) in examples:
    record = RandomData()
    record.title = title
    record.description = description
    record.save()

if you try to render the following template:

{% for record in content %}
    {{ record.title }} {{ record.description }} {% if loop.first -%}
        (This is first)
    {%- endif -%}{%- if loop.last -%}
        (this is last)
    {%- endif -%}
{% endfor %}

with this code:

from jinja2 import Environment
from jinja2 import PackageLoader
from jinja2 import FileSystemLoader

from model import RandomData

from mongoengine import connect

env = Environment(loader=FileSystemLoader(searchpath='./'))

template = env.get_template('display.tmpl')

connect('mongoengine_iterable')
data = RandomData.objects.all()

print template.render(content=data)

The first line will be repeated. If you remove the loop.last check, it works without a problem in 0.8.0.

[jbiason]

Apparently, anything that tries to check the length of the data (including something like loop.length will reset the cursor.

[rozza]

Any idea what jinja is doing in the background to determine the length?

There is no __len__ on querysets anymore due to side effects of having a
len method for example:

list(queryset) # will call __len__ if it exists

[jbiason]

I'm no expert in Jinja2 code (I'm just using MongoEngine to manage my models and Jinja2 as template engine), but it seems that:

Because __len__ was removed, https://github.com/mitsuhiko/jinja2/blob/master/jinja2/runtime.py#L282 now fails and doesn't get the length of the list/content/queryset right at the start. Due that, later it checks the length later at https://github.com/mitsuhiko/jinja2/blob/master/jinja2/runtime.py#L324, converting the data to a tuple, ensuring that there is a length property. But, then again, it's all my guesswork and I can't really see how converting the (Jinja2 internal) iterator would mess with MongoEngine cursor ('cause that's what it seems to be happening: the cursor is being reset somewhere).

I suppose if I cast the result of a queryset to a list will solve the problem, but that also means I'd double my memory usage every time (not counting the process of going over the whole data to convert to a list instead of lazily displaying it).

[rozza]

Hi this is an interesting one!

Here is a cleaner test (which also fails).

    def test_jinja2(self):

        import jinja2
        env = jinja2.Environment()

        class RandomData(Document):
            title = StringField()
            description = StringField()

        RandomData.drop_collection()

        examples = [('A', '1'),
                    ('B', '2'),
                    ('C', '3')]

        for title, description in examples:
            RandomData(title=title, description=description).save()

        tmpl = """
{%- for record in content -%}
    {%- if loop.first -%}{ {%- endif -%}
    "{{ record.title }}": "{{ record.description }}"
    {%- if loop.last -%} }{%- else -%},{% endif -%}
{%- endfor -%}
"""
        ctx = {'content': RandomData.objects}
        template = env.from_string(tmpl)
        rendered = template.render(**ctx)

        self.assertEqual('{"A": "1","B": "2","C": "3"}', rendered)

So when getting the length: https://github.com/mitsuhiko/jinja2/blob/master/jinja2/runtime.py#L324-L333 the iterator returned has the full length 4. Calling tuple(self._iterator) calls mongoengines __iter__() function which if called when iterating it clones the queryset and rewinds it.

I need to look at the implications of removing that code: https://github.com/MongoEngine/mongoengine/blob/master/mongoengine/queryset/queryset.py#L112-L118

[rozza]

removing it causes a couple of scenarios to fail:

https://github.com/MongoEngine/mongoengine/blob/master/tests/queryset/queryset.py#L798-L814
https://github.com/MongoEngine/mongoengine/blob/master/tests/queryset/queryset.py#L3234-L3273

[rozza]

OK I think we'll fix this and anyone expecting the queryset to be too magical will have to use clone and rewind manually - as the current way is broken.

[rozza]

Ok will push an RC4 with a fix