2.9 regression when assigning a variable inside a loop #641

Open
ThiefMaster opened this Issue Jan 7, 2017 · 32 comments

Projects

None yet

7 participants

@ThiefMaster
Member

2.9:

>>> jinja2.Template('{% set a = -1 %}{% for x in range(5) %}[{{ a }}:{% set a = x %}{{ a }}] {% endfor %}{{ a }}').render()
u'[:0] [:1] [:2] [:3] [:4] -1'

2.8:

>>> jinja2.Template('{% set a = -1 %}{% for x in range(5) %}[{{ a }}:{% set a = x %}{{ a }}] {% endfor %}{{ a }}').render()
u'[-1:0] [0:1] [1:2] [2:3] [3:4] -1'

Originally reported on IRC:

A change in jinja2 scoping appears to affect me, and I'm unsure of the correct fix. Specifically the problem is the assignment of year here: https://github.com/kennethlove/alex-gaynor-blog-design/blob/551172/templates/archive.html#L13-L24

@mitsuhiko mitsuhiko self-assigned this Jan 7, 2017
@mitsuhiko
Member

While the current behavior is incorrect, the old behavior is definitely incorrect as well. The {% set %} tag was never intended to override outer scopes. I'm surprised it did in some cases. Not sure what to do here.

@mitsuhiko
Member

The behavior I would assume were correct is an output like [-1:0] [-1:1] [-1:2] [-1:3] [-1:4] -1.

@alex
alex commented Jan 7, 2017

In this specific case I resolved it by rewriting it to use groupby.

@ThiefMaster
Member

I have the feeling this is a somewhat common use case - also stuff like {% set found = true %} in a loop and then checking it afterwards. It's surely likely to break things for people when this stops working...

@mitsuhiko
Member
mitsuhiko commented Jan 7, 2017 edited

I'm shocked this worked before. Did this always work?

@ThiefMaster
Member

apparently yes:

>>> import jinja2
>>> jinja2.__version__
'2.0'
>>> jinja2.Template('{% for x in range(5) %}[{{ a }}:{% set a = x %}{{ a }}] {% endfor %}{{ a }}').render()
u'[:0] [0:1] [1:2] [2:3] [3:4] '
@mitsuhiko
Member

Great. Because the issue here is that this is absolutely not sound. Which variable is it supposed to override? What if there is a function scope in between. eg: a macro or something like that. I have no idea how to support this now.

@ThiefMaster
Member

hmm maybe similar to python's scoping assuming no nonlocal was used)? ie don't allow overriding something defined in an outer scope (ie if there's a macro inbetween) but do allow overriding it otherise?

@mitsuhiko
Member
mitsuhiko commented Jan 7, 2017 edited

Apparently before this was caught down with a template assertion error:

>>> Template('{% set x = 0 %}{% for y in [1, 2, 3] recursive %}{{ x }}{% set x = y %}{% endfor %}').render()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
jinja2.exceptions.TemplateAssertionError: It's not possible to set and access variables derived from an outer scope! (affects: x (line 1)
@ThiefMaster
Member

the behavior seems to be different with/without recursive (2.8.1 gives me an UnboundLocalError error with recursive and '012' without)

@mitsuhiko
Member

The unbound local error should be resolved on master.

@mitsuhiko
Member

I'm not sure how to progress here. I really dislike that it was apparently possible to do this sort of thing.

@mitsuhiko mitsuhiko added a commit that referenced this issue Jan 8, 2017
@mitsuhiko mitsuhiko Implement consistent scoping for sets in loops
While technically this applies to any scope and not just for loops
it comes up most commonly in the context of for loops.  This now
defines the behavior for scoping in a way that is consistent but
different than it was in the past.  There is an ongoing conversation
if we should keep it that way or not.

References #641
8172db0
@mitsuhiko
Member

With the last changes I'm going to close this as "works as intended". If there is further fallout from this we can investigate alternatives again.

@mitsuhiko mitsuhiko closed this Jan 8, 2017
@foosel foosel added a commit to foosel/OctoPrint that referenced this issue Jan 9, 2017
@foosel foosel Pinning Jinja to <2.9 for now
Variables defined in an outer scope can no longer be set from an inner scope (see
pallets/jinja#641). Regardless of whether that is right or wrong, we can't control if
people are using such constructs in their plugins, which versions of Jinja >= 2.9 would
now break out of the blue, regardless of OctoPrint version. That is unacceptable sadly
and requires pinning for now, until plugin authors have had a chance to adapt
accordingly.

Also see #1697.
1014712
@MinchinWeb

Got sent to this issue (see #656) after this change blew up my blog template on upgrading from v.2.8.1 to v2.9.4.

I was using it keep track if various pieces of data were changing between loop iteration. I was able to fix it because I wrote the original template code (see MinchinWeb/seafoam@8eb7608 and MinchinWeb/seafoam@89d555d), but I doubt I would have been able to otherwise. The new code is harder to follow as the comparisons are now done in-line. For example, my old code (v.2.8.1):

{%- set last_day = None -%}

{% for article in dates %}
    {# ... #}
    <div class="archives-date">
        {%- if last_day != article.date.day %}
            {{ article.date | strftime('%a %-d') }}
        {% else -%}
            &mdash;
        {%- endif -%}
    </div>
    {%- set last_day = article.date.day %}
{% endfor %}

and the new code, with in-line comparisons (v2.9.4):

{% for article in dates %}
    {# ... #}
    <div class="archives-date">
        {%- if ((article.date.day == dates[loop.index0 - 1].date.day) and
                (article.date.month == dates[loop.index0 - 1].date.month) and 
                (article.date.year == dates[loop.index0 - 1].date.year)) %}
                    &mdash;
        {% else -%}
                    {{ article.date | strftime('%a %-d') }}
        {%- endif -%}
    </div>
{% endfor %}

So I just wanted to say that the "feature" (or "hack", if you prefer) is used and is already missed.

If the scoping issues are too complex to figure out sensibly at the moment, could something (at a minimum) be added to the changelog so it bites less people unaware?

@mitsuhiko
Member

I was not aware this is so widely abused :-/ Annoyingly there is really no way to make this work in any reliable way. However I wonder if we can isolate the common use cases here and introduce a nicer api.

In particular maybe we want to add something like this (loop.changed_from_last):

{% for article in dates %}
    <div class="archives-date">
        {%- if loop.changed_from_last(article.date.day) %}
            {{ article.date | strftime('%a %-d') }}
        {% else -%}
            &mdash;
        {%- endif -%}
    </div>
{% endfor %}
@mitsuhiko mitsuhiko reopened this Jan 12, 2017
@ThiefMaster
Member

changed_from_last sounds good - at least functionality-wise, the name itself is a bit awkward IMO. Maybe just changed would be clear enough?

I guess the first element would always be considered "changed" no matter what it is? If that behavior is not OK for someone they could always add a not loop.first check anyway.

@davidism
Member

Maybe previous_context just holds the entire previous context, rather than trying to decide what the users will do with it.

@ThiefMaster
Member
ThiefMaster commented Jan 13, 2017 edited

just previous or prev? "context" sounds rather confusing in this context (no pun intended)

.....and now I can already imagine someone asking for a next :/

@davidism
Member
davidism commented Jan 13, 2017 edited

Or maybe an explicit statement for writing outside scope: set_outer or something.

@mitsuhiko
Member
mitsuhiko commented Jan 13, 2017 edited

i was thinking this:

class LoopContextBase(object):

    def __init__(self, recurse=None, depth0=0):
        ...
        self._last_iteration = missing

    def changed(self, *value):
        if self._last_iteration != value:
            self._last_iteration = value
            return True
        return False
@mitsuhiko
Member

@davidism we cannot do set_outer without breaking the entire scoping system. This would majorly screw up the entire thing.

@davidism
Member

Yeah, figured that would be the case. I'm anticipating "what if I want to know if the current value is greater than the previous", or something similar. But I like the changed method too.

@foosel
foosel commented Jan 13, 2017

As an additional data point, it fell on my feet for a case where I need to inject a special class into generated HTML, but only for the first element of the iterated list that does not also have some specific property set. I can't modify the iterated data. I can't use loop.first (as much as I'd love to) or compare with anything from the last element to reliably do what I need to do here, which led to experiment with the evil construct which worked perfectly (and it was not clear to me that it was actually me abusing a bug).

Additionally I offer extension capabilities through third party plugins and have no way to police how the authors structure their templates, causing sudden breakage for end users after update to Jinja 2.9+. I've pinned the latest 2.8 version now in my program for that matter to remain backwards compatible (as my versioning scheme promises), until I can find a way to get authors to update their templates.

@roganov
roganov commented Jan 13, 2017 edited

Could someone please clarify why jinja2 scopes cannot work exactly as python works? i.e., jinja2 templates correspond to python modules, macros correspond to functions (and have their own scope), for loops don't have their own scope and so forth. While @mitsuhiko says jinja 2.8 and below had lots of scoping bugs, it was more intuitive for me to understand how scoping works.

For python programmers, the behavior from first post (Jinja 2.8) is obvious. Same applies to #660, I don't understand why python library should implement javascript behavior (I understand that python's handling of default args is not ideal, but any python developer is aware of it).

Alternatively, there should be a document which describes how scoping in jinja works so that we don't have to guess.

Also please don't take my comment negatively, I'm very grateful for jinja.

@mitsuhiko
Member

@roganov a few notes on this:

Could someone please clarify why jinja2 scopes cannot work exactly as python works

Because Python's scoping is in my opinion really bad, particularly in templates because you can easily accidentally override important information.

While @mitsuhiko says jinja 2.8 and below had lots of scoping bugs, it was more intuitive for me to understand how scoping works.

Note that this was the an exception due to a bug and it only worked in some limited situations and purely by accident. Jinja had always the same documented scoping rules and this behavior was never documented. At all times it was said that variables do not propagate to outer scopes. You can easily see this because after the end of the loop the variable was never set.

Alternatively, there should be a document which describes how scoping in jinja works so that we don't have to guess.

I improved the docs already a few days back for this

@MinchinWeb
MinchinWeb commented Jan 14, 2017 edited

What if there was a way to explicitly pass a variable into a loop? Perhaps syntax like this (borrowing from text filters):

{%- set last_day = None -%}

{% for article in dates | pass_variable ( last_day ) %}
    {# ... #}
    <div class="archives-date">
        {%- if last_day != article.date.day -%}
            {{ article.date | strftime('%a %-d') }}
        {%- else %}
            &mdash;
        {% endif -%}
    </div>
    {%- set last_day = article.date.day %}
{% endfor %}

Or is there a way to do something similar to scoped as it currently applies to macros?


Another use case would be maintaining some sort of counter between loops. For example, how many rows total? how many rows meet some criteria? The total of some property for selected rows (to print a total row)?

@ThiefMaster
Member

Filter syntax would not be possible here since filters are already possible on the iterable.

A possible syntax for this that wouldn't look to bad could be this:

{% for article in dates with last_day %}

Especially since with already does scoping stuff in Jinja when used e.g. as {% with %}. OTOH, here it would be the opposite since with blocks open a new scope, not make outer scope vars accessible..

Not sure though if that's a feature Jinja needs or not.

@mitsuhiko
Member

No syntax will change this because it would still subvert the scoping rules. You cannot do that with either the current compilation or id tracking system. Even if it works for that simple case, what happens if one has a for loop that is recursive. What if someone defines a macro in a for loop. What if a call block appears in a for loop.

I think it would make more sense to introduce a global object that acts as a storage namespace and can then be modified with . Eg:

{% set foo = namespace() %}
{% set foo.iterated = false %}
{% for item in seq %}
  {% set foo.iterated = true %}
{% endfor %}

However this would require the set tag to fundamentally change.

@davidism
Member
davidism commented Jan 14, 2017 edited

@mitsuhiko and that sort of pattern is already in use with the do tag: http://stackoverflow.com/a/4880398/400617

@ThiefMaster
Member

FWIW, the solution with do is extremely awful. Just like the workaround people use in Python 2 when they'd need nonlocal...

@davidism
Member
davidism commented Jan 14, 2017 edited

I completely agree, just pointing out that it's out there. And like Alex pointed out, a lot of these problems can be rewritten, either with filters or by putting some of the logic in Python.

@kdeldycke kdeldycke added a commit to kdeldycke/kevin-deldycke-blog that referenced this issue Jan 14, 2017
@kdeldycke kdeldycke Pin-down Jinja to bypass variable scoping in for loop in 2.9.
This cause side-by-side ellipsis items to not be detected when rendering
pagination widget in Plumage:
https://github.com/kdeldycke/plumage/blob/0.8/templates/pagination.html#L20

Issue has been reported to Jinja: pallets/jinja#641
2a55102
@MinchinWeb

Another (ugly) hack is to use a list, append, and pop: http://stackoverflow.com/a/32700975/4276230

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment