Skip to content
This repository

Fix loop.last memory issue #92

Merged
merged 1 commit into from over 2 years ago

2 participants

Jason Kotenko Armin Ronacher
Jason Kotenko

We are using a generator to input data to jinja2 templates. When we render, the data is streamed out using TemplateStream, which works great unless we have loop.last in our template.

The code for loop.last currently reads in the whole iterator (in our case a generator which reads files) in order to determine the length of the loop, and check to see if this is the last iteration.

There is an alternative to checking the length, which is to keep the next element in the iteration stored in memory, and checking that element for a sentinel value to see if the loop is on the last iteration.

That is the approach I took in this pull request. We are patching our jinja2 packages internally, in order to allow us to use loop.last on big data (though loop.length is still dangerous, and by extension, the repr method on LoopContext). Thanks.

Jason Kotenko Fixed loop.last to not consume the entire iterator to determine if th…
…is is

the last iteration of the loop.
776567c
Armin Ronacher mitsuhiko merged commit ea89feb into from
Joe Abbate jmafc referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Jan 24, 2012
Jason Kotenko Fixed loop.last to not consume the entire iterator to determine if th…
…is is

the last iteration of the loop.
776567c
This page is out of date. Refresh to see the latest.

Showing 1 changed file with 14 additions and 2 deletions. Show diff stats Hide diff stats

  1. +14 2 jinja2/runtime.py
16 jinja2/runtime.py
@@ -266,10 +266,12 @@ def __call__(self):
266 266
267 267 class LoopContext(object):
268 268 """A loop context for dynamic iteration."""
  269 + End = object()
269 270
270 271 def __init__(self, iterable, recurse=None):
271 272 self._iterator = iter(iterable)
272 273 self._recurse = recurse
  274 + self._after = self._safe_next()
273 275 self.index0 = -1
274 276
275 277 # try to get the length of the iterable early. This must be done
@@ -288,7 +290,7 @@ def cycle(self, *args):
288 290 return args[self.index0 % len(args)]
289 291
290 292 first = property(lambda x: x.index0 == 0)
291   - last = property(lambda x: x.index0 + 1 == x.length)
  293 + last = property(lambda x: x._after is LoopContext.End)
292 294 index = property(lambda x: x.index0 + 1)
293 295 revindex = property(lambda x: x.length - x.index0)
294 296 revindex0 = property(lambda x: x.length - x.index)
@@ -299,6 +301,12 @@ def __len__(self):
299 301 def __iter__(self):
300 302 return LoopContextIterator(self)
301 303
  304 + def _safe_next(self):
  305 + try:
  306 + return next(self._iterator)
  307 + except StopIteration:
  308 + return self.End
  309 +
302 310 @internalcode
303 311 def loop(self, iterable):
304 312 if self._recurse is None:
@@ -344,7 +352,11 @@ def __iter__(self):
344 352 def next(self):
345 353 ctx = self.context
346 354 ctx.index0 += 1
347   - return next(ctx._iterator), ctx
  355 + if ctx._after is LoopContext.End:
  356 + raise StopIteration
  357 + next_elem = ctx._after
  358 + ctx._after = ctx._safe_next()
  359 + return next_elem, ctx
348 360
349 361
350 362 class Macro(object):

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.