I added rewind() to TokenStream. look() now uses it to get the next token and then rewind. push() is still there and working, but I'm not sure if it should still exist.
Added rewind() to TokenStream
Why is that necessary?
For one thing, although this is arguable, it's cleaner to have a rewind method in a lexer. For the more practical answer, the django <-> jinja2 compatibility layer I'm working on, and which I will soon make available, needs rewind for the template tags part, to compile Extensions out of simple_tag, inclusion_tag and other template-sugar-like helper function decorators that work both in jinja2 and django.
About that, by the way, I now have the exact same unmodified (really big) templates codebase working in both jinja2 and django (with my compatibility layer) and I can easily measure the benefits of using jinja2 vs django in the exact same page by swapping engines: 4697ms (django) vs 468ms (jinja2)
Closing this for the time being until a patch lands that depends on it.