Use for instead of next in contextlib.contextmanager

[brandtbucher]

This is a fun little optimization. According to some local microbenchmarks (basically entering and exiting context managers a million times), turning these next calls into for loops saves about 25% of the overhead this class introduces (~5% due to __enter__ and ~20% due to __exit__). This is because:

• The wrapped iterator is usually a generator. We specialize for loops for generators, so instead of calling through the C code for next and re-entering the interpreter, we can "inline" the frame push instead. (This is also more JIT-friendly.)
• We don't care about the actual StopIteration exceptions here, just whether or not they were raised. Raising exceptions is expensive, and the interpreter has an optimization to avoid actually raising StopIteration in normal for loops. (This is especially helpful in __exit__, where we advance the generator, expect a StopIteration to be raised, and then just throw it away!)

This change isn't worth it for the async variant of this function, since neither of the above optimizations apply to async for loops.

[picnixz]

For tracking purposes, could you create an issue for that please? I also think it's worth a NEWS entry for people who are interested in this kind of optimization and understanding it.

[serhiy-storchaka]

The wrapped iterator is usually a generator.

What if it is not? for calls __iter__, not only __next__.

[brandtbucher]

Sure, I can create an issue/NEWS.

What if it is not? for calls __iter__, not only __next__.

The docs for contextlib.contextmanager say that this function only works with functions that return generator-iterators (which define __iter__ the normal way):

The function being decorated must return a generator-iterator when called.

Besides, this still works (just as it does today) if another type of iterator happens to be returned, since they're required by the iteration protocol to define an __iter__ returning self anyways:

Iterators are required to have an __iter__() method that returns the iterator object itself so every iterator is also iterable and may be used in most places where other iterables are accepted.