AsyncIterator on 3.7: __aiter__ no longer honors finally blocks

[asksol]

BPO	36403
Nosy	@ned-deily, @asvetlov, @1st1, @twisteroidambassador, @tirkarthi
Files	test_iterators_and_finally_blocks.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-03-23.00:15:07.175>
labels = ['3.8', 'type-bug', '3.7', 'expert-asyncio']
title = 'AsyncIterator on 3.7: __aiter__ no longer honors finally blocks'
updated_at = <Date 2020-06-12.08:32:55.224>
user = 'https://bugs.python.org/asksol'

bugs.python.org fields:

activity = <Date 2020-06-12.08:32:55.224>
actor = 'ned.deily'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['asyncio']
creation = <Date 2019-03-23.00:15:07.175>
creator = 'asksol'
dependencies = []
files = ['48229']
hgrepos = []
issue_num = 36403
keywords = ['3.7regression']
message_count = 7.0
messages = ['338630', '338638', '338642', '338647', '345805', '345808', '371341']
nosy_count = 6.0
nosy_names = ['ned.deily', 'asvetlov', 'asksol', 'yselivanov', 'twisteroid ambassador', 'xtreak']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36403'
versions = ['Python 3.7', 'Python 3.8']

[asksol]

We use finally blocks in __aiter__ like this:

class AsyncFinallyIterator:

   def __aiter__(self):
       for i in range(10):
           try:
               yield i
           finally:
               print('FINALLY')

Attached is a test for both iterators and async iterators.
The tests pass on Python 3.6, but only the non-async iterator test pass under Python 3.7.

Thanks for your attention!

This worked perfectly well in Python 3.6, but stopped working in Python 3.7.

I also verified that Iterator supports the same construct (and this works in both Python 3.6 and 3.7):

class FinallyIterator:

    def __iter__(self):
        for i in range(10):
            try:
                yield i
            finally:
                print('FINALLY')

[tirkarthi]

The attached script fails on master too. On bisecting could this be possibly caused due to 41e5ec3 (bpo-34769) ? Tagging Ned since it was introduced from 3.7.1rc2 and also backported to 3.6 .

➜ cpython git:(41e5ec3) git checkout 41e5ec3 && make -s -j4 > /dev/null
HEAD is now at 41e5ec3 bpo-34769: Thread safety for _asyncgen_finalizer_hook(). (GH-9716)
➜ cpython git:(41e5ec3) ./python.exe ../backups/bpo36403.py
F.
======================================================================
FAIL: test_main (main.TestAsyncIteratorFinally)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "../backups/bpo36403.py", line 30, in test_main
    self.assertTrue(it.finally_executed)
AssertionError: False is not true

---

Ran 2 tests in 0.002s

FAILED (failures=1)
➜ cpython git:(41e5ec3) git checkout 41e5ec3~1 && make -s -j4 > /dev/null
Previous HEAD position was 41e5ec3 bpo-34769: Thread safety for _asyncgen_finalizer_hook(). (GH-9716)
HEAD is now at 0ce31d3 bpo-32962: Fix test_gdb failure in debug build with -mcet -fcf-protection -O0 (GH-9656)
➜ cpython git:(0ce31d3) ./python.exe ../backups/bpo36403.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.003s

OK

[asksol]

Ah, so the extra call_soon means it needs a:

[code]
loop.run_until_complete(asyncio.sleep(0))```
[/code]

before the self.assertTrue(it.finally_executed)

to finish executing agen.close().

Why is create_task different? Does it execute an iteration of the generator immediately?

Seems good for this behavior to be consistent, but not sure how difficult that would be.

[asksol]

Perhaps we could add a self._finally to the event loop itself?
Like loop._ready, but a list of callbacks run_until_complete will call before returning?

[ned-deily]

Ping. This was marked as a 3.7regression. Is a change needed?

[asvetlov]

From my understanding yield inside __aiter__ is a forbidden construction. Even if parser allows it the statement doesn't make any sense.

I'm very interested to hear Yuri opinion.

[ned-deily]

I note this is marked as a 3.7regression and still open. Since the cutoff for the final 3.7 bugfix mode release is in a few days, I'm assuming this means that 3.7 users will have to live with this regression. If you feel that is a problem, speak up now.

[kumaraditya303]

I don't see how this is a bug or regression. There is no guarantee when generators are finalized, code relying on it is already subtly broken. I'll close this unless anyone objects, marking pending.