Generator expressions trace differently on Windows than on Mac

[nedbat]

BPO	44450
Nosy	@nedbat, @markshannon

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/markshannon'
closed_at = None
created_at = <Date 2021-06-18.11:50:04.188>
labels = ['interpreter-core', '3.10']
title = 'Generator expressions trace differently on Windows than on Mac'
updated_at = <Date 2021-09-01.00:25:32.384>
user = 'https://github.com/nedbat'

bugs.python.org fields:

activity = <Date 2021-09-01.00:25:32.384>
actor = 'nedbat'
assignee = 'Mark.Shannon'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2021-06-18.11:50:04.188>
creator = 'nedbat'
dependencies = []
files = []
hgrepos = []
issue_num = 44450
keywords = ['3.10regression']
message_count = 9.0
messages = ['396050', '396126', '396129', '396218', '396227', '396232', '396233', '399502', '400796']
nosy_count = 2.0
nosy_names = ['nedbat', 'Mark.Shannon']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue44450'
versions = ['Python 3.10']

[nedbat]

Here is a trace involving generator expressions. Using 3.10.0b3 on Mac, there are "line" events within the expression. Those events are missing on Windows.

--- 8< -------------------------------

import linecache, sys

def trace(frame, event, arg):
    # The weird globals here is to avoid a NameError on shutdown...
    if frame.f_code.co_filename == globals().get("__file__"):
        lineno = frame.f_lineno
        print("{} {}: {}".format(event[:4], lineno, linecache.getline(__file__, lineno).rstrip()))
    return trace

def doit():
    o = ((1,2), (3,4))
    o = (a for a in o)
    for tup in o:
        x = tup[0]
        y = tup[1]

print(sys.version)
sys.settrace(trace)
doit()

---

When run on Mac, it produces this output:

3.10.0b3 (default, Jun 18 2021, 06:43:38) [Clang 12.0.0 (clang-1200.0.32.29)]
call 10: def doit():
line 11: o = ((1,2), (3,4))
line 12: o = (a for a in o)
line 13: for tup in o:
call 12: o = (a for a in o)
line 12: o = (a for a in o)
retu 12: o = (a for a in o)
line 14: x = tup[0]
line 15: y = tup[1]
line 13: for tup in o:
call 12: o = (a for a in o)
line 12: o = (a for a in o)
retu 12: o = (a for a in o)
line 14: x = tup[0]
line 15: y = tup[1]
line 13: for tup in o:
call 12: o = (a for a in o)
retu 12: o = (a for a in o)
retu 13: for tup in o:

When run on Windows, it produces this output:

3.10.0b3 (tags/v3.10.0b3:865714a, Jun 17 2021, 20:39:25) [MSC v.1929 64 bit (AMD64)]
call 10: def doit():
line 11: o = ((1,2), (3,4))
line 12: o = (a for a in o)
line 13: for tup in o:
call 12: o = (a for a in o)
retu 12: o = (a for a in o)
line 14: x = tup[0]
line 15: y = tup[1]
line 13: for tup in o:
call 12: o = (a for a in o)
retu 12: o = (a for a in o)
line 14: x = tup[0]
line 15: y = tup[1]
line 13: for tup in o:
call 12: o = (a for a in o)
retu 12: o = (a for a in o)
retu 13: for tup in o:

On Windows, the "line 12" events are missing.

[markshannon]

Ned, is this a regression (does 3.9 do the right thing on Windows) or an inconsistency between Mac and Windows?

I suspect this might have something to do with the PREDICT macros. If that the were the case 3.9 should show the same inconsistency between Windows and the Mac.

[nedbat]

This happens with all the 3.10 betas, and does not happen in 3.9 and earlier:
https://github.com/nedbat/coveragepy/runs/2864611225
(Sorry, other failures with earlier 3.10 betas obscured the mac/win difference.)

[markshannon]

I think this is a combination of https://bugs.python.org/issue44297 and the PREDICT macros.

I don't have a windows machine to confirm this on, but I suspect that if you rewrite doit as:

def doit():
    o = ((1,2), (3,4))
    o = (a for
         a in
         o)
    for tup in o:
        x = tup[0]
        y = tup[1]

then you should be able to observe a difference between Windows and Mac on 3.9 as well.

[nedbat]

I tried adding that rewritten doit as a new test, and it does not show a mac/win difference on 3.9. In fact, it doesn't show a mac/win difference on 3.10!

https://github.com/nedbat/coveragepy/actions/runs/956791631

[markshannon]

Hmm, I'm a bit puzzled by that.

Did you test with 3.10b3 or the latest build from the 3.10 branch with the fix to https://bugs.python.org/issue44297 included?

[nedbat]

This was with 3.10.0b3. I haven't got a way (yet) to build with my own-built versions of CPython.

[markshannon]

Ned, is this still an issue with the release candidate of 3.10?

[nedbat]

Looks like this is fixed with 3.10.0rc1, thanks.