Early tracing has lineno=None for modules

[nedbat]

BPO	46911
Nosy	@nedbat, @markshannon

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-03-03.13:36:59.929>
labels = ['interpreter-core', '3.11']
title = 'Early tracing has lineno=None for modules'
updated_at = <Date 2022-03-05.23:50:12.504>
user = 'https://github.com/nedbat'

bugs.python.org fields:

activity = <Date 2022-03-05.23:50:12.504>
actor = 'nedbat'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2022-03-03.13:36:59.929>
creator = 'nedbat'
dependencies = []
files = []
hgrepos = []
issue_num = 46911
keywords = ['3.11regression']
message_count = 3.0
messages = ['414438', '414516', '414602']
nosy_count = 2.0
nosy_names = ['nedbat', 'Mark.Shannon']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46911'
versions = ['Python 3.11']

[nedbat]

Coverage.py has a trick to measure the early execution of stdlib modules. It has an "encodings.py" file that sets a trace function, then gets out of the way to let the real encodings.py execute. In 3.11.0a5, that early trace function gets None values for the line numbers when modules were first executed.

To reproduce, create two files, encodings.py and prog.py:

--- 8< ------------------------------------------

# encodings.py
import sys

class FullCoverageTracer:
    def __init__(self):
        self.traces = []

    def fullcoverage_trace(self, *args):
        frame, event, arg = args
        if frame.f_lineno is None:
            self.traces.append((frame.f_code.co_name, frame.f_code.co_filename))
        return self.fullcoverage_trace

sys.settrace(FullCoverageTracer().fullcoverage_trace)

parentdir = max(filter(__file__.startswith, sys.path), key=len)
sys.path.remove(parentdir)
del sys.modules['encodings']
import encodings

---

--- 8< ------------------------------------------

# prog.py
import sys

print(sys.version)
trace = sys.gettrace()
print("None traces:", trace.__self__.traces)
print("Hello")

---

Then run:

% PYTHONPATH=$(pwd) python3.10 prog.py
3.10.2 (main, Jan 15 2022, 05:51:59) [Clang 12.0.0 (clang-1200.0.32.29)]
None traces: []
Hello

% PYTHONPATH=$(pwd) python3.11 prog.py
3.11.0a5 (main, Feb  3 2022, 19:11:58) [Clang 12.0.0 (clang-1200.0.32.29)]
None traces: [('\<module\>', '/usr/local/pyenv/pyenv/versions/3.11.0a5/lib/python3.11/encodings/__init__.py'), ('\<module\>', '\<frozen codecs\>'), ('\<module\>', '/usr/local/pyenv/pyenv/versions/3.11.0a5/lib/python3.11/encodings/aliases.py'), ('\<module\>', '/usr/local/pyenv/pyenv/versions/3.11.0a5/lib/python3.11/encodings/utf_8.py'), ('\<module\>', '\<frozen io\>'), ('\<module\>', '\<frozen abc\>'), ('\<module\>', '\<frozen site\>'), ('\<module\>', '\<frozen os\>'), ('\<module\>', '\<frozen stat\>'), ('\<module\>', '\<frozen \_collections_abc\>'), ('\<module\>', '\<frozen posixpath\>'), ('\<module\>', '\<frozen genericpath\>'), ('\<module\>', '\<frozen \_sitebuiltins\>'), ('\<module\>', '\<string\>'), ('\<module\>', '/System/Volumes/Data/root/src/bugs/fullcov/prog.py')]
Hello

[markshannon]

This is a bit of a tricky one.

The problem is that the line number for an instruction is used for two purposes.

1. To calculate the line number in frame.f_lineno for tracebacks and events
2. By dis to determine which lines are present and where they start.

If we set the lineno for the initial RESUME instruction in a module, we break dis and coverage for modules that do not have code on line 1.

If we add an explicit line-start table, we would be able to fix this, as the RESUME would have a line number for case 1, but wouldn't start a line, thus not breaking dis.

An explicit line-start table has other advantages, so we probably will implement it, but maybe not until 3.12.

Ned,
Would it make sense in coverage.py to ignore "call" events when doing coverage, and only use "line" events?
All the traces in your example are from "call" events.

[nedbat]

Maybe I'm missing something during normal execution, but I'm only seeing this effect now during this super-early encodings.py trick. I don't mind just special-casing the Nones in this case.

[iritkatriel]

#90973 is about implementing the line start table.

[iritkatriel]

@nedbat what is the status of this? Is it still an issue?

(We've abandoned 90973 because it's no longer needed).