linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension

[graingert]

BPO	44446
Nosy	@nedbat, @markshannon, @graingert, @pablogsal, @FFY00, @iritkatriel
PRs	bpo-44446: support lineno being None in traceback.FrameSummary #26781 bpo-44446: set lineno in generator instructions #26782 [3.10] bpo-44446: support lineno being None in traceback.FrameSummary (GH-26781) #27072

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-07-08.16:47:55.422>
created_at = <Date 2021-06-17.18:49:12.478>
labels = ['3.10']
title = 'linecache.getline TypeError when formatting tracebacks in stacks containing an async list comprehension'
updated_at = <Date 2022-03-02.13:56:47.227>
user = 'https://github.com/graingert'

bugs.python.org fields:

activity = <Date 2022-03-02.13:56:47.227>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-07-08.16:47:55.422>
closer = 'pablogsal'
components = []
creation = <Date 2021-06-17.18:49:12.478>
creator = 'graingert'
dependencies = []
files = []
hgrepos = []
issue_num = 44446
keywords = ['patch']
message_count = 13.0
messages = ['396014', '396022', '396026', '396101', '396230', '396231', '397135', '397140', '397149', '397155', '397156', '414346', '414353']
nosy_count = 7.0
nosy_names = ['nedbat', 'Mark.Shannon', 'graingert', 'pablogsal', 'FFY00', 'iritkatriel', 'peter.roelants']
pr_nums = ['26781', '26782', '27072']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44446'
versions = ['Python 3.10']

[graingert]

demo:

import traceback
import io

async def foo():
    yield 1
    traceback.print_stack(file=io.StringIO())
    yield 2

async def bar():
    return [chunk async for chunk in foo()]

next(bar().__await__(), None)
print("working!")

Traceback (most recent call last):
  File "/home/graingert/projects/anyio/foo.py", line 13, in <module>
    next(bar().__await__(), None)
  File "/home/graingert/projects/anyio/foo.py", line 10, in bar
    return [chunk async for chunk in foo()]
  File "/home/graingert/projects/anyio/foo.py", line -1, in <listcomp>
  File "/home/graingert/projects/anyio/foo.py", line 6, in foo
    traceback.print_stack(file=io.StringIO())
  File "/usr/lib/python3.10/traceback.py", line 203, in print_stack
    print_list(extract_stack(f, limit=limit), file=file)
  File "/usr/lib/python3.10/traceback.py", line 224, in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
  File "/usr/lib/python3.10/traceback.py", line 379, in extract
    f.line
  File "/usr/lib/python3.10/traceback.py", line 301, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/usr/lib/python3.10/linecache.py", line 31, in getline
    if 1 <= lineno <= len(lines):
TypeError: '<=' not supported between instances of 'int' and 'NoneType'

[FFY00]

I bissected this to 088a15c and submitted a patch making traceback.FrameSummary take into consideration that lineno might be None, I believe this is probably the correct fix.

[FFY00]

Upon further investigation, there are actually two issues here. The first would be the one I identified already, traceback.FrameSummary not being prepared for lineno being None, but there is also a regression in lineno being invalid in this situation in the first place.

With only #70968, the traceback will look like the following:

File "/home/anubis/git/cpython/rep.py", line 13, in <module>
next(bar().__await__(), None)
File "/home/anubis/git/cpython/rep.py", line 10, in bar
return [chunk async for chunk in foo()]
File "/home/anubis/git/cpython/rep.py", line None, in <listcomp>
File "/home/anubis/git/cpython/rep.py", line 6, in foo
traceback.print_stack()
working!

which is different from 3.9

File "/home/anubis/git/cpython/rep.py", line 13, in <module>
next(bar().__await__(), None)
File "/home/anubis/git/cpython/rep.py", line 10, in bar
return [chunk async for chunk in foo()]
File "/home/anubis/git/cpython/rep.py", line 10, in <listcomp>
return [chunk async for chunk in foo()]
File "/home/anubis/git/cpython/rep.py", line 6, in foo
traceback.print_stack()
working!

I bisected the second issue to b37181e which moves generators to bytecode, and when doing so changes the behavior to set lineno to -1. I have opened a #70969 to fixing this.

[pablogsal]

Mark, can you take a look at this?

[markshannon]

This appears to be a duplicate of https://bugs.python.org/issue44297

[markshannon]

With the latest 3.10, I get:

File "/home/mark/test/test.py", line 13, in <module>
next(bar().__await__(), None)
File "/home/mark/test/test.py", line 10, in bar
return [chunk async for chunk in foo()]
File "/home/mark/test/test.py", line 10, in <listcomp>
return [chunk async for chunk in foo()]
File "/home/mark/test/test.py", line 6, in foo
traceback.print_stack()
working!

Thomas, can you confirm?

[pablogsal]

Beta 4 is in a few days. Can someone confirm of this is fixed or if it still needs a patch?

[FFY00]

The issue that made the line number be missing is fixed, but #70968 is needed to account for 088a15c, even though it is no longer triggered in this situation.

[pablogsal]

New changeset 91a8f8c by Filipe Laíns in branch 'main':
bpo-44446: support lineno being None in traceback.FrameSummary (GH-26781)
91a8f8c

[pablogsal]

New changeset 61eb9b5 by Pablo Galindo in branch '3.10':
[3.10] bpo-44446: support lineno being None in traceback.FrameSummary (GH-26781) (GH-27072)
61eb9b5

[pablogsal]

Feel free to reopen if we are missing anything

[peterroelants]

If I understand correctly this should be fixed? In which 3.10.* version should this be fixed?

The reason why I'm asking is that I ran into this issue when using Dask (2022.02.0) with multithreading on Python 3.10.2:

Exception in thread Profile:
Traceback (most recent call last):
  File "./lib/python3.10/site-packages/distributed/profile.py", line 115, in process
    d = state["children"][ident]
KeyError: '_all_objs;./lib/python3.10/site-packages/bokeh/embed/bundle.py;357'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "./lib/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "./lib/python3.10/site-packages/distributed/profile.py", line 274, in _watch
    process(frame, None, recent, omit=omit)
  File "./lib/python3.10/site-packages/distributed/profile.py", line 119, in process
    "description": info_frame(frame),
  File "./lib/python3.10/site-packages/distributed/profile.py", line 72, in info_frame
    line = linecache.getline(co.co_filename, frame.f_lineno, frame.f_globals).lstrip()
  File "./lib/python3.10/linecache.py", line 31, in getline
    if 1 <= lineno <= len(lines):
TypeError: '<=' not supported between instances of 'int' and 'NoneType'

[pablogsal]

This was fixed in 3.10.0 if I am not mistaken. Could you provide a reproducer, please?