`inspect.BlockFinder` handles `tokenize.NEWLINE` too strictly

[BabakkGraphcore]

Bug report

Bug description:

In python 3.13.8 calling inspect.getsourcelines on a decorated function with a comment between the decorator and the name only returns the function declaration, the body is skipped.

import inspect

def dummy(fn):
    return fn

@dummy
def test1(a, b):
    return a

@dummy
#comment
def test2(a, b):
    return a

print(inspect.getsourcelines(test1))
print("------------------")
print(inspect.getsourcelines(test2))

returns

(['@dummy\n', 'def test1(a, b):\n', '    return a\n'], 6)
------------------
(['@dummy\n', '#comment\n', 'def test2(a, b):\n'], 10)

the source of test2 is incomplete and missing the return statement.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-139783: Fix inspect.getsourcelines() for the case when a decorator is followed by a comment or an empty line #139836
• [3.14] gh-139783: Fix inspect.getsourcelines() for the case when a decorator is followed by a comment or an empty line (GH-139836) #139889
• [3.13] gh-139783: Fix inspect.getsourcelines() for the case when a decorator is followed by a comment or an empty line (GH-139836) #139890

[picnixz]

I'll have a look at this.

[picnixz]

The issue is with the BlockFinder which, because of the comment, considers that the body is not part of the function anymore.

[picnixz]

Similar issues:

@dummy

@dummy
def f():
    return 1

print(inspect.getsourcelines(f))  # ['@dummy\n', '\n', '@dummy\n']

So it's not only comments that are poorly handled, it's anything that "breaks" the "prelude" of the function (that is decorators). I'm not entirely sure we can do much thing here without adding a layer of complexity. It shouldn't be possible to say that the source lines of something only contains decorators or new lines/comments as this is not valid Python code. So, we should do something else.

It would be quite easy to actually preprocess the source and eliminate "un-necessary" lines but then we'll need to handle the offset properly. In a single pass, we'll need to determine whether the block ended correctly or not.

Similarly:

@\
dummy
def f():
    return 1

This is equivalent to use @dummy but again, the source lines will only consist of dummy and nothing else! But this is a separate issue actually, where findsource already struggles with that (it will not put @ and dummy together...)

[picnixz]

I need to go now but I'll try to do something tomorrow or later this week. Thanks for reporting this!

[BabakkGraphcore]

I need to go now but I'll try to do something tomorrow or later this week. Thanks for reporting this!

Great, thanks for picking it up so quickly1

[serhiy-storchaka]

I worked with that code recently, so if you need help, I can help.

[picnixz]

I am a bit time constrained so I would appreciate your help especially if it is something you recently worked on (I only skimmed through the implementation)

[albertvillanova]

I confirm this issue and I give some information about severity and impact.

Additional Impact: Breaking PyTorch JIT + HuggingFace Transformers

This bug is affecting production code in the PyTorch and HuggingFace ecosystem.

Real-world Example

The HuggingFace transformers library contains multiple instances of this pattern. For example, in transformers/models/deberta_v2/modeling_deberta_v2.py (lines 105-108): https://github.com/huggingface/transformers/blob/be3fa93b298eec4182a0bb6c5e64270a5f1e8020/src/transformers/models/deberta_v2/modeling_deberta_v2.py#L105-L108

@torch.jit.script
# Copied from transformers.models.deberta.modeling_deberta.c2p_dynamic_expand
def c2p_dynamic_expand(c2p_pos, query_layer, relative_pos):
    return c2p_pos.expand([query_layer.size(0), query_layer.size(1), query_layer.size(2), relative_pos.size(-1)])

When this module is imported on Python 3.13.8, the @torch.jit.script decorator calls inspect.getsourcelines(), which returns only:

@torch.jit.script
# Copied from transformers.models.deberta.modeling_deberta.c2p_dynamic_expand
def c2p_dynamic_expand(c2p_pos, query_layer, relative_pos):

PyTorch then tries to parse this truncated source with ast.parse(), which fails with:

File "<unknown>", line 3
    def c2p_dynamic_expand(c2p_pos, query_layer, relative_pos):
                                                               ^
  IndentationError: expected an indented block after function definition on line 3

Root Cause Confirmed

I traced through the tokenization and can confirm the issue is in BlockFinder.tokeneater():

1. After the decorator's NEWLINE token, self.indecorator is set to False
2. The COMMENT token then enters the if not self.started and not self.indecorator: branch
3. Since the token is not "def" or "class", it sets self.singleline = True
4. This causes BlockFinder to treat the comment as a lambda/generator expression
5. When the function signature's NEWLINE is encountered, it raises EndOfBlock prematurely

Regression Confirmation

✅ Python 3.13.7: Works correctly
❌ Python 3.13.8: Broken

This appears to be a side effect of the fix for gh-137477 (generator expressions):

• inspect.getsourcelines() mishandles multi-line generator expressions #137477