Inspect library ignore comments at the end of a function (inspect.getsource)

[noureddinehamid]

BPO	42116
Nosy	@taleinat, @1st1, @miss-islington, @isidentical, @iritkatriel
PRs	bpo-42116: Fix inspect.getsource handling of trailing comments #23630 [3.9] bpo-42116: Fix inspect.getsource handling of trailing comments (GH-23630) #23643 [3.8] bpo-42116: Fix inspect.getsource handling of trailing comments (GH-23630) #23644

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-12-04.20:21:43.707>
created_at = <Date 2020-10-22.10:48:43.969>
labels = ['3.8', 'type-feature', 'library', '3.9', '3.10']
title = 'Inspect library ignore comments at the end of a function (inspect.getsource)'
updated_at = <Date 2020-12-15.13:21:15.932>
user = 'https://bugs.python.org/noureddinehamid'

bugs.python.org fields:

activity = <Date 2020-12-15.13:21:15.932>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2020-12-04.20:21:43.707>
closer = 'taleinat'
components = ['Library (Lib)']
creation = <Date 2020-10-22.10:48:43.969>
creator = 'noureddine.hamid'
dependencies = []
files = []
hgrepos = []
issue_num = 42116
keywords = ['patch']
message_count = 9.0
messages = ['379289', '382416', '382417', '382505', '382526', '382527', '382528', '383047', '383054']
nosy_count = 6.0
nosy_names = ['taleinat', 'yselivanov', 'miss-islington', 'BTaskaya', 'iritkatriel', 'noureddine.hamid']
pr_nums = ['23630', '23643', '23644']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue42116'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[noureddinehamid]

inspect.getsource ignore comments at the end of the function:

for example this function:

def matmul_single(A, x, out):
  from numpy import matmul
  out[:] = matmul(A, x)
  # Some comment here...

using the inspect library:
>>> inspect.getsource(matmul_single)                                                                                                                                     
>>> "def omp_matmul_single(A, x, out):\n  from numpy import matmul\n out[:] = matmul(A, x)\n"

the result does not contain the comments at the end of the function.

[iritkatriel]

1. For a comment line, the tokenizer emits a COMMENT token followed by an NL token for the newline. The inspect.BlockFinder.tokeneater increments its "last" field to the last line it identified as belonging to the code block. Currently it increments it when it sees a NEWLINE token, but not for an NL token.

2. For a comment line, the tokenizer does not emit an INDENT/DEDENT token, so the indentation level when it is processes is assumed to be equal to that of the previous line.

PR 23630 aims to include comment lines in the block if their start column is after the start column of the opening line of the block:

   def f():
      return 42

 \# this is a part of f

# this is not a part of f

[iritkatriel]

For reference - this script:
---------------------------------------------

import inspect
import tokenize
from pprint import pprint as pp

src=[
 'def f():\n',
 '    return 1\n',
 '    #that was fun',
 '\n',
 '#Now comes g\n',
 'def g():\n',
 '    return 2\n']

pp(list(tokenize.generate_tokens(iter(src).__next__)))
---------------------------------------------

Outputs:
---------------------------------------------
[TokenInfo(type=1 (NAME), string='def', start=(1, 0), end=(1, 3), line='def f():\n'),
TokenInfo(type=1 (NAME), string='f', start=(1, 4), end=(1, 5), line='def f():\n'),
TokenInfo(type=54 (OP), string='(', start=(1, 5), end=(1, 6), line='def f():\n'),
TokenInfo(type=54 (OP), string=')', start=(1, 6), end=(1, 7), line='def f():\n'),
TokenInfo(type=54 (OP), string=':', start=(1, 7), end=(1, 8), line='def f():\n'),
TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 8), end=(1, 9), line='def f():\n'),
TokenInfo(type=5 (INDENT), string=' ', start=(2, 0), end=(2, 4), line=' return 1\n'),
TokenInfo(type=1 (NAME), string='return', start=(2, 4), end=(2, 10), line=' return 1\n'),
TokenInfo(type=2 (NUMBER), string='1', start=(2, 11), end=(2, 12), line=' return 1\n'),
TokenInfo(type=4 (NEWLINE), string='\n', start=(2, 12), end=(2, 13), line=' return 1\n'),
TokenInfo(type=60 (COMMENT), string='#that was fun', start=(3, 4), end=(3, 17), line=' #that was fun'),
TokenInfo(type=61 (NL), string='', start=(3, 17), end=(3, 17), line=' #that was fun'),
TokenInfo(type=61 (NL), string='\n', start=(4, 0), end=(4, 1), line='\n'),
TokenInfo(type=60 (COMMENT), string='#Now comes g', start=(5, 0), end=(5, 12), line='#Now comes g\n'),
TokenInfo(type=61 (NL), string='\n', start=(5, 12), end=(5, 13), line='#Now comes g\n'),
TokenInfo(type=6 (DEDENT), string='', start=(6, 0), end=(6, 0), line='def g():\n'),
TokenInfo(type=1 (NAME), string='def', start=(6, 0), end=(6, 3), line='def g():\n'),
TokenInfo(type=1 (NAME), string='g', start=(6, 4), end=(6, 5), line='def g():\n'),
TokenInfo(type=54 (OP), string='(', start=(6, 5), end=(6, 6), line='def g():\n'),
TokenInfo(type=54 (OP), string=')', start=(6, 6), end=(6, 7), line='def g():\n'),
TokenInfo(type=54 (OP), string=':', start=(6, 7), end=(6, 8), line='def g():\n'),
TokenInfo(type=4 (NEWLINE), string='\n', start=(6, 8), end=(6, 9), line='def g():\n'),
TokenInfo(type=5 (INDENT), string=' ', start=(7, 0), end=(7, 4), line=' return 2\n'),
TokenInfo(type=1 (NAME), string='return', start=(7, 4), end=(7, 10), line=' return 2\n'),
TokenInfo(type=2 (NUMBER), string='2', start=(7, 11), end=(7, 12), line=' return 2\n'),
TokenInfo(type=4 (NEWLINE), string='\n', start=(7, 12), end=(7, 13), line=' return 2\n'),
TokenInfo(type=6 (DEDENT), string='', start=(8, 0), end=(8, 0), line=''),
TokenInfo(type=0 (ENDMARKER), string='', start=(8, 0), end=(8, 0), line='')]

[taleinat]

New changeset 6e1eec7 by Irit Katriel in branch 'master':
bpo-42116: Fix inspect.getsource handling of trailing comments (GH-23630)
6e1eec7

[miss-islington]

New changeset 81ac030 by Miss Islington (bot) in branch '3.9':
bpo-42116: Fix inspect.getsource handling of trailing comments (GH-23630)
81ac030

[miss-islington]

New changeset 3b14f18 by Miss Islington (bot) in branch '3.8':
bpo-42116: Fix inspect.getsource handling of trailing comments (GH-23630)
3b14f18