pygettext doesn't extract docstrings for functions with type annotated params

[Tobotimus]

BPO	32222
Nosy	@serhiy-storchaka, @Tobotimus, @miss-islington, @iritkatriel
PRs	bpo-32222: Fix pygettext skipping docstrings for funcs with arg typehints #4745 [3.7] bpo-32222: Fix pygettext skipping docstrings for funcs with arg typehints (GH-4745) #5915 [3.6] bpo-32222: Fix pygettext skipping docstrings for funcs with arg typehints (GH-4745) #5916
Files	test_typehinted_funcs.py: Input file for pygettext to demonstrate behaviour pygettext_output.pot: Output file from pygettext for test_typehinted_funcs.py test_returntypehinted_funcs.py: Demonstrating behaviour with return type annotations

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2020-10-06.01:16:00.210>
created_at = <Date 2017-12-05.10:36:15.662>
labels = ['type-bug', '3.7']
title = "pygettext doesn't extract docstrings for functions with type annotated params"
updated_at = <Date 2020-10-06.01:16:00.210>
user = 'https://github.com/Tobotimus'

bugs.python.org fields:

activity = <Date 2020-10-06.01:16:00.210>
actor = 'methane'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2020-10-06.01:16:00.210>
closer = 'methane'
components = ['Demos and Tools']
creation = <Date 2017-12-05.10:36:15.662>
creator = 'Tobotimus'
dependencies = []
files = ['47316', '47317', '47318']
hgrepos = []
issue_num = 32222
keywords = ['patch']
message_count = 6.0
messages = ['307652', '307654', '307791', '312967', '312970', '377943']
nosy_count = 4.0
nosy_names = ['serhiy.storchaka', 'Tobotimus', 'miss-islington', 'iritkatriel']
pr_nums = ['4745', '5915', '5916']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32222'
versions = ['Python 3.6', 'Python 3.7']

[Tobotimus]

### Expected Behaviour
When running pygettext with the -D CLI flag, all module, class, method and function docstrings should be extracted and outputted to the .pot file.

### Actual Behaviour
In the case of functions whose parameters have PEP-484 type annotations, their docstrings are not being extracted. I have attached two files, one .py file and its corresponding .pot file, as examples of this behaviour.

[Tobotimus]

Correction for above: the type annotation syntax is actually from PEP-3107, not PEP-484.

I should also point out that this behaviour is occurring for annotated return types of functions and methods as well.

This is occurring on both Windows and Linux.

[Tobotimus]

Another correction; I said that this is occurring for return type annotations; this is not true. This is only occurring for functions with annotated *arguments*.

The cause of this issue is in pygettext's TokenEater.__suiteseen method. This method eats tokens until it sees a colon operator, after which it assumes a function docstring may follow. In the case of a function with annotated arguments, obviously this is incorrect, as colons would appear before the final colon of the 'suite'.

[serhiy-storchaka]

New changeset eee72d4 by Serhiy Storchaka (Tobotimus) in branch 'master':
bpo-32222: Fix pygettext skipping docstrings for funcs with arg typehints (GH-4745)
eee72d4

[miss-islington]

New changeset 51d95ff by Miss Islington (bot) in branch '3.7':
bpo-32222: Fix pygettext skipping docstrings for funcs with arg typehints (GH-4745)
51d95ff

[iritkatriel]

I think this is resolved and can be closed.