pydoc: move __future__ imports out of the DATA block

[anntzer]

BPO	26120
Nosy	@serhiy-storchaka, @vedgar, @iritkatriel
PRs	bpo-26120: make pydoc exclude __future__ imports from the data block of the module #30888 bpo-26120: do not exclude __future__ Features in pydoc of the __future__ module itself #32180

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-29.22:07:47.957>
created_at = <Date 2016-01-15.06:31:39.957>
labels = ['3.11', 'type-feature', 'library', 'docs']
title = 'pydoc: move __future__ imports out of the DATA block'
updated_at = <Date 2022-03-29.22:07:47.956>
user = 'https://github.com/anntzer'

bugs.python.org fields:

activity = <Date 2022-03-29.22:07:47.956>
actor = 'iritkatriel'
assignee = 'docs@python'
closed = True
closed_date = <Date 2022-03-29.22:07:47.957>
closer = 'iritkatriel'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2016-01-15.06:31:39.957>
creator = 'Antony.Lee'
dependencies = []
files = []
hgrepos = []
issue_num = 26120
keywords = ['patch']
message_count = 15.0
messages = ['258272', '407922', '407939', '411604', '416086', '416155', '416160', '416192', '416212', '416213', '416263', '416264', '416276', '416284', '416301']
nosy_count = 4.0
nosy_names = ['docs@python', 'serhiy.storchaka', 'veky', 'iritkatriel']
pr_nums = ['30888', '32180']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26120'
versions = ['Python 3.11']

[anntzer]

Currently, for a module that uses future imports, the DATA section of pydoc foo contains these imports interspersed with the "real" data from the module.

Even though it is fully-featured _Feature objects that are imported, it probably makes sense to move them out of this section.

[iritkatriel]

Reproduced on 3.11.

[vedgar]

I thought that _Feature starts with an underscore precisely to evade such listings. Do other "private" module data also get listed?

[iritkatriel]

It's like this:

>>> import foo
>>> dir(foo)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'annotations', 'x']
>>> foo.annotations
_Feature((3, 7, 0, 'beta', 1), (3, 11, 0, 'alpha', 0), 16777216)

So the attribute in foo is called annotations, i.e. it's not private.

[serhiy-storchaka]

I am positive about this idea, but we must also think about the possible negative consequences. For example, the future annotations will be included in the star-import by default and can override some global names. The fact that some names not visible in the module help can override global names can be confusing.

I think it should be discussed with a wider auditory.

[iritkatriel]

That's a good point. I see that the __future__ imports appear in the dir() of the module, and indeed they are imported with 'from m import *'.

But I wonder if that is actually a bug. If you try this:

% cat x.py

from __future__ import annotations

% cat y.py
from x import *

print(dir())

class D:
    def f(self, a: D):
        return 42

% ./python.exe y.py    
['__annotations__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'annotations']
Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython-654/y.py", line 5, in <module>
    class D:
    ^^^^^^^^
  File "/Users/iritkatriel/src/cpython-654/y.py", line 6, in D
    def f(self, a: D):
                   ^
NameError: name 'D' is not defined

---

but if you add "from __future__ import annotations" at the top of y.py, then it does run.

So perhaps the future imports should be excluded by "from m import *"?

[serhiy-storchaka]

I once proposed to exclude modules from the star import by default, but this proposition was rejected. You can try, maybe your proposition will be more acceptable.

[serhiy-storchaka]

Module objects are not shown in the help unless they are submodules of the specified module, even if they are imported with the star import. With this precedence I think it is okay to exclude the __future__ annotations as well.

[iritkatriel]

New changeset 15ba816 by Irit Katriel in branch 'main':
bpo-26120: make pydoc exclude __future__ imports from the data block of the module (GH-30888)
15ba816

[iritkatriel]

Thank you, Serhiy!

[serhiy-storchaka]

I am sorry that I did not test the changes manually, but it seems that __future__ annotations are now disappeared from the pydoc output for the __future__ module help.

    $ ./python -m pydoc __future__

It is now difficult to get a list of available features from the help.

They should be kept in the DATA section for the __future__ module.

[serhiy-storchaka]

Yet one bug: PR 30888 only changed the text output. But there is also the html output generator.

[iritkatriel]

Yet one bug: PR 30888 only changed the text output. But there is also the html output generator.

I'm not sure - if I revert the change like this:

         if isinstance(getattr(obj, name, None), __future__._Feature):
-            return False
+            pass # return False

then test_html_doc fails:

======================================================================
FAIL: test_html_doc (test.test_pydoc.PydocDocTest.test_html_doc)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython/Lib/test/test_pydoc.py", line 414, in test_html_doc
    self.assertEqual(text_lines, expected_lines)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Lists differ: ['tes[1392 chars]t]', "print_function = _Feature((2, 6, 0, 'alp[156 chars]ody'] != ['tes[1392 chars]t]', 'type_union1 = typing.Union[int, str]', '[72 chars]ody']

First differing element 55:
"print_function = _Feature((2, 6, 0, 'alp[36 chars]576)"
'type_union1 = typing.Union[int, str]'

First list contains 1 additional elements.
First extra element 61:
'Nobody'

['test.pydoc_mod (version 1.2.3.4)',
'This is a test module for test_pydoc',
'Modules',
'types',
'typing',
'Classes',
'builtins.object',
'A',
'B',
'C',
'class A(builtins.object)',
'Hello and goodbye',
'Methods defined here:',
'__init__()',
'Wow, I have no function!',
'Data descriptors defined here:',
'__dict__',
'dictionary for instance variables (if defined)',
'__weakref__',
'list of weak references to the object (if defined)',
'class B(builtins.object)',
'Data descriptors defined here:',
'__dict__',
'dictionary for instance variables (if defined)',
'__weakref__',
'list of weak references to the object (if defined)',
'Data and other attributes defined here:',
"NO_MEANING = 'eggs'",
"__annotations__ = {'NO_MEANING': <class 'str'>}",
'class C(builtins.object)',
'Methods defined here:',
'get_answer(self)',
'Return say_no()',
'is_it_true(self)',
'Return self.get_answer()',
'say_no(self)',
'Class methods defined here:',
'__class_getitem__(item) from builtins.type',
'Data descriptors defined here:',
'__dict__',
'dictionary for instance variables (if defined)',
'__weakref__',
'list of weak references to the object (if defined)',
'Functions',
'doc_func()',
"This function solves all of the world's problems:",
'hunger',
'lack of Python',
'war',
'nodoc_func()',
'Data',
"__xyz__ = 'X, Y and Z'",
'c_alias = test.pydoc_mod.C[int]',
'list_alias1 = typing.List[int]',
'list_alias2 = list[int]',

• "print_function = _Feature((2, 6, 0, 'alpha', 2), (3, 0, 0, 'alpha', 0), "
• '1048576)',
  'type_union1 = typing.Union[int, str]',
  'type_union2 = int | str',
  'Author',
  'Benjamin Peterson',
  'Credits',
  'Nobody']

[serhiy-storchaka]

Yet one bug: PR 30888 only changed the text output. But there is also the html output generator.

Sorry, I was wrong. The changed code is common for all generators. Perhaps I tested with wrong version.

[iritkatriel]

New changeset 63f32fa by Irit Katriel in branch 'main':
bpo-26120: do not exclude __future__ import in pydoc of the __future__ module itself (GH-32180)
63f32fa