pdb help fails with AttributeError when using Windows embeddable package

[sparrowt]

BPO	46434
Nosy	@pfmoore, @tjguk, @zware, @zooba, @miss-islington, @sparrowt
PRs	bpo-46434: Handle missing docstring in pdb help #30705 [3.9] bpo-46434: Handle missing docstrings in pdb help (GH-30705) #30754 [3.10] bpo-46434: Handle missing docstrings in pdb help (GH-30705) #30755 [3.10] bpo-46434: Handle missing docstrings in pdb help (GH-30705) #30759

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-21.19:43:19.079>
created_at = <Date 2022-01-19.16:41:48.748>
labels = ['3.10', 'library', '3.9', 'OS-windows', '3.11']
title = 'pdb help fails with AttributeError when using Windows embeddable package'
updated_at = <Date 2022-01-21.19:43:19.078>
user = 'https://github.com/sparrowt'

bugs.python.org fields:

activity = <Date 2022-01-21.19:43:19.078>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-21.19:43:19.079>
closer = 'steve.dower'
components = ['Library (Lib)', 'Windows']
creation = <Date 2022-01-19.16:41:48.748>
creator = 'sparrowt'
dependencies = []
files = []
hgrepos = []
issue_num = 46434
keywords = ['patch']
message_count = 11.0
messages = ['410970', '410971', '410977', '411016', '411024', '411026', '411027', '411149', '411159', '411175', '411176']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'miss-islington', 'sparrowt']
pr_nums = ['30705', '30754', '30755', '30759']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue46434'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[sparrowt]

When running python from an embeddable windows zip file such as:
https://www.python.org/ftp/python/3.9.10/python-3.9.10-embed-amd64.zip

the help command within pdb is broken due to missing docstrings, as demonstrated below.

Python 3.9.10 (tags/v3.9.10:f2f3f53, Jan 17 2022, 15:14:21) [MSC v.1929 64 bit (AMD64)] on win32
>>> import pdb
>>> pdb.set_trace()
--Return--
> <stdin>(1)<module>()->None
(Pdb) help interact
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "bdb.py", line 92, in trace_dispatch
  File "bdb.py", line 151, in dispatch_return
  File "pdb.py", line 294, in user_return
  File "pdb.py", line 357, in interaction
  File "pdb.py", line 322, in _cmdloop
  File "cmd.py", line 138, in cmdloop
  File "pdb.py", line 422, in onecmd
  File "cmd.py", line 217, in onecmd
  File "pdb.py", line 1496, in do_help
AttributeError: 'NoneType' object has no attribute 'rstrip'
>>>
>>> print(pdb.Pdb.do_interact.__doc__)
None

[sparrowt]

Perhaps we should have a more generic check for command.__doc__ as it appears this is a case not handled by the existing sys.flags.optimize >= 2 check?

https://github.com/python/cpython/blob/v3.9.10/Lib/pdb.py#L1492-L1496

[zooba]

The PR looks good. I assume you've started the CLA process, so once that clears we'll be fine to merge.

[sparrowt]

Thanks, yes CLA should now be working its way through the system.

I guess a side question is: is it expected for docstring to have been stripped from the windows embeddable distribution, and how is that done if not with -OO? (I found --without-doc-strings but not any build scripts using it)

This patch is still valid either way I think - better to show a helpful error than to explode - but I am intrigued as to the reason behind this case, deliberate or not. e.g. is there an assumption that the embed redistributable is never going to be used interactively & thus we might as well save space?

[zooba]

See

cpython/PC/layout/main.py

Lines 256 to 289 in b04dfbb

	def _compile_one_py(src, dest, name, optimize, checked=True):
	import py_compile
	if dest is not None:
	dest = str(dest)
	mode = (
	py_compile.PycInvalidationMode.CHECKED_HASH
	if checked
	else py_compile.PycInvalidationMode.UNCHECKED_HASH
	)
	try:
	return Path(
	py_compile.compile(
	str(src),
	dest,
	str(name),
	doraise=True,
	optimize=optimize,
	invalidation_mode=mode,
	)
	)
	except py_compile.PyCompileError:
	log_warning("Failed to compile {}", src)
	return None
	# name argument added to address bpo-37641
	def _py_temp_compile(src, name, ns, dest_dir=None, checked=True):
	if not ns.precompile or src not in PY_FILES or src.parent in DATA_DIRS:
	return None
	dest = (dest_dir or ns.temp) / (src.stem + ".pyc")
	return _compile_one_py(src, dest, name, optimize=2, checked=checked)

Basically, the .pyc files in the embeddable distro have been compiled with optimize=2, which is the equivalent of -OO. So yes, docstrings (and asserts) are expected to have been removed.

It shouldn't be removing docstrings from other modules that you provide yourself.

[sparrowt]

Gottit thanks for explaining - so the existing check in do_help didn't catch this case because https://docs.python.org/3/library/sys.html#sys.flags only reflects command line flags, rather than whether or not it was actually optimised.

[zooba]

the existing check in do_help didn't catch this case because https://docs.python.org/3/library/sys.html#sys.flags only reflects command line flags, rather than whether or not it was actually optimised.

Precisely. Your check is much more appropriate for what the actual issue
is going to be, though keeping the existing check there means we can
provide a more specific error message when we *know* that the user
deliberately chose that behaviour.

[zooba]

New changeset 60705cf by Tom Sparrow in branch 'main':
bpo-46434: Handle missing docstrings in pdb help (GH-30705)
60705cf

[miss-islington]

New changeset c3ad850 by Miss Islington (bot) in branch '3.9':
bpo-46434: Handle missing docstrings in pdb help (GH-30705)
c3ad850

[miss-islington]

New changeset 05063fa by Miss Islington (bot) in branch '3.10':
bpo-46434: Handle missing docstrings in pdb help (GH-30705)
05063fa

[zooba]

Thanks for the contribution!