pydoc.synopsis fails to load binary modules

[ericsnowcurrently]

BPO	20123
Nosy	@gpshead, @ncoghlan, @vstinner, @larryhastings, @bitdancer, @ericsnowcurrently, @koobs, @iritkatriel
Files	issue20123-fix-pydoc-synopsis.diff koobs-freebsd9-3.4-build57.log

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-12-12.00:38:04.644>
created_at = <Date 2014-01-04.22:27:05.794>
labels = ['type-bug']
title = 'pydoc.synopsis fails to load binary modules'
updated_at = <Date 2021-12-12.00:38:04.643>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2021-12-12.00:38:04.643>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-12.00:38:04.644>
closer = 'iritkatriel'
components = []
creation = <Date 2014-01-04.22:27:05.794>
creator = 'eric.snow'
dependencies = []
files = ['33312', '34796']
hgrepos = []
issue_num = 20123
keywords = ['patch']
message_count = 21.0
messages = ['207319', '207321', '207328', '207330', '207334', '207335', '207337', '207344', '207345', '207346', '207347', '207348', '207349', '207350', '207351', '207354', '207356', '207673', '215997', '231470', '408354']
nosy_count = 10.0
nosy_names = ['gregory.p.smith', 'ncoghlan', 'vstinner', 'larry', 'Arfrever', 'r.david.murray', 'python-dev', 'eric.snow', 'koobs', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20123'
versions = ['Python 3.3']

[ericsnowcurrently]

I'm guessing this is a very seldom (never?) used code path. I've included a patch to test and fix the problem. The patch includes several related tests for pydoc.

$ py3 -c 'import pydoc; pydoc.synopsis("/opt/python3.4/lib/python3.4/lib-dynload/time.cpython-34m.so")'
Traceback (most recent call last):
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 368, in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 96: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/python3.4/lib/python3.4/pydoc.py", line 229, in synopsis
    file = tokenize.open(filename)
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 429, in open
    encoding, lines = detect_encoding(buffer.readline)
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 409, in detect_encoding
    encoding = find_cookie(first)
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 373, in find_cookie
    raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for '/opt/python3.4/lib/python3.4/lib-dynload/time.cpython-34m.so'

[ericsnowcurrently]

FYI, I found this while working on issue bpo-19703.

[larryhastings]

Well please get a second opinion. I don't know why you added me, I'm not qualified.

[ericsnowcurrently]

Not sure why I nosy'ed you, either. :)

[bitdancer]

In 2.7 the code just does an open. Victor changed it to call tokenize.open in 3.2, but tokenize.open obviously only works on python source files. So the logic of that method is now completely wrong.

I'm not sure the logic made a lot of sense even before...if the extension is binary it just closes it again. So the initial open is for its side effect of returning None if the file can't be opened.

Anyway, the logic of that method clearly needs to be rewritten. And obviously there are some missing tests....

I don't see any patch attached to the issue, by the way.

[ericsnowcurrently]

Thanks for taking a look. Here's the patch that I totally forget to attach.

[bitdancer]

Is it the case that given a filename, it might be possible to load a module even if open(filename) fails?

I think the logic is clearer in the form where it is not pulled out into a separate helper function. You can avoid the double check on the extension by doing:

    if filename.endswith(importlib.machinery.BYTECODE_SUFFIXES):
        loader = importlib.machinery.SourcelessFileLoader('__temp__',
                                                          filename)
    elif filename.endswith(importlib.machinery.EXTENSION_SUFFIXES):
        loader = importlib.machinery.ExtensionFileLoader('__temp__',
                                                         filename)
    else:
        loader = None

    if loader:
        xxxxx
    else:
        xxxxx

[ericsnowcurrently]

Sounds good to me. Here's an updated patch.

[bitdancer]

Looks good, except you can replace those any calls with simple calls to endswith using a tuple of strings (I forgot the call to tuple in my example).

[python-dev]

New changeset d6c3fb8d5f84 by Eric Snow in branch 'default':
bpo-20123: Fix pydoc.synopsis() for "binary" modules.
http://hg.python.org/cpython/rev/d6c3fb8d5f84

[ericsnowcurrently]

Ah, I missed that in your earlier suggestion. I followed your recommendation. Thanks for that. :)

[ericsnowcurrently]

This broke one of the FreeBSD buildbots:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%203.x/builds/6102

[python-dev]

New changeset ff3be21338d5 by Eric Snow in branch 'default':
bpo-20123: try using a different builtin module in a pydoc test.
http://hg.python.org/cpython/rev/ff3be21338d5

[python-dev]

New changeset efcf163d04f5 by Eric Snow in branch 'default':
bpo-20123: Disable a problematic test.
http://hg.python.org/cpython/rev/efcf163d04f5

[ericsnowcurrently]

I've run out of time to trouble-shoot the failure (specific to 1 buildbot). Until I can get back to it, I've disabled the problematic test (even though it's only a problem on 1 buildbot).

[ericsnowcurrently]

The buildbot is happy again. I'll address fixing that test in issue bpo-20128.

[larryhastings]

Thanks for seeing this through, Eric.

[ericsnowcurrently]

I realized today that this should probably be fixed in 3.3 as well, as RDM implied by marking it as also a 3.3 bug. :)

[koobs]

koobs-freebsd9 (3.4) buildbot has also been failing for a while on what seems to be this changeset:

======================================================================
ERROR: test_synopsis_sourceless (test.test_pydoc.PydocDocTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/test/test_pydoc.py", line 504, in test_synopsis_sourceless
    synopsis = pydoc.synopsis(filename)
  File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/pydoc.py", line 238, in synopsis
    mtime = os.stat(filename).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: '/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/__pycache__/os.cpython-34.pyc'

Full buildlog is attached.

[gpshead]

fyi - tracking the new issue koobs reported in http://bugs.python.org/issue22910

[iritkatriel]

Re-enabling the test is tracked on 20128, so this issue is no longer needed.