pydoc.synopsis fails to load binary modules

[ericsnowcurrently]

BPO	20123
Nosy	@gpshead, @ncoghlan, @vstinner, @larryhastings, @bitdancer, @ericsnowcurrently, @koobs, @iritkatriel
Files	issue20123-fix-pydoc-synopsis.diff koobs-freebsd9-3.4-build57.log

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-12-12.00:38:04.644>
created_at = <Date 2014-01-04.22:27:05.794>
labels = ['type-bug']
title = 'pydoc.synopsis fails to load binary modules'
updated_at = <Date 2021-12-12.00:38:04.643>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2021-12-12.00:38:04.643>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-12.00:38:04.644>
closer = 'iritkatriel'
components = []
creation = <Date 2014-01-04.22:27:05.794>
creator = 'eric.snow'
dependencies = []
files = ['33312', '34796']
hgrepos = []
issue_num = 20123
keywords = ['patch']
message_count = 21.0
messages = ['207319', '207321', '207328', '207330', '207334', '207335', '207337', '207344', '207345', '207346', '207347', '207348', '207349', '207350', '207351', '207354', '207356', '207673', '215997', '231470', '408354']
nosy_count = 10.0
nosy_names = ['gregory.p.smith', 'ncoghlan', 'vstinner', 'larry', 'Arfrever', 'r.david.murray', 'python-dev', 'eric.snow', 'koobs', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20123'
versions = ['Python 3.3']

[ericsnowcurrently]

I'm guessing this is a very seldom (never?) used code path. I've included a patch to test and fix the problem. The patch includes several related tests for pydoc.

$ py3 -c 'import pydoc; pydoc.synopsis("/opt/python3.4/lib/python3.4/lib-dynload/time.cpython-34m.so")'
Traceback (most recent call last):
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 368, in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 96: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/python3.4/lib/python3.4/pydoc.py", line 229, in synopsis
    file = tokenize.open(filename)
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 429, in open
    encoding, lines = detect_encoding(buffer.readline)
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 409, in detect_encoding
    encoding = find_cookie(first)
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 373, in find_cookie
    raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for '/opt/python3.4/lib/python3.4/lib-dynload/time.cpython-34m.so'

[ericsnowcurrently]

FYI, I found this while working on issue bpo-19703.

[larryhastings]

Well please get a second opinion. I don't know why you added me, I'm not qualified.

[ericsnowcurrently]

Not sure why I nosy'ed you, either. :)

[bitdancer]

In 2.7 the code just does an open. Victor changed it to call tokenize.open in 3.2, but tokenize.open obviously only works on python source files. So the logic of that method is now completely wrong.

I'm not sure the logic made a lot of sense even before...if the extension is binary it just closes it again. So the initial open is for its side effect of returning None if the file can't be opened.

Anyway, the logic of that method clearly needs to be rewritten. And obviously there are some missing tests....

I don't see any patch attached to the issue, by the way.

[ericsnowcurrently]

Thanks for taking a look. Here's the patch that I totally forget to attach.

[bitdancer]

Is it the case that given a filename, it might be possible to load a module even if open(filename) fails?

I think the logic is clearer in the form where it is not pulled out into a separate helper function. You can avoid the double check on the extension by doing:

    if filename.endswith(importlib.machinery.BYTECODE_SUFFIXES):
        loader = importlib.machinery.SourcelessFileLoader('__temp__',
                                                          filename)
    elif filename.endswith(importlib.machinery.EXTENSION_SUFFIXES):
        loader = importlib.machinery.ExtensionFileLoader('__temp__',
                                                         filename)
    else:
        loader = None

    if loader:
        xxxxx
    else:
        xxxxx

[ericsnowcurrently]

Sounds good to me. Here's an updated patch.

[bitdancer]

Looks good, except you can replace those any calls with simple calls to endswith using a tuple of strings (I forgot the call to tuple in my example).

[python-dev]

New changeset d6c3fb8d5f84 by Eric Snow in branch 'default':
bpo-20123: Fix pydoc.synopsis() for "binary" modules.
http://hg.python.org/cpython/rev/d6c3fb8d5f84

[ericsnowcurrently]

Ah, I missed that in your earlier suggestion. I followed your recommendation. Thanks for that. :)

[ericsnowcurrently]

This broke one of the FreeBSD buildbots:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%203.x/builds/6102