UnicodeEncodeError on pydoc's CLI

[christoph]

BPO	6625
Nosy	@vstinner, @merwok, @akitada, @florentx, @Bluehorn, @berkerpeksag
Files	unicode.patch: Patch encoding output with system encoding pydoc_unicode_testcase_notworking.diff: Added test for unicode (defunc) issue6625_pydoc.diff: Patch, apply to trunk

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-01-13.08:40:41.805>
created_at = <Date 2009-08-02.13:00:46.380>
labels = ['type-bug', 'library']
title = "UnicodeEncodeError on pydoc's CLI"
updated_at = <Date 2014-01-13.10:56:30.737>
user = 'https://bugs.python.org/christoph'

bugs.python.org fields:

activity = <Date 2014-01-13.10:56:30.737>
actor = 'torsten'
assignee = 'none'
closed = True
closed_date = <Date 2014-01-13.08:40:41.805>
closer = 'berker.peksag'
components = ['Library (Lib)']
creation = <Date 2009-08-02.13:00:46.380>
creator = 'christoph'
dependencies = []
files = ['14626', '14656', '16296']
hgrepos = []
issue_num = 6625
keywords = ['patch']
message_count = 10.0
messages = ['91182', '91295', '91315', '99716', '99717', '99723', '99728', '208006', '208013', '208018']
nosy_count = 8.0
nosy_names = ['ggenellina', 'vstinner', 'christoph', 'eric.araujo', 'akitada', 'flox', 'torsten', 'berker.peksag']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6625'
versions = ['Python 2.7']

[christoph]

pydoc fails with a UnicodeEncodeError for properly specified Unicode
docstrings (u"""...""") on the command line interface.

See attached patch that encodes the output with the system's encoding.

[ggenellina]

After applying your patch, are string (not unicode) docstrings still
being handled properly?
Adding a test case (in Lib/test/test_pydoc.py) would be nice too.

[christoph]

Here is a diff for test/test_pydoc.py (against Python2.6) which though
doesn't trigger due to how Python handles output encoding. This test
here will pass, but pydoc will still fail:

$ pydoc test/pydoc_mod.py > /dev/null
Traceback (most recent call last):
  File "/usr/bin/pydoc", line 5, in <module>
    pydoc.cli()
  File "/usr/lib/python2.5/pydoc.py", line 2226, in cli
    help.help(arg)
  File "/usr/lib/python2.5/pydoc.py", line 1691, in help
    else: doc(request, 'Help on %s:')
  File "/usr/lib/python2.5/pydoc.py", line 1482, in doc
    pager(title % desc + '\n\n' + text.document(object, name))
  File "/usr/lib/python2.5/pydoc.py", line 1300, in pager
    pager(text)
  File "/usr/lib/python2.5/pydoc.py", line 1398, in plainpager
    sys.stdout.write(plain(text))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in
position 936: ordinal not in range(128)

[Bluehorn]

This problem persists with current Python 2.7 trunk (I used hg mirror):
52aa81c14f87 (trunk) tip

I also checked with the py3k branch and it does not show this limitation. Just as one would have expected...

[Bluehorn]

As Subversion is still the master repo (correct me if I am wrong), the hg version tested maps to r78290 in subversion on /trunk.

[florentx]

This patch do not work for me.

See the patch attached on duplicate bug bpo-7625, with more tests:
http://bugs.python.org/file15831/issue7675_pydoc_v2.diff

[florentx]

Patch with same tests as the previous one, but using better heuristic for output encoding (like Christoph patch).
Added the "replace" error handling, if the output encoding cannot encode all characters.

[akitada]

I suppose this is a duplicate of bpo-1065986.

[berkerpeksag]

I suppose this is a duplicate of bpo-1065986.

Yes, it is. I created a test file from the tests in issue6625_pydoc.diff and ran it on the current 2.7 branch.

def foo():
    u"""fooo bar
    baz
    \xfcnicode\u2026"""
    return 42

def bar():
    u"f\xfcr Elise"
    return 11

[Bluehorn]

I tested this as well and it seems to work now. :-) Thanks for fixing it!