pydoc help (or help('help')) should show the doc for help

[ssbr]

BPO	13691
Nosy	@terryjreedy, @ncoghlan, @ssbr, @ezio-melotti, @merwok, @bitdancer, @serhiy-storchaka, @ultimatecoder, @CuriousLearner
PRs	bpo-13691: Fixes pydoc help (or help('help')) to show the doc for help #172
Files	r74214.diff i13691.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/merwok'
closed_at = None
created_at = <Date 2012-01-01.08:22:31.379>
labels = ['3.7', 'type-bug', 'library']
title = "pydoc help (or help('help')) should show the doc for help"
updated_at = <Date 2017-06-08.10:50:22.574>
user = 'https://github.com/ssbr'

bugs.python.org fields:

activity = <Date 2017-06-08.10:50:22.574>
actor = 'ncoghlan'
assignee = 'eric.araujo'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2012-01-01.08:22:31.379>
creator = 'Devin Jeanpierre'
dependencies = []
files = ['24121', '25933']
hgrepos = []
issue_num = 13691
keywords = ['patch']
message_count = 28.0
messages = ['150427', '150750', '150778', '150783', '150799', '151335', '151463', '162585', '162590', '162593', '162595', '162597', '162608', '172356', '172358', '172359', '172593', '172603', '172605', '248686', '248694', '272588', '288109', '288112', '291365', '291371', '291372', '295401']
nosy_count = 13.0
nosy_names = ['terry.reedy', 'ncoghlan', 'Devin Jeanpierre', 'ezio.melotti', 'eric.araujo', 'r.david.murray', 'mikehoy', 'jbitcm-', 'serhiy.storchaka', 'Cubky', 'jairotrad', 'jaysinh.shukla', 'CuriousLearner']
pr_nums = ['172']
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13691'
versions = ['Python 3.6', 'Python 3.7']

[ssbr]

What follows is a copy-paste of a shell session. Notice that at the end, rather than being inside the online help utility, I'm still in the interactive interpreter. I was able to duplicate this on python3.2, python2.7, and python2.6 (verifying it on other versions would have required installing them). Reading the source in trunk, there is nothing that looks like it actually should run this interactive help session. It's just missing.

I guess nobody used this, eh? I've attached a patch that should fix it. I'm not sure how you want to handle adding a test for this, so please advise me on that.

-----

>> help('help')

Welcome to Python 3.2! This is the online help utility.

If this is your first time using Python, you should definitely check out
the tutorial on the Internet at http://docs.python.org/tutorial/.

Enter the name of any module, keyword, or topic to get help on writing
Python programs and using Python modules. To quit this help utility and
return to the interpreter, just type "quit".

To get a list of available modules, keywords, or topics, type "modules",
"keywords", or "topics". Each module also comes with a one-line summary
of what it does; to list the modules whose summaries contain a given word
such as "spam", type "modules spam".

>>

[merwok]

IMO, help('help') should document the help function, not start an interactive help session (that’d be help()).

[terryjreedy]

I agree. It should explain the three options:
help(object): help on object or class of object, except
help('name'): help on object/module named 'name'
help(): run utility, which starts with utility help

help(help) prints unhelpful
"Help on _Helper in module site object:

class _Helper(builtins.object)
 |  Define the builtin 'help'.
 |  This is a wrapper around pydoc.help (with a twist).
 |  
 |  Methods defined here:
...
"
which says not at all how to use help().

I have the feeling that there is a related issue on the tracker, but searching for 'help', 'help()', or 'help(help)' all return 620 hits.

[ssbr]

IMO, help('help') should document the help function, not start an interactive help session (that’d be help()).

Ahhh, that explains it. help('help') isn't ever meant to be called; it's supposed to be:

>>> help()
...
help> help
...

the call to "help" at the help> prompt is equivalent to help('help').

help('help') should do something different, such as say how to use the help function, as you say. Whereas "help> help" makes sense.

[merwok]

I’m interested in this, so I’m taking assignment. If a contributor is interested in making a patch I’ll review it, otherwise I’ll say when I start on a patch.

[jbitcm]

I am working on it

[merwok]

Just a heads-up: I’ll be offline between January 19 and the end of the month, so don’t worry if you make a patch and it’s not reviewed immediately (at least not by me, other developers may do it :)

[Cubky]

Is anyone still working on this? If not I would like to make this work.
Although I'm not exactly sure how to tackle this problem since the built-in help function is defined as a wrapper around pydoc.help and so the docstring used for printing help('help') is actually a docstring of the site._Helper() class.
I made a patch that adds a documentation-like message to the pydoc.help function when it is called with a 'help' argument, but it doesn't seem clean to me.

[bitdancer]

It doesn't look like anyone is working on it currently.

Yes, getting this right is a bit tricky for exactly the reason you say. You have to make sure you don't break the other uses, and glancing at the patch (I haven't looked at the code you are patching) it seems likely that the change to the if statement is going to break *something*. Probably what happens when you type 'help' at the 'help>' prompt, at a guess.

[Cubky]

Help at the help> prompt does work, I tested that. But if you (or anyone) thought of a better way to fix this issue I would be glad to change it.

[bitdancer]

"works" in the sense that it produces output. But it doesn't produce the output it used to, which is the same text as the initial banner when you type help(). I believe the existing behavior in this case is correct. With your patch it produces the same output as "help('help')" at the python prompt does, and what that output describes doesn't work at the 'help>' prompt.

Thanks for tackling this.

Hmm. How about adding the special case to _Helper.__call__?

[bitdancer]

Oh, that wouldn't fix 'pydoc help', would it. So there are probably two places where the special case has to be added, which means the code itself should go in pydoc and be called from _Helper.__call__ somehow. (Note, I'm just throwing out ideas here, I haven't looked carefully at the code.)

[Cubky]

I see. So calling help('help') should produce the documentation on the help() function and typing help at the help> prompt should print the help for the prompt.
Tricky indeed. I think I'll look at it during the day after tomorrow and post some results (if any).

[mikehoy]

I imported this patch on 3.4 and it worked as expected on my system.

help('help') brings up info about help()
help(object) prints the docstring
help() invokes the help

[merwok]

What about "pydoc help"?

[mikehoy]

> What about "pydoc help"?

$ ./python -m pydoc help

Help on built-in function help:

help(...)
Invoke the built-in help system.

    help()
    The interactive help system starts on the interpreter
    console.

    help(string)
    The string is looked up as the name of a module, function, class, method,
    keyword, or documentation topic, and a  help page is printed on the
    console.   

    help(object)
    Generates a help page on the given object.

Or did you mean inside the interpreter?

[merwok]

Cool! Can you add tests?

[mikehoy]

>> Cool! Can you add tests?

Nope, I can't add tests.

[merwok]

I was addressing the author of the patch :)

[jairotrad]

I just tested this issue in Python 3.6.0a0 and got this behavior:

help('help') brings

"Help on _Helper in module _sitebuiltins object:"

help(help) brings:
Help on _Helper in module _sitebuiltins object:

help() invokes the help command line.

So this was fixes on the way to python3.6, I can make the test for 3.4 but is 3.4 closed for bug fixing?

Can someone check this in 3.5?

[terryjreedy]

3.5 is the same as 3.6 in that now both help(help) and help('help') show the not terribly helpful "Help on _Helper in module site object:...". Buried in the text is
| Calling help() at the Python prompt starts an interactive help session.
| Calling help(thing) prints help for the python object 'thing'.

As before, I think the _Helpter stuff should be eliminated and
explanation added for help('topic') and a description of what 'topic's are specially recognized.

There has been this improvement
>>> help('jslfjslfdjlskfj')  # 3.4
no Python documentation found for 'jslfjslfdjlskfj'

>>> help('jsflksjflkjsl')  # 3.5
No Python documentation found for 'jsflksjflkjsl'.
Use help() to get the interactive help utility.
Use help(str) for help on the str class.

but the latter omits decription of what 'topic' will give special output. By experiment, keywords (help('if')), builtins, and unimported modules (help('itertools')) works.

Yes, 3.4 can be patched.

[bitdancer]

No 3.4 can't be patched, this is not a security issue. A patch to make the behvior what we want should be generated against either 3.5 or 3.6, and the committer will deal with any issues arising from 3.5 vs 3.6 issues.

[CuriousLearner]

Hi, I'm working on fixing this issue.

[ncoghlan]

Recapping the situations that need test cases before this can be merged:

• behaviour when "help" is entered at the interactive help prompt (as the current behaviour is correct and should *not* change)
• behaviour when calling "help(help)"
• behaviour when calling "help('help')"
• behaviour when running "python3 -m pydoc help"

In the latter 3 cases, the default site._Helper docs currently shown are not particularly helpful.

It does make me wonder whether it might be worth defining a __help__ magic method that completely overrides what help(obj) displays. It would limit the fix to 3.7+, but it would make it possible to do this cleanly just by defining __help__ on site._Helper.

[ncoghlan]

After reviewing Sanyam's PR at #172 I ended up rejecting it as an approach to resolving this issue.

The core problem with the approach is that accessing _sitebuiltins._Helper is an implementation detail twice over:

• Python implementations aren't required to have a _sitebuiltins module
• that module isn't required to contain a _Helper class

That means importing it into a general purpose module like pydoc would couple pydoc specifically to the way CPython works, which is something we try to avoid doing.

By contrast, a richer __help__() protocol to override the __doc__ based default behaviour would allow any object to implement it, and _sitebuiltins._Helper could just use that protocol to special case itself.

[serhiy-storchaka]

It does make me wonder whether it might be worth defining a __help__ magic method that completely overrides what help(obj) displays.

This should be very complex protocol if take to account that help() outputs formatted and highlighted text and pydoc can output to html format, and alternative Python shells can provide alternative help().

[ncoghlan]

Aye, it's definitely a tricky problem. While I didn't take them into account in my review, I now realise those issues also impacted the attempted workaround - I'd be surprised if it worked as intended in those other contexts (even the "pydoc -b" rendered HTML view).

As far as how to deal with the rendering complexity in a protocol, one possible way to go would be to say that rather than returning a fully rendered string, __help__() methods should return either a Python function object or a type() instance (that doesn't define __help__()), and then pydoc would render the help based on that.

Regardless of the exact approach, it would be a task requiring a PEP to resolve, since rendered documentation appears in so many different contexts.

[ncoghlan]

Belatedly removing the "easy" tag, since that turned out to be thoroughly incorrect...