rlcompleter should be enabled automatically

[cben]

BPO	5845
Nosy	@loewis, @ronaldoussoren, @mdickinson, @cben, @jaraco, @abalkin, @pitrou, @larryhastings, @giampaolo, @ned-deily, @ezio-melotti, @merwok, @stevendaprano, @bitdancer, @florentx, @xdegaye, @serhiy-storchaka
Files	fix-5845.diff c43e264256e4.diff defaultreadline.patch issue5845_osx_fix.patch completer.py completer.py history.py rl_history_guard.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-12-08.19:18:07.030>
created_at = <Date 2009-04-25.23:27:18.758>
labels = ['deferred-blocker', 'type-feature', 'library']
title = 'rlcompleter should be enabled automatically'
updated_at = <Date 2015-06-13.20:09:00.871>
user = 'https://github.com/cben'

bugs.python.org fields:

activity = <Date 2015-06-13.20:09:00.871>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2013-12-08.19:18:07.030>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2009-04-25.23:27:18.758>
creator = 'cben'
dependencies = []
files = ['22936', '30098', '30099', '30148', '30149', '31095', '31096', '31914']
hgrepos = ['9']
issue_num = 5845
keywords = ['patch']
message_count = 66.0
messages = ['86570', '86632', '86660', '123696', '123697', '123698', '123703', '123705', '123709', '132117', '132120', '132158', '142371', '142393', '142395', '143548', '143554', '143608', '143609', '143618', '143621', '143961', '188241', '188385', '188386', '188505', '188511', '188517', '188521', '188523', '188524', '188525', '188532', '188533', '188534', '188535', '188540', '188542', '188543', '188549', '188552', '188553', '188555', '188557', '188560', '188589', '188592', '193981', '193986', '194006', '194010', '194427', '198636', '198640', '198642', '198645', '198648', '198653', '205569', '205595', '209472', '212972', '215784', '215975', '245325', '245328']
nosy_count = 23.0
nosy_names = ['loewis', 'ronaldoussoren', 'mark.dickinson', 'cben', 'jaraco', 'belopolsky', 'pitrou', 'larry', 'techtonik', 'giampaolo.rodola', 'ned.deily', 'ezio.melotti', 'eric.araujo', 'steven.daprano', 'r.david.murray', 'flox', 'dabeaz', 'lesmana', 'xdegaye', 'tshepang', 'python-dev', 'serhiy.storchaka', 'Victor.Varvariuc']
pr_nums = []
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue5845'
versions = ['Python 3.4']

[cben]

An interactive prompt should offer working completion out-of-the-box,
without requiring every Python user on earth to create a $PYTHONSTARTUP
with '''import readline; readline.parse_and_bind("tab: complete")'''.

Note that it should work not only when Python is run without arguments,
but also when running with the -i option.
And ideally, it should only install the completion function when
dropping to the interpreter, not from the start, so that code that runs
before dropping to the interpreter is not affected. But this is really
not important.

Safety: if 'rlcompleter' or 'readline' are not availiable (e.g. broken
python install), the user must still get a working interpreter, just
without completion.

Portability: there won't be completion on windows (as long as pyreadline
or similar is not part of Python). But that shouldn't delay unix support!

[merwok]

What would be the right place for this to happen? I first thought of
site.py, but -S would then turn off readline support. Does it have to be
done somewhere in C land?

But perhaps tab isn’t the right key to bind. I think inputrc could set
it to something different, perhaps shell rc files too. Is there an API
to get this setting, does readline.so already do the right thing, or
does it have to be added?

Moreover, in my PYTHONSTARTUP I not only bind tab to complete, but also
set a history file. Should this be automatic too? UNIX rules would
suggest a standard ~/.pyhistory file, or we could follow newer
freedesktop.org specifications [1].

So, I don’t know the history and choices behind readline.so, but from a
user point of view, being fed up with lack of tab completion finally had
me put up a PYTHONSTARTUP file with these settings, and then some useful
functions, and so on. This lack of completion was ultimately a good
thing for me. Still, completion could be made to work out of the box,
and the man page could link to the doc explaining how to customize this
behavior.

[1] XDG Base Directory Specification:
http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

Cheers, Merwok

[pitrou]

He, adding completion is also something I find myself adding on every
box on which I use Python...

[bitdancer]

for what it is worth, I am +1 on having completion and history file work by default. The sqlite3 command line does this, for example. I think it is what unix user expect nowadays, and I think it is reasonable.

Looking at my home directory, it would appear that the de-facto standard history file name would be .python_history. This is based on .mysql_history, .psql_history, and .sqlite_history, none of which I configured explicitly to support history saving.

[merwok]

Okay, that’s one question answered. Still to solve: How to bind the right key? (“But perhaps tab isn’t the right key to bind. I think inputrc could set it to something different, perhaps shell rc files too. Is there an API to get this setting, does readline.so already do the right thing, or does it have to be added?”)

[pitrou]

I think TAB is the key expected by most people, so let's make it the default.
As for the location, site.py is an adequate one IMO.

[ned-deily]

Keep in mind that the Python readline module may be linked to either GNU readline or the BSD editline (libedit) library and they have different command strings. Note the warning here:

http://docs.python.org/dev/py3k/library/readline.html

Here's a snippet of what I have today in my startup file:

    import rlcompleter
    if 'libedit' in readline.__doc__:
        readline.parse_and_bind("bind ^I rl_complete")
    else:
        readline.parse_and_bind("tab: complete")

See bpo-10666 for more details and some possible changes.

[ned-deily]

Nosying Martin: any Windows installer concerns?

[loewis]

There is no readline support on Windows at all, so I don't think the Windows installer can be affected. I'm uncertain what the proposed change to Python is at this point, though.

[merwok]

Ned, does readline.read_init_file() with libedit?

[ned-deily]

readline.read_init_file() does work with libedit. The directives read have to be in libedit format.

[merwok]

Please ignore the generated patches, left here for Martin’s benefit, and review the manually attached fix-5845.diff.

I have used read_init_file instead of hard-coding tab (it’s anyway the default for readline).

I may have repeated the same info in too much doc files, but I preferred to have more than one place where people could learn about the new behavior. Advice welcomed.

If the basic approach looks good, especially if site still feels the right place for this, I will add tests.

[merwok]

Updated patch (not using Mercurial, looks like I haven’t enough bandwidth to push all those changesets).

[pitrou]

Updated patch (not using Mercurial, looks like I haven’t enough
bandwidth to push all those changesets).

I think there's a bug in the way you are detecting interactive mode:

$ ./python -c "import sys; print(sys.stdin.isatty())"
True

[pitrou]

Interestingly, there's already the following code in Modules/main.c:

    if ((Py_InspectFlag || (command == NULL && filename == NULL && module == NULL)) &&
        isatty(fileno(stdin))) {
        PyObject *v;
        v = PyImport_ImportModule("readline");
        if (v == NULL)
            PyErr_Clear();
        else
            Py_DECREF(v);
    }

...meaning readline already gets imported automatically when desired. And indeed:

$ ./python -S
Python 3.3.0a0 (default:50f1922bc1d5, Aug 18 2011, 00:09:47) 
[GCC 4.5.2] on linux2
>>> import sys
>>> sys.modules['readline']
<module 'readline' from '/home/antoine/cpython/default/build/lib.linux-x86_64-3.3-pydebug/readline.cpython-33dm.so'>

So perhaps we could simply change this code to import another, private module (e.g. "_setupinteractive.py") which would setup readline and rlcompleter?

[merwok]

We can’t just decide to enable completion by checking “'readline' in sys.modules” in site, because it always returns False. site is imported in Python/pythonrun.c, and I guess that Modules/main.c is run afterwards. More importantly, I think the import of readline in Modules/main.c is a CPython implementation detail, and I’d prefer to have code in site that is directly useful for other VMs. (I’d go as far as proposing to remove the import readline from main.c, when my patch makes it obsolete.)

I re-read the docs for sys.argv and the -m switch and decided to translate to C checks (“command == NULL && filename == NULL && module == NULL”) to “if not sys.argv[0]”. Unfortunately, sys.argv is not available when site is imported, so this is a dead end :(

So, I could try your _setupinteractive module idea, even though I think it’s a bit unclean, or drink the cool aid and set up completion in main.c.

[cben]

Easily detecting interactive mode is of general interest for customization.

1. What if C also set sys.flags.interactive in "python" mode,
   or exposed sys.flags.implicit_interactive (but with better name)?

2. It's more useful to have a hook called when entering interactive mode,
   rather than a flag that's set from the beginning:

   $ python -i -c 'import sys; print sys.flags.interactive'
   1
   >>>

   For this, importing _setupinteractive is a step forward;
   calling e.g. sys.__interactivehook__ sounds even better.
   (site.py would set it by default to a function that enables
   rlcompleter, user can always override...)

BTW, drawback of doing any such setup in site.py: "python -S" would
be unfriendly!

[merwok]

Easily detecting interactive mode is of general interest for
customization.
Thanks for the feedback. We could open a feature request for that, and/or ask python-ideas.

What if C also set sys.flags.interactive in "python" mode, or exposed
sys.flags.implicit_interactive (but with better name)?
An attribute in the sys module sounds like a good idea. It would not be an attribute of sys.flags though, as those are strictly command-line arguments.

It's more useful to have a hook called when entering interactive mode, rather than a flag
that's set from the beginning:
We already have such a hook: $PYTHONSTARTUP

calling e.g. sys.__interactivehook__ sounds even better.
That’s another interesting idea.

BTW, drawback of doing any such setup in site.py: "python -S" would be unfriendly!
People using -S don’t want the customizations done in site, so I don’t think there’s a problem here.

[pitrou]

> It's more useful to have a hook called when entering interactive mode, rather than a flag
> that's set from the beginning:
We already have such a hook: $PYTHONSTARTUP

$PYTHONSTARTUP doesn't work with -i

[cben]

[sorry, html mail was bad idea]

On Tue, Sep 6, 2011 at 17:54, Antoine Pitrou <report@bugs.python.org> wrote:

Éric Araujo <merwok@netwok.org> added the comment:

> It's more useful to have a hook called when entering interactive mode,
> rather than a flag
> that's set from the beginning:
We already have such a hook: $PYTHONSTARTUP

Good point!
It covers the user's desire customization very well (esp. if it worked with -i).
sys.__interactivehook__ has the benefit of being cleanly settable from python code.
But it might well be a YAGNI idea.

$PYTHONSTARTUP doesn't work with -i

Perhaps it should?
I can't think of a thing that makes sense in $PYTHONSTARTUP that I wouldn't want with -i.
(and if there is one, one can add a test for sys.flags.interactive, or run with env PYTHONSTARTUP='')

Point to watch out for: errors in $PYTHONSTARTUP.
One of the uses of "python -i script.py" is doing pdb.pm() on an exception thrown by the script;
ideally a broken $PYTHONSTARTUP would not overr

> BTW, drawback of doing any such setup in site.py: "python -S"
> would be unfriendly!
People using -S don’t want the customizations done in site, so I
don’t think there’s a problem here.

python -S doesn't disable readline. What makes completions more of a "customization" than editing?
The fact that it'd be implemented in site.py?
Yes, obviously, if it's implemented in site.py, -S should disable it. My point was that it doesn't have to be implemented there. You could drink the cool aid instead :-)

[merwok]

sys.__interactivehook__ has the benefit of being cleanly settable from python code.
But it might well be a YAGNI idea.
I’ll ask python-dev about that. For the moment, I prefer the idea of a new sys.interactive attribute (boolean).

> $PYTHONSTARTUP doesn't work with -i
Perhaps it should?
-i means two things: interactive and inspect. The second meaning is used when you want python to execute a script or module just as usual but without exiting afterward, to let you inspect the state of the program. Supporting PYTHONSTARTUP would change these use cases and be an unacceptable change.

> BTW, drawback of doing any such setup in site.py: "python -S"
> would be unfriendly!
People using -S don’t want the customizations done in site, so I
don’t think there’s a problem here.

python -S doesn't disable readline.
What makes completions more of a "customization" than editing?
The fact that it'd be implemented in site.py?
If you go back to the first messages of this report, you’ll see that I wasn’t sure what a good location would be, and decided site because it did not look out of place. There isn’t a stronger rationale than that. One could argue that even now, site mixes disparate functionality: one program could wish to use -S to disable sys.path munging but run into bugs because the builtin exit is not defined anymore for example.

Yes, obviously, if it's implemented in site.py, -S should disable it.
We agree.

My point was that it doesn't have to be implemented there.
I really want to explore all options before renouncing to implement it in site. I’d like the implementation to be maintainable and shareable with other Python VMs.

[merwok]

FTR, I tried checking sys.ps1 instead of argv but it’s the same problem.

[pitrou]

In the spirit of pushing this forward, here is an updated patch using the sys.__interactivehook__ approach. I didn't add any tests since it doesn't seem very easy to write any.

If nobody objects, I would like to commit this soon.

[python-dev]

New changeset d5ef330bac50 by Antoine Pitrou in branch 'default':
Issue bpo-5845: Enable tab-completion in the interactive interpreter by default, thanks to a new sys.__interactivehook__.
http://hg.python.org/cpython/rev/d5ef330bac50

[pitrou]

End-users will hopefully rejoice :)

[bitdancer]

Well, post it to python-dev and see what reaction you get :)

I could be wrong, but I don't think I am (obviously).

[ronaldoussoren]

I expect that a lot of users use the tab key to indent in the repl (as well as in editors, smart enough editors can convert the tab presses to spaces)

[pitrou]

Well, post it to python-dev and see what reaction you get :)

I'm not interested in python-dev reactions here. For this
kind of issue, we'll have hundreds of messages for and against
the change without any useful content.

The commit adds an often-requested feature (to the point that many
people were already taking the pain of activating it manually).
But, yes, we could add Steven's refinement so that all bases are
covered.

[pitrou]

I expect that a lot of users use the tab key to indent in the repl
(as well as in editors, smart enough editors can convert the tab
presses to spaces)

The interpreter prompt is not a text editor at all. You can misuse
it as one, but that's a loss of time and energy IMO.

Really, everyone I've ever showed tab-completion to has always
been enthusiastic about it ("why isn't this enabled by default?").
None has ever raised the concern of being able to use the tab key
to indent code under the prompt.

(probably because noone is crazy enough to type long chunks of
code under the interactive prompt anyway)

[abalkin]

I am in AP's camp on the tab issue, but I think we can preserve "tab inserts tab" behavior at the continuation prompt. I don't like "indent at beginning of line." I have rlcompleter enabled in Python 2.6 and i get the following when I press tab:

Python 2.6.6 (r266:84292, Feb 22 2013, 00:00:18)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> <tab>
Display all 173 possibilities? (y or n) y
ArithmeticError(            abs(                        input(
AssertionError(             all(                        int(
AttributeError(             and                         intern(
BaseException(              any(                        is
..

I find this rather useful.

At the ... prompt, however, a tab (or better four spaces) is arguably the right completion at the beginning of the line.

[pitrou]

Take it with a large grain of Internet salt, but Ezio pointed me to the following reactions:
http://www.reddit.com/r/Python/comments/1dq7sh/python_repl_finally_gets_tab_completion_by_default/

[tshepang]

Also made it to the front-page of Hacker News[1], with better quality comments than the reddit thread.

[1] https://news.ycombinator.com/item?id=5658062

[larryhastings]

This issue seems to have stalled. But it's still marked as a release blocker, which means I can't release Python 3.4a1 in two days if this issue is still open.

One of three things will happen:

1. This issue is marked "closed".
2. This issue is downgraded from "release blocker".
3. The alpha slips.

IMO the optimal solution is that tab preceded by only whitespace indents, and tab preceded by any non-whitespace character attempts to complete. Can we goad readline into behaving this way?

[stevendaprano]

On 31/07/13 17:14, Larry Hastings wrote:

IMO the optimal solution is that tab preceded by only whitespace indents, and tab preceded by any non-whitespace character attempts to complete. Can we goad readline into behaving this way?

Yes we can. Attached are a pair of lightweight modules I've used for this for the last 3-4 years, and it works fine in Python 2.4 through 3.3. I put something like this in my startup.py:

import completer
import history
history = history.History()
completer = completer.Completer(
         bindings=(r'"\C-xo": overwrite-mode',
                   r'"\C-xd": dump-functions',
                  ))

Originally they were in a single class, but I was persuaded that it was cleaner to separate them.

[loewis]

If R. David agrees (assuming no patch is coming forward), it could be degrated to deferred-blocker, and the alpha phases could be used to see how annoyed people are with "tab not working".

[bitdancer]

Yeah, deferred blocker is fine with me.

[serhiy-storchaka]

Do you know that the readline module first force rebind of TAB to insert-tab?

[xdegaye]

There is a backward compatibility issue with changeset d5ef330bac50
that enables tab-completion in the interactive interpreter by default.

When a user is not aware of this new feature and has been implementing
up to now her/his PYTHONSTARTUP file with the first example given in
python 3.3 documentation at

http://docs.python.org/3/library/readline.html?highlight=readline#example

then each new interactive python 3.4 session reads (appending them)
both history files which are saved on exit. Thus those files double
their size after each interactive python 3.4 session and pretty soon
become few mega bytes long with a python start up time that becomes
quite noticeable.

[pitrou]

When a user is not aware of this new feature and has been implementing
up to now her/his PYTHONSTARTUP file with the first example given in
python 3.3 documentation at

http://docs.python.org/3/library/readline.html?highlight=readline#example

then each new interactive python 3.4 session reads (appending them)
both history files which are saved on exit. Thus those files double
their size after each interactive python 3.4 session and pretty soon
become few mega bytes long with a python start up time that becomes
quite noticeable.

Interesting. A way to avoid doing this would be to only load history if
no history is already present. Since the sys.__interactivehook__ is
executed after PYTHONSTARTUP, this would allow the latter to take
precedence.

[pitrou]

Attaching patch to only load history file if no history exists.

[python-dev]

New changeset 687dd81cee3b by Antoine Pitrou in branch 'default':
Issue bpo-5845: In site.py, only load readline history from ~/.python_history if no history has been read already. This avoids double writes to the history file at shutdown.
http://hg.python.org/cpython/rev/687dd81cee3b

[xdegaye]

The patch fixes the problem on my setup. A very minor glitch: after
manually emptying or removing the PYTHONSTARTUP history file, the
history is loaded with the content of ~/.python_history on the next
session.

[pitrou]

The patch fixes the problem on my setup. A very minor glitch: after
manually emptying or removing the PYTHONSTARTUP history file, the
history is loaded with the content of ~/.python_history on the next
session.

Yes, the only way to know if a history file has already been loaded with
readline seems to be to query the history itself. So, if the history is
empty, we consider no file has been loaded.

[serhiy-storchaka]

What is the status of this issue?

[pitrou]

I would say it's now closed. If there's some fine tuning needed, separate issues should be opened.

[jaraco]

Added bpo-20411, revealed in 3.4b2.

[python-dev]

New changeset 69c451851c71 by R David Murray in branch 'default':
whatsnew: sys.__interactivehook__. (bpo-5845)
http://hg.python.org/cpython/rev/69c451851c71

[dabeaz]

Funny thing, this feature breaks the interactive interpreter in the most basic way on OS X systems. For example, the tab key won't even work to indent. You can't even type the most basic programs into the interactive interpreter. For example:

>>> for i in range(10):
...      print(i)

Oh sure, you can make it work by typing the space bar a bunch of times, but it's extremely annoying.

The only way I was able to get a working interactive interpreter on my machine was to manually edit site.py and remove the call to enablerlcompleter() from main().

I hope someone reconsiders this feature and removes it as default behavior.

[bitdancer]

This issue should have gone back to being a release blocker after the alpha release to fix the tab-as-indent issue, but obviously that didn't happen (I forgot about it myself). Please open a new issue requesting a fix for this bug (that tab doesn't work as indent at the ... prompt), referencing the discussion in this issue.

[VictorVarvariuc]

Here is the new issue (putting the link here for reference): http://bugs.python.org/issue22086

[bitdancer]

The active issue for this is now 23441.