Improve autocomplete UX

[mlucool]

Hi,

IPython recently introduced hinted suggestions as input is being typed. The user can accept the suggestion by pressing END or RIGHT. Alternatively, the user can scroll through other prefix matches in the history by pressing UP (and DOWN). However, all these modes of navigation move the cursor to the end of the line. It is impossible to accept a prefix of a hinted suggestion.

We would like to propose keeping the cursor in place and naturally supporting the cursor movement with the logic of “accept anything that is on the LHS of the cursor”. This would allow reusing long prefixes without the need to accept the entire line and make edits from its end.

For example, suppose the history includes
very.long.module.foo.func()
very.long.module.bar.Baz()
The user types the prefix “very.”. The hinted suggestion would be “long.module.bar.Baz()”. The user can now press:

1. RIGHT/CTRL-RIGHT to accept letters/words from the hinted completion. This would make it easier to type “very.long.module.bar.Zap()”.
2. END to accept the entire suggestion.
3. UP to replace the hinted suggestion with “long.module.foo.func()” (cursor stays in place).
4. BACKSPACE to delete last character and resume hinting from the new prefix (currently, it aborts hinting).
5. LEFT to accept the hint and move the cursor to the left.
6. New key binding to accept the suggestion but keep cursor in place.

Thoughts?

[krassowski]

I like this proposal, especially suggestions (1), (2), (3), (4).

Currently accepting word-by-word is possible using Alt + f shortcut (or Esc followed by f), but the behaviour could be improved and better documented. Currently the definition of "word" is very simplistic (based on spaces only), e.g. def output_many_nodes(tag: str, n=50000): completes in order:

• def
• output_many_nodes(tag:
• str,
• n=50000):

To achieve property-by-property completion (very.long.module.foo.func()) we should use a proper Python tokenizer instead, possibly merging consecutive 1-character tokens together, so that n=, ): and () are completed together.

(5) LEFT to accept the hint and move the cursor to the left.

This might not always what the user wants and even annoying outside the happy path. If the user makes a typo left to the cursor and get incorrect hint as a consequence, they may want to edit the typo and then get resumed hinting rather than get the (incorrect) hint accepted (and then need to fix both the typo and remove incorrect hint)

(6) New key binding to accept the suggestion but keep cursor in place.

There are already many key bindings for this feature and more were proposed (#12589). It would be fine if we can find something which is intuitive for most users, but otherwise maybe this could be behind a setting to avoid adding too many default key bindings?

[krassowski]

Fish-shell also uses Alt + → (in addition to Alt + f) for word-by-word suggestions (ref).

[mlucool]

This might not always what the user wants and even annoying outside the happy path.

That makes sense, let's drop 5.

It would be fine if we can find something which is intuitive for most users, but otherwise maybe this could be behind a setting to avoid adding too many default key bindings?

Yes. Similar to #13879 (comment), maybe we should allow for much greater user customization of key-bindings.

[Carreau]

Trying @krassowski implementation the main problem I feel with UP to replace the hinted suggestion with “long.module.foo.func()” (cursor stays in place). Is that it becomes hard to navigate to previous line as it's not obvious how to dismiss the autosuggestion.

One of the question we can add is what difference do you see between the completer and autosuggest in practice (right now, completer inspect and autosuggest is from history).
Should we extend the completion pop up to also have suggestion from history ?

[kylebarron]

I was about to make a "regression in 8.9" issue until I saw this in the release notes. It's really awesome to see work being done on this, but very much looking forward to having these settings be configurable. Using → to autocomplete the entire line is very hard-wired into my brain, and zsh's autocomplete also uses → to autocomplete the entire line, so it's pretty hard to deal with the context switching of different commands to use in different terminals.

[maple3142]

Not being able to autocomplete the suggestion with a single → in IPython 8.9.0 is pretty annoying, especially https://github.com/zsh-users/zsh-autosuggestions also completes the entire suggestion with a single →.

How about changing the default behavior of → to accept the full suggestion or make it configurable instead?

Temporary workaround:
Change

ipython/IPython/terminal/shortcuts/__init__.py

Lines 390 to 392 in 14faa28

	kb.add("right", filter=has_suggestion & has_focus(DEFAULT_BUFFER))(
	auto_suggest.accept_character
	)

to

 kb.add("right", filter=has_suggestion & has_focus(DEFAULT_BUFFER))( 
     auto_suggest.accept
 ) 

It might be necessary to delete __pycache__ too if you are editing files under /usr/lib/python3.10/site-packages/IPython.

[krassowski]

Thank you all for the feedback! Reopening to make this issue easier to find.

I plan to work on enabling shortcut customisation this weekend. In the meantime, you do not need to modify IPython code to revert to the previous behaviour - it is sufficient to hook in the following line:

get_ipython().pt_app.key_bindings.remove('right')

Please also note that you can use Ctrl + Right to complete token-by-token.

[oscarbenjamin]

it is sufficient to hook in the following line:

get_ipython().pt_app.key_bindings.remove('right')

Thanks. The one I was looking for is

get_ipython().pt_app.key_bindings.remove('up')

Is there a way to make that setting persistent?

The behaviour that I use a lot is to type the first character(s) of a previously entered line and then push up (potentially more than once) and then enter to rerun the line (usually without any modifications). For example the line might be edit temp.py and I probably end up running it repeatedly which I can do very quickly with e, up, enter but sometimes it will be more like e, up, up, enter or perhaps a bit of up and down to find the line I want.

Now the most recent match is shown automatically after e and when I press up the previous matches are shown but enter will not run the whole line unless I first press the end key. Now it's e, end, enter for the most recent match or e, up, end, enter and going from up to end is awkward on this keyboard. It also means I have to choose whether to press e, up or e, end depending on whether I want the most recent match because if you press e, end then it's too late to try pressing up afterwards and if you press e, up then you've already skipped the most likely match. Previously it was a very quick e, up then I have a split second to see if it's the right line and I can choose between enter or pressing up again.

Also not all keyboards have an end key e.g. I have a Mac laptop which does not. Ctrl-e works there instead but that's also a more awkward key combination to aim for than just enter.

[krassowski]

Thank you for a very detailed example. As mentioned before, I am working on enabling persistent shortcut customisation (including disabling shortcuts). You can also use esc to dismiss the autosuggestions, this will allow you to use up/down.

[t4rf9]

I see complaints about the right behavior change, which I also bother with.
I use right to accept a suggested command quite often as well.
Since the large amount of users who may be used to this, I recommend the old behavior to be set as default.

[cottrell]

Thank you for a very detailed example. As mentioned before, I am working on enabling persistent shortcut customisation (including disabling shortcuts). You can also use esc to dismiss the autosuggestions, this will allow you to use up/down.

Great! I hope all the cranky people like me don't dissuade you too much from contributing. It's always a pain making changes to structure to heavily used and important projects precisely because of the humans who have baked in certain muscle memory for usage. Thanks for all the work.

[krassowski]

I would like to propose that we restore revert right to previous behaviour (complete full suggestion, currently possible with end) and assign the current "complete a character" to alt + right. This means that the following shortcuts will be available to complete the auto-suggestion:

• right or end will complete everything and moves the cursor to the end
• ctrl + down complete everything and keeps the cursor in-place
• ctrl + left complete everything and moves the cursor left
• ctrl + right complete a token/word
• alt + right complete a single character

Additionally, esc discards the auto-suggestion.

#13928 implements shortcuts customization via TerminalInteractiveShell.shortcuts traitlet (configurable via Python/JSON file or using %config magic) to make shortcut customization easy.

[krassowski]

As for navigating up/down, this can be disabled by changing the auto-suggestion provider (TerminalInteractiveShell.autosuggestions_provider) from the default 'NavigableAutoSuggestFromHistory' to 'AutoSuggestFromHistory'. There is a small bug with this config traitlet which was fixed (#13914) but not yet released, so apologies if it does not work for you just yet.

I was expecting more feedback on history navigation (but most comments focused on right change), so it is not as clear if we want to revert to old behaviour here. Any further feedback (especially from all off you who commented/added a reaction about the right arrow behaviour but have not mentioned up/down behaviour) is welcome!

[csm10495]

Agree that we should rollback to previous behavior.

Ultimately to me it makes more sense to make it configurable so folks can change it if they want as opposed to forcing changes to years built muscle memory.

[pmlandwehr]

I'm a fan of the old behavior, given my muscle memory, but a quick right right to autocomplete an entire line would also be a nice fix. (Might be quite hard to implement though?)

(Anaconda has I believe just pushed the new ipython version to their defaults.)

[GuillaumeLeclerc]

Any idea when this will land in the new releases. It is really hard to overcome muscle memory.

[krassowski]

Usually new version gets released last Friday of the month which happens to be next week.

[krassowski]

Would anyone here be interested to vote on a proposal to add another shortcut: #13987 or share some constructive thoughts over there?

[oscarbenjamin]

I have been trying out the new behaviour for up/down for a month now and I still don't understand why it would be preferred to the previous behaviour. The previous behaviour that you type a partial match and push up to get a previous command is not just an IPython feature as it is the same behaviour in other REPLs like Julia, Matlab, Sage etc. I think Sage actually uses IPython or some modification of it. It is likely that Julia and Matlab actually copied this behaviour from IPython because they could see it was useful. This change now makes IPython inconsistent with the others though because they all use a single up to complete the most recent command whereas IPython now completes the second most recent command.

Where IPython is different from those others is that it has the ability to paste a block of lines and have it treated as a single command in the command history which is great for e.g. pasting out of a GitHub issue. However completion of multiline blocks is now broken from this change. For example copy this:

long_list = [1, 2, 3, 4]
print(sum(long_list))

Paste it into IPython (after running other commands):

In [1]: a = 1

In [2]: b = 2

In [3]: long_list = [1, 2, 3, 4]
   ...: print(sum(long_list))
10

Now if you push up you can rerun the whole block of two lines which is great. However if you push up twice you don't get back to b = 2 because you have to scroll through each line of the multiline block (which is often much longer than two lines). So to get back to the a = 1 line skipping over any multiline blocks you can type a and complete that line (previously using up but now only using right).

However there is no longer any way to complete the multiline block. If you type the first letters of "long_list" then the completion with right or up will only offer the first line of that block. Meaning that you have to specify each line one by one. In previous IPython versions you could type long or just l and then push up to complete the whole block.

Using get_ipython().pt_app.key_bindings.remove("up") does restore the previous behaviour but I don't think that the changed behaviour of the up/down keys should be enabled by default. If you are pushing up then it should be to get the most recent matching item and not the penultimate match and it should not be necessary to do up *right* enter when previously up enter was sufficient and is what other similar systems use.

[cottrell]

Note to see here #13938 ... I think a rollback to previous behaviour is still not yet in.

[krassowski]

@oscarbenjamin

Using get_ipython().pt_app.key_bindings.remove("up") does restore the previous behaviour

I would not recommend this workaround anymore. Instead you can revert to the previous navigation behaviour by modifying TerminalInteractiveShell.autosuggestions_provider as explained in #13878 (comment) and #13979 (comment) (with the latter tracking exposing it more in user-facing documentation).

but I don't think that the changed behaviour of the up/down keys should be enabled by default.

This is fair. There is a number of opinions on this topic and I respect yours. Since this issue is now closed and has grown long with a discussion on multiple aspects of the autosuggestion changes, would you mind opening a new issue to track your proposal of switching the default to:

TerminalInteractiveShell.autosuggestions_provider = 'AutoSuggestFromHistory'

However there is no longer any way to complete the multiline block. If you type the first letters of "long_list" then the completion with right or up will only offer the first line of that block. [...] In previous IPython versions you could type long or just l and then push up to complete the whole block.

I see what you mean. In the code terms we do not call it "completion" but "navigating to history with search". There is a way to do so without disabling navigable suggestions provider:

• type long or just l
• discard the suggestion (currently you need to press Esc and wait 2 seconds; with proposed Del binding there would be no delay)
• press Up

The delay after pressing Esc is required to allow for composite keys like Esc + F (which I do not like but this is what existing keybindings are)

[krassowski]

@cottrell I closed the issue that you linked in favour of #13979 as #13938 did not have any outstanding action items (other than improvements to the documentation) and was conflating expectations from completer and autosuggestions.

[oscarbenjamin]

Thanks @krassowski.

would you mind opening a new issue to track your proposal of switching the default to:

TerminalInteractiveShell.autosuggestions_provider = 'AutoSuggestFromHistory'

Is there any difference between that and the existing issue gh-13979?

Btw I tried this:

In [5]: from IPython.terminal.interactiveshell import TerminalInteractiveShell

In [6]: TerminalInteractiveShell.autosuggestions_provider = 'AutoSuggestFromHistory'

But it did not seem to change anything.

[krassowski]

Is there any difference between that and the existing issue #13979?

Not a substantial one, but 13979 is formulated as a user question which I will answer with a PR improving user-facing documentation. I would prefer to have a separate issue requesting change of behaviour in the title so that contributors and active users who subscribe to issues (by getting notifications or emails) would be explicitly notified and know that they should vote one way or the other (as with #13987).

But it did not seem to change anything.

If you want to change a config from REPL you need to run:

%config TerminalInteractiveShell.autosuggestions_provider = 'AutoSuggestFromHistory'

See Introduction to IPython configuration.

[oscarbenjamin]

I would prefer to have a separate issue requesting change of behaviour in the title

I opened gh-13993 for this.

[lfagundes]

Thanks for all community efforts to improve this.

My productivity has been severely burdened by the loss of the autocomplete with the right arrow key. My fingers are faster than the ssh connections so I can't rely on visual feedback. My finger joints have endured 35 yrs of long typing journeys and my left pinkie complains immediately when I have to hold the ctrl key for so long while hitting the arrow 8 times. My autism does not help to adapt.

Today I decided to dig into this for real, read changelogs and discussions, and my conclusion is that the best to do is just hold on v8.8 for as long as it's possible, because it's too much to follow just to be able to keep coding normally, and I'm already too burdened by this issue.

I don't know if this is a proper place for this message, but this is how far I could get, and just needed to express that having to be aware of so much debate for just keep coding as I used too is not accessible for me. I don't want to complain, just to raise awareness so we can better deal with UX improvements.

Balancing changes on an production software is a huge challenge. I suggest next time this kind of change be a major release, not a minor one.

[cottrell]

@lfagundes Thanks for sharing. I am also frozen at v8.7.0 for acccessibility reasons and am wondering if you might create a new issue for people like us to subscribe to so that we might know when to test a new version where the old, accessible autocomplete API is available for testing. It would avoid the spam that is caused when we reply to these old threads.

[krassowski]

@lfagundes @cottrell have you tried ipython 8.14.0 released two days ago?

[RuRo]

@krassowski The default autocomplete in version 8.14.0 is still different from what it used to be.

[cottrell]

@krassowski just tried it out quickly and so far so good! Will try it out full time and report back if I notice anything.

[lfagundes]

@lfagundes @cottrell have you tried ipython 8.14.0 released two days ago?

It took me a while to be able to install it, it's great, thanks!

[cottrell]

@krassowski I've been using it for a while now and feels great. Thanks!