Fix completion issue

[casatir]

Completion implemented in #1155 is not fully working with JQuery terminal in console.html. It actually drops every completions that does not start with the prefix it computes (using whitespace splitting). See this comment.

One way to fix this is to prepend the prefix to each completion choice but this PR goes another way. CPython has a builtin completion system to work with readline called rlcompleter which you can use without standalone. This is the completion used in CPython's REPL. This will result in a faster completion experience, without any extra dependency (CPython's core instead of jedi/parso). Moreover, it already computes the completions with the prefix JQuery terminal is looking for.

This PR re-implements the completion in InteractiveConsole while fixing the issue exposed in this comment.

[rth]

So if I understand correctly, IPython is also using rlcompleter with an option use of jedi if it's installed? https://github.com/ipython/ipython/blob/master/IPython/core/completer.py#L68 Then rlcompleter should indeed be sufficient for us as well. jedi package is 1.4 MB compressed so it's clearly not negligible for web usage.

[casatir]

One little issue remains. Completion works only on prefixes preceded by a whitespace as JQuery Terminal split words at whitespaces.

>>> str.isa <tab> <tab>
str.isalnum(        str.isalpha(        str.isascii(
>>> a = 5 ; str.isa <tab> <tab>
str.isalnum(        str.isalpha(        str.isascii(
>>> a = 5 ;str.isa <tab> <tab>  # no proposition

This is known upstream and could be fixed with wordAutocomplete: false and a bit of work. Should this be fixed in this PR? WDYT?

[rth]

Should this be fixed in this PR? WDYT?

I think having this PR as is is already good. I'll try to review later today. We could always improve it in a follow up PR.

[casatir]

So if I understand correctly, IPython is also using rlcompleter with an option use of jedi if it's installed? https://github.com/ipython/ipython/blob/master/IPython/core/completer.py#L68

Thanks, haven't noticed:

This module started as fork of the rlcompleter module in the Python standard
library. The original enhancements made to rlcompleter have been sent
upstream and were accepted as of Python 2.3,

Then rlcompleter should indeed be sufficient for us as well. jedi package is 1.4 MB compressed so it's clearly not negligible for web usage.

Totally agree.

[casatir]

I think having this PR as is is already good. I'll try to review later today. We could always improve it in a follow up PR.

Great!

[hoodmane]

By the way, a major concern with using jedi for autocompletions is that jedi can cause fatal errors #821. It isn't great if requesting a completion causes the Python runtime to segfault. Surely rlcompleter doesn't suffer from this issue so I think it's a much better idea to use it over jedi.

[rth]

Minor comment otherwise LGTM.

Could you please add separate changelog entry for the switch from jedi to rlcompleter in console.html?

[rth]

This versions seems faster,

In [10]: %%timeit 
    ...: regex = '|'.join(map(re.escape, completer_word_break_characters)
    ...: ) 
    ...: re.search(regex, source[::-1]) 
    ...:  
    ...:                                                                 
8.74 µs ± 18.7 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [11]: %timeit max(map(source.rfind, completer_word_break_characters)) 
    ...: + 1                                                             

2.7 µs ± 11.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each

particularly given that this benchmark is biased toward regex since regex compilation will be cached.

So I think we can remove the comment. In any case it's µs so it shouldn't matter much.

[casatir]

Great! Thanks for the benchmark!

[rth]

It's strange that mypy can't find these attributes, type annotations are likely incorrect somewhere in CPython. Not much we can do about it on our side.