Elpy seems partially incompatible with Emacs 25's 'native completion' feature

[cpitclaudel]

In Emacs 25, when starting Python with C-c C-c, I get the following message:

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "ipython3" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list. Native completions have been disabled locally.

On the other hand, if I disable elpy, start the repl, and re-enable it, everything works fine. Is elpy introducing a process filter for the python process, or something similar? That warning comes from this bit of python.el:

(defun python-shell-completion-native-setup ()
  "Try to setup native completion, return non-nil on success."

which deep down relies on explicitly doing an accept-process-output, which sounds rather bad.

In any case, I'm enclined to think that python.el is doing something wrong and elpy doing something right, but I'd love to get a bit of help pinpointing the bad interaction before opening an Emacs ticket, if we determine this to be the right course of action; especially since Emacs 25 is in a pretty deep freeze, so there's little chance that this will get fixed on the Emacs side before the release.

Let me know if I can help :)

[jorgenschaefer]

Hello, and thanks for the report. What's the value of python-shell-interpreter with and without Elpy enabled? Are you using a virtualenv in either case?

[cpitclaudel]

No virtualenv in any case. The bug happens regardless of the value I give to python-shell-interpreter in my .emacs (python3 or ipython3). Enabling or disabling elpy doesn't seem to change that value.

[jorgenschaefer]

Could you copy the definition of the function python-shell-completion-native-setup into your *scratch* buffer and edit it so that the part before pytho.el: native completion setup failed reads like this:

        print ('python.el: native completion setup loaded')                     
    except:                                                                     
        raise
        print ('python.el: native completion setup failed')                     

i.e. add a raise to the except branch so we can see the error, hopefully. Evaluate the resulting function and try C-c C-c in Elpy again.

[cpitclaudel]

Sorry, I don't think I explained this clearly. There is no error; the python repl buffer contains this after the warning is shown:

Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
Type "copyright", "credits" or "license" for more information.

IPython 1.2.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 
In [1]: python.el: native completion setup loaded

The reason the warning is shown seems to be that the accept-process-output call that python.el does times out without returning data; strangely enough, though, it doesn't happen reliably...

[jorgenschaefer]

There's nothing Elpy does here that should interfere with that code. I have no idea what's going on there. :-(

[cpitclaudel]

I'll close this then, and I'll try to investigate it myself next time it pops up :)

[rthoma]

I receive the same warning as cpitclaudel running Emacs 25.1.50 (9.0) built with homebrew on my mac.

[ghost]

I also have been getting this same warning, I ended up just setting python-shell-completion-native to nil, as I just couldn't get it to work.

I do have working tab-completion in python shell buffers with that setting with ipython as my interpreter though.

[cpitclaudel]

Looks like this is happening again on master :/

[cpitclaudel]

Interestingly, even setting elpy-modules to nil doesn't suppress the problem. What does elpy do that's not in a module?

[jorgenschaefer]

Interestingly, even setting elpy-modules to nil doesn't suppress the problem. What does elpy do that's not in a module?

Not much. This is a problem with python-mode in Emacs, not Elpy specifically.

I suspect this is yet another problem with the newer ipython stuff. As a wild shot, could you add (setenv "IPY_TEST_SIMPLE_PROMPT" "1") and/or (setenv "JUPYTER_CONSOLE_TEST" "1") to your config file and see if that fixes this problem?

[cpitclaudel]

Hmm. I'm not using IPython (I use the regular CPython); besides, everything works fine in both emacs -Q and in my regular Emacs, as long as I don't load Elpy.

[jorgenschaefer]

But the output you showed above was from ipython?

[cpitclaudel]

Indeed; that's what I guess I was testing out at the time. This happens with 2.7 and 3.5 as well.

[jorgenschaefer]

Could you copy the definition of the function python-shell-completion-native-setup into your *scratch* buffer and edit it so that the part before pytho.el: native completion setup failed reads like this:

        print ('python.el: native completion setup loaded')                     
    except:                                                                     
        raise
        print ('python.el: native completion setup failed')                     

i.e. add a raise to the except branch so we can see the error, hopefully. Evaluate the resulting function and try C-c C-c in Elpy again.

[cpitclaudel]

@jorgenschaefer I'm confused. Didn't you ask me to do this before? :)

I made the requested change, but there is no error to see:

Python 3.5.2 (default, Jul  5 2016, 12:43:10) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded
>>> 

The except path is never taken.

[jorgenschaefer]

Yes I did, but that was with ipython which might have had different code paths :-) When trying to debug some weird behavior, it's confusing when the environment changes randomly.

What does M-: (python-shell-completion-native-try) return for you?

I'm still confused why anything that Elpy does would affect this. If you add (setq elpy-modules nil) before elpy-enable, does this still happen?

[cpitclaudel]

I'll try this. I'll also try to debug this on my own a bit, because this is getting complex and annoying, and I don't think I'm being very helpful. Let me create a minimal test case, and see if that helps me; otherwise, I'll at least have stuff to report here :)

[ghost]

I too am facing same problem as @cpitclaudel.
When I do M-: (python-shell-completion-native-try), I get error

Debugger entered--Lisp error: (wrong-type-argument processp nil)
  process-buffer(nil)
  python-shell-completion-native-get-completions(nil nil "")
  python-shell-completion-native-try()
  eval((python-shell-completion-native-try) nil)
  eval-expression((python-shell-completion-native-try) nil)
  funcall-interactively(eval-expression (python-shell-completion-native-try) nil)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

Even with (setq elpy-modules nil) I still get the "Native completions have been disabled locally" warning and completions do not work.
Seems to be due to: bug#22897

[ghost]

I have a question. How can I elpy-shell-send-region-or-buffer to an external Python interpreter process (not running in Emacs), but attached to a terminal emulator?

[cpitclaudel]

Closing this: after a while I ended up opening an Emacs bug report, and was pointed to another, existing one. See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24401

Sorry for the confusion, and thanks for the help and the hard work on Elpy!

[jorgenschaefer]

Thank you for finding that upstream bug! That solves one of my headaches here :-D

[yichao-cai]

Hi @npostavs,

There is only a line in the *Python* which says about the setup (which I posted in my last comments):

>>> python.el: native completion setup loaded

BTW, where can I find the python code for testing that you mentioned?

Thank you so much!

[npostavs]

EDIT:
A surprising thing is that my issue has gone, after two days and I don't know why. (No restart of my computer. I've been using emacs24 instead of emacs25 these days. When I tried emacs25 today it just gone.)

Oh, huh. I guess it's all good then.

BTW, where can I find the python code for testing that you mentioned?

It's in python.el, <f1> f python-shell-completion-native-setup RET and then hit the python.el link in the *Help* buffer will bring you to the right place.

[fulminemizzega]

I'm sorry to keep this going, I'm running Fedora 27, emacs 25.3.1, python 3.6.4, elpy 20180318.856 but I still see this issue. I have set python-shell-interpreter to "python3".
I tried this: run emacs -Q, then M-x set-variable python-shell-interpreter "python3", then open a python script, C-p C-c and C-c C-c does not give any warning.
If I run just emacs, then C-c C-c gives me the warning message.

[npostavs]

If I run just emacs, then C-c C-c gives me the warning message.

What's the value of python-shell-interpreter in this case? Could you bisect your .emacs to figure out which part of your configuration causes this?

[cpitclaudel]

I have the same problem, but only with C-c C-c (that is, C-c C-p followed by C-c C-c works, but C-c C-c alone gives the error message). IOW, the problem happens when Python is started by the same function that sends the contents of the buffer (elpy-shell-send-region-or-buffer).

[npostavs]

If I try to run C-c C-c first from emacs -Q, then I get

user-error: Start a Python process first with ‘M-x run-python’ or ‘C-c C-p’.

Does elpy suppress/circumvent this?

[cpitclaudel]

Does elpy suppress/circumvent this?

It does, yes. I think that's the source of the issue.
All if this is confusing, of course, because originally this was a legit Emacs issue, and now it seems to be mostly elpy doing something wrong.

(Btw, (progn (run-python (python-shell-calculate-command) nil t) (python-shell-send-buffer nil t)) seems to work just fine. Question: why doesn't C-c C-c in plain Emacs just call run-python if needed?)

[npostavs]

Hmm, the waiting loop in elpy-shell--ensure-shell-running looks pretty dubious, not sure if Emacs should adopt something like that.

https://github.com/jorgenschaefer/elpy/blob/1.18.0/elpy-shell.el#L257

[cpitclaudel]

not sure if Emacs should adopt something like that.

I don't think so, no. And it does look like the issue is related to that loop, at least on my machine.

[craigmac]

Still getting the 'readline' warning on Windows 7, using Emacs 25.3_1 release, which from what I read people were saying would fix this issue. I also tried starting with no init file (emacs -Q) to make sure it wasn't my .emacs.

[npostavs]

@saltycraig there is no fix on Windows, just (setq python-shell-completion-native-disabled-interpreters '("python")) to suppress the warning (for Emacs 26, the warning is already suppressed on Windows).

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28580
https://emacs.stackexchange.com/questions/35306/python-shell-interpreter-doesn-t-seem-to-support-readline

[craigmac]

Thank you @npostavs , this is what I ended up doing and just guarded it with a 'windows-nt' system-type detect, hopefully it really is suppressed on Emacs 26, as I did read several answers on threads that this was supposed to have been fixed at or around 25.1.

[bgoodr]

I'm encountered this problem when running a conda-based python 2.7.14
when using the C-c C-c binding that is normally bound to
elpy-shell-send-region-or-buffer. I did some extensive debugging and
came up with a hackaround as below:

(defun bg-python-mode-hook ()
  ...
  (define-key elpy-mode-map (kbd "C-c C-c") 'bg-elpy-shell-send-region-or-buffer)
  ...)

(add-hook 'python-mode-hook 'bg-python-mode-hook)

(defun bg-elpy-shell-send-region-or-buffer (&optional arg)
  (interactive "P")
  ;; Force the process to start completely by sitting a bit to avoid this warning:
  ;;
  ;;   Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native-enable’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally.
  ;;
  ;; Refer to https://github.com/jorgenschaefer/elpy/issues/887
  ;;
  (elpy-shell-get-or-create-process 0.001)
  (elpy-shell-send-region-or-buffer arg))

What I have not been able to determine is why (run-python "python -i")
works but the elpy-shell-send-region-or-buffer function does not. I
speculate that something that elpy is doing behind the scenes is
slowing down python startup to the point that the underlying
python-shell-completion-native-setup function can't see it.

         (save-excursion
           (re-search-backward
            (regexp-quote "python.el: native completion setup loaded") nil t 1))

returns nil, but if you look in the Python process buffer, that
"setup loaded" line shows up. So something is not waiting long enough.

The call to (elpy-shell-get-or-create-process 0.001) above waits just
long enough for the python to invoke. The value of 0.001 may not be
large enough depending upon the latency of the filesystem serving
python (e.g., slow or overloaded NFS), but at least this hack allows
me to increase it slightly.

Side note: Not even sure this is relevant: I discovered I had package
called "color-moccur.el" imported and it defined a global variable
called simply "regexp" that also fouled up the logic inside
python-shell-accept-process-output where by this expression:

(or regexp comint-prompt-regexp)

evaluated to nil even though comint-prompt-regexp is non-nil. This
occurs because the global variable regexp is used even though the
default value is not specified in the particular call to
python-shell-accept-process-output. I've since disabled importing
color-moccur.el since no package should be defining variables simply
named as "regexp".

M-x emacs-version returns:

GNU Emacs 25.2.2 (x86_64-pc-linux-gnu, GTK+ Version 3.22.21) of 2017-09-22, modified by Debian

lsb_release -r -i returns:

Distributor ID:	Ubuntu
Release:	17.10

M-x elpy-config returns:

Elpy Configuration

Virtualenv........: None
RPC Python........: 2.7.14 (/home/brentg/conda/Ubuntu.17.10.miniconda2/envs/env1/bin/python)
Interactive Python: python (/home/brentg/conda/Ubuntu.17.10.miniconda2/envs/env1/bin/python)
Emacs.............: 25.2.2
Elpy..............: 1.19.0
Jedi..............: 0.12.0
Rope..............: Not found
Autopep8..........: 1.3.5
Yapf..............: Not found
Syntax checker....: flake8 (/home/brentg/conda/Ubuntu.17.10.miniconda2/envs/env1/bin/flake8)

[raoulhatterer]

I solved the problem with brew doctor indications. A lot of brew link where missing.

[sungwonida]

@jojojames Thanks! You saved my days. Python 3.7 installed by brew works perfectly.

[emanjavacas]

I was having a similar error with no autocompletion and it was solved by downgrading ipython from 7.2.0 to 6.2.1. Just in case it helps anybody I am writing this here.

[avatar-lavventura]

I also have the same error. Given log, when I apply run-python:

-UU-:---F2  doo.py  All (11,0)  :~/hack/Q1/  (User-MacBook.local) -------------|-UUU:**-F2  *Python*  All (7,4)  :~/hack/Q1/  (Alimoglu-MacBook.local) ------------
Python 3.7.3 (default, Apr 24 2020, 18:51:23)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import codecs, os;__pyfile = codecs.open('''/var/folders/mx/m3c2fd193h169gg6nd5x3p480000gn/T/pyyNbUtw''', encoding='''utf-8''');__code =
__pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/mx/m3c2fd193h169gg6nd5x3p480000gn/T/pyyNbUtw''');exec(compile(__code, '''/var/folders/mx/m3c2fd193h169gg6nd5x3p480000gn/T/pyyNbUtw''', 'exec'));
python.el: native completion setup failed, <class 'Exception'>: libedit based readline is known not to work,
      see etc/PROBLEMS under "In Inferior Python mode, input is echoed".
>>>

[maksadbek]

This error everytime gets longer and longer:

Python 3.9.1 (default, Jan  6 2021, 06:04:49) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import codecs, os;__pyfile = codecs.open('''/var/folders/51/77zp4bqd4_53zn95hng9pt3h0000gn/T/pyoyHfVB''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/51/77zp4bqd4_53zn95hng9pt3h0000gn/T/pyoyHfVB''');exec(compile(__code, '''/Users/max/code/binsearch.py''', 'exec'));;
>>> -1 15 7
>>> 7 15 11
>>> 11 15 13
>>> 13 15 14
15

In [2]: iimmppoorrtt  ccooddeeccss,,  ooss;;____ppyyffiillee  ==  ccooddeeccss..ooppeenn((''''''//vvaarr//ffoollddeerrss//5511//7777zzpp44bbqqdd44__5533zznn9955hhnngg99pptt33hh00000000ggnn//TT//ppyyccBBiibbhhCC'''''',,  eennccooddiinngg==''''''uuttff--88''''''));;____ccooddee  ==  ____ppyyffiillee..rreeaadd(())..eennccooddee((''''''uuttff--88''''''));;____ppyyffiillee..cclloossee(());;ooss..rreemmoovvee((''''''//vvaarr//ffoollddeerrss//5511//7777zzpp44bbqqdd44__5533zznn9955hhnngg99pptt33hh00000000ggnn//TT//ppyyccBBiibbhhCC''''''));;eexxeecc((ccoommppiillee((____ccooddee,,  ''''''//UUsseerrss//mmaaxx//ccooddee//bbiinnsseeaarrcchh..ppyy'''''',,  ''eexxeecc''))));;;;
>>> -1 15 7
>>> 7 15 11
>>> 11 15 13
>>> 13 15 14
15

In [3]: iiimmmpppooorrrttt   cccooodddeeecccsss,,,   ooosss;;;______pppyyyfffiiillleee   ===   cccooodddeeecccsss...ooopppeeennn((('''''''''///vvvaaarrr///fffooollldddeeerrrsss///555111///777777zzzppp444bbbqqqddd444___555333zzznnn999555hhhnnnggg999pppttt333hhh000000000000gggnnn///TTT///pppyyyPPPFFFPPPsssYYYuuu''''''''',,,   eeennncccooodddiiinnnggg==='''''''''uuutttfff---888''''''''')));;;______cccooodddeee   ===   ______pppyyyfffiiillleee...rrreeeaaaddd((()))...eeennncccooodddeee((('''''''''uuutttfff---888''''''''')));;;______pppyyyfffiiillleee...ccclllooossseee((()));;;ooosss...rrreeemmmooovvveee((('''''''''///vvvaaarrr///fffooollldddeeerrrsss///555111///777777zzzppp444bbbqqqddd444___555333zzznnn999555hhhnnnggg999pppttt333hhh000000000000gggnnn///TTT///pppyyyPPPFFFPPPsssYYYuuu''''''''')));;;eeexxxeeeccc(((cccooommmpppiiillleee(((______cccooodddeee,,,   '''''''''///UUUssseeerrrsss///mmmaaaxxx///cccooodddeee///bbbiiinnnssseeeaaarrrccchhh...pppyyy''''''''',,,   '''eeexxxeeeccc'''))))));;;;;;
>>> -1 15 7
>>> 7 15 11
>>> 11 15 13
>>> 13 15 14
15

In [4]: iiiimmmmppppoooorrrrtttt    ccccooooddddeeeeccccssss,,,,    oooossss;;;;________ppppyyyyffffiiiilllleeee    ====    ccccooooddddeeeeccccssss....ooooppppeeeennnn((((''''''''''''////vvvvaaaarrrr////ffffoooollllddddeeeerrrrssss////55551111////77777777zzzzpppp4444bbbbqqqqdddd4444____55553333zzzznnnn99995555hhhhnnnngggg9999pppptttt3333hhhh0000000000000000ggggnnnn////TTTT////ppppyyyynnnnJJJJppppmmmmUUUUbbbb'''''''''''',,,,    eeeennnnccccooooddddiiiinnnngggg====''''''''''''uuuuttttffff----8888''''''''''''))));;;;________ccccooooddddeeee    ====    ________ppppyyyyffffiiiilllleeee....rrrreeeeaaaadddd(((())))....eeeennnnccccooooddddeeee((((''''''''''''uuuuttttffff----8888''''''''''''))));;;;________ppppyyyyffffiiiilllleeee....cccclllloooosssseeee(((())));;;;oooossss....rrrreeeemmmmoooovvvveeee((((''''''''''''////vvvvaaaarrrr////ffffoooollllddddeeeerrrrssss////55551111////77777777zzzzpppp4444bbbbqqqqdddd4444____55553333zzzznnnn99995555hhhhnnnngggg9999pppptttt3333hhhh0000000000000000ggggnnnn////TTTT////ppppyyyynnnnJJJJppppmmmmUUUUbbbb''''''''''''))));;;;eeeexxxxeeeecccc((((ccccoooommmmppppiiiilllleeee((((________ccccooooddddeeee,,,,    ''''''''''''////UUUUsssseeeerrrrssss////mmmmaaaaxxxx////ccccooooddddeeee////bbbbiiiinnnnsssseeeeaaaarrrrcccchhhh....ppppyyyy'''''''''''',,,,    ''''eeeexxxxeeeecccc''''))))))));;;;;;;;
>>> -1 15 7
>>> 7 15 11
>>> 11 15 13
>>> 13 15 14
15

In [5]: ```

[kourosh2]

Hi,
I am also experiencing same Warning. Below is the output of elpy-config,

Elpy Configuration

Emacs.............: 27.1
Elpy..............: 1.35.0
Virtualenv........: main_env (/Users/kourosh/.virtualenvs/main_env)
Interactive Python: python 3.9.2 (/Users/kourosh/.virtualenvs/main_env/bin/python)
RPC virtualenv....: main_env (/Users/kourosh/.virtualenvs/main_env)
 Python...........: python 3.9.2 (/Users/kourosh/.virtualenvs/main_env/bin/python)
 Jedi.............: 0.18.0
 Rope.............: Not found (0.18.0 available)
 Autopep8.........: 1.5.6
 Yapf.............: 0.31.0
 Black............: 20.8b1
Syntax checker....: flake8 (/Users/kourosh/.virtualenvs/main_env/bin/flake8)

My OS is macOS BIg Sur (Version 11.1).
I have noticed that when I start Python shell using M-x run-python, I don't get the warning but when I start it with C-c C-c I get the warning.
I am also able to import readline in python shell,

>>> import readline
>>> print(readline.__doc__)
Importing this module enables command line editing using GNU readline.
>>> 

Any help is much appreciated,
Thanks,
Kourosh

[galaunay]

Looks like something problem I already saw on macos.
Python.el is displaying those lines (not elpy), Elpy is supposed to filter it our to avoid the inconvenience.

I would try to address this error message:

python.el: native completion setup failed, <class 'Exception'>: libedit based readline is known not to work,
      see etc/PROBLEMS under "In Inferior Python mode, input is echoed".

and eventually report to python.el

[Evangeline-T]

I have the same warning with elpy 1.35.0 in GNU Emacs 28.2 on MacOS 12.6.7.
I find some discussion on this situation.

The warning occurs because I set python-shell-interpreter to 'ipython', which do not support readline. While python-shell-completion-native-enable was t. This variable was introduced in version 25.1 of Emacs. You would be fine if you use version 24.

So, I add (add-to-list 'python-shell-completion-native-disabled-interpreters "ipython") to init.el to suppress the warning.

Hope this helps.