Unable to execute code using Jupyter as interpreter

[mjappels]

Hello!

I am having issues getting my elpy configuration to work smoothly, and I was hoping somebody here could help.

For starters, I am running GNU emacs 26.1 on MacOS 10.13.5 (High Sierra), installed via homebrew. I am running a python 3.6 installation from the anaconda distribution.

My problem is that with jupyter set up as the interpreter, I seem to be unable to run any code. The issue exhibits itself in two ways:

1. I enter the shell using C-c C-z, then type a simple line of code next to the In [1]: prompt, hit return and nothing happens.
2. If I run some code using C-c C-c, emacs will either open the console and consequently do nothing, or, if the console was already running, emacs simply freezes up and I need to terminate emacs.

If I remove the section that sets jupyter as my interpreter, the regular python interpreter executes the code as expected.

So, any suggestions are welcome.

Result of (elpy-config)

Elpy Configuration

Virtualenv........: None
RPC Python........: 3.6.5 (/Users/mike/anaconda3/bin/python)
Interactive Python: jupyter (/Users/mike/anaconda3/bin/jupyter)
Emacs.............: 26.1
Elpy..............: 1.22.0
Jedi..............: 0.12.0 (0.12.1 available)
Rope..............: 0.10.7
Autopep8..........: 1.3.4 (1.3.5 available)
Yapf..............: 0.22.0
Black.............: Not found (18.6b4 available)
Syntax checker....: flake8 (/Users/mike/anaconda3/bin/flake8)

You have not activated a virtual env. While Elpy supports this, it is
often a good idea to work inside a virtual env. You can use M-x
pyvenv-activate or M-x pyvenv-workon to activate a virtual env.

The directory ~/.local/bin/ is not in your PATH. As there is no active
virtualenv, installing Python packages locally will place executables
in that directory, so Emacs won't find them. If you are missing some
commands, do add this directory to your PATH -- and then do
`elpy-rpc-restart'.

Elpy configuration in my init.el

;; init.el --- Emacs configuration

;; INSTALL PACKAGES
;; --------------------------------------

(require 'package)
(add-to-list 'package-archives
             '("melpa-stable" . "https://stable.melpa.org/packages/"))

(package-initialize)
(when (not package-archive-contents)
  (package-refresh-contents))

(defvar myPackages
  '(elpy))

(mapc #'(lambda (package)
    (unless (package-installed-p package)
      (package-install package)))
      myPackages)

;; BASIC CUSTOMISATION
;; --------------------------------------

(setq inhibit-startup-message t) ;; hide the startup message
(global-linum-mode t) ;; enable line numbers globally


;; ELPY CUSTOMISATION
;; --------------------------------------

(elpy-enable)

(setq python-shell-interpreter "jupyter"
      python-shell-interpreter-args "console --simple-prompt"
      python-shell-prompt-detect-failure-warning nil)
(add-to-list 'python-shell-completion-native-disabled-interpreters
             "jupyter")


(setq elpy-shell-echo-input nil)

[galaunay]

Hi,
I tried your configuration, but I couldn't reproduce the issue (Archlinux, python3.6, emacs26.1).

Just to check if the problem comes from elpy or python.el: you can deactivate elpy (M-x elpy-mode) and try to open a shell buffer with python.el (M-x run-python).

If it is still not working, my best guess would be python.el having trouble parsing jupyter output. What kind of output do you have when running jupyter console --simple-prompt from the terminal ?

[mjappels]

Hi,

Thanks for responding. Ive just loaded emacs purely with the config file as specified above. Confirmed that the problem is still occurring with elpy. After disabling elpy as you suggested, I opened a shell which loaded the jupyter console and it is now working, seemingly without issue. I can run code using C-c C-c without running into the problem I mentioned earlier. Seems strange?

Is there any further information that I can provide?

[galaunay]

Ok,
So this seems to be indeed a problem with Elpy.
When you deactivate Elpy, Emacs falls back to python.el mode.
Elpy is build on top of this mode, that gives you basic interactions with the python shell.

What you can try to do is enabling debug (M-x toggle-debug-on-quit), reproduce the bug, and hit C-g when Emacs is hanging. It should give you a traceback with indications on where Emacs got stuck.
If you can post that traceback here, we may be able to understand what happen.

[mjappels]

Ok, so I've gone through and enabled toggle-debug-on-quit and tried two different things:

1. I first tried running a basic python script (a la "hello world"), which got sent to the shell but never got executed. I hit C-g which didn't produce anything, I suspect because, as I described above, emacs doesn't hang at that point.

2. with the process still running (and seemingly stuck on the first execute command), I hit run again, triggering the hang. I hit C-g as you suggested and this is the output in the backtrace:

Debugger entered--Lisp error: (quit)
  process-send-string(#<process Python> "import codecs, os, ast;__pyfile = codecs.open('''/var/folders/gw/b1tyw32d2xv5yj9h8j24s4p40000gn/T/pyVIYF9P''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/gw/b1tyw32d2xv5yj9h8j24s4p40000gn/T/pyVIYF9P''');__block = ast.parse(__code, '''/path/to/hello.py''', mode='exec');__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);__block.body.pop() if __isexpr else None;exec(compile(__block, '''/path/to/hello.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''/path/to/hello.py''', mode='eval')) if __isexpr else None")
  comint-send-string(#<process Python> "import codecs, os, ast;__pyfile = codecs.open('''/var/folders/gw/b1tyw32d2xv5yj9h8j24s4p40000gn/T/pyVIYF9P''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/gw/b1tyw32d2xv5yj9h8j24s4p40000gn/T/pyVIYF9P''');__block = ast.parse(__code, '''/path/to/hello.py''', mode='exec');__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);__block.body.pop() if __isexpr else None;exec(compile(__block, '''/path/to/hello.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''/path/to/hello.py''', mode='eval')) if __isexpr else None")
  python-shell-send-string("import codecs, os, ast;__pyfile = codecs.open('''/var/folders/gw/b1tyw32d2xv5yj9h8j24s4p40000gn/T/pyVIYF9P''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/gw/b1tyw32d2xv5yj9h8j24s4p40000gn/T/pyVIYF9P''');__block = ast.parse(__code, '''/path/to/hello.py''', mode='exec');__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);__block.body.pop() if __isexpr else None;exec(compile(__block, '''/path/to/hello.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''/path/to/hello.py''', mode='eval')) if __isexpr else None" #<process Python>)
  python-shell-send-file("/path/to/hello.py" #<process Python> "/var/folders/gw/b1tyw32d2xv5yj9h8j24s4p40000gn/T/pyVIYF9P" t)
  python-shell-send-string("print('Hello World')\nfor i in range(10):\n    print(i)\n" #<process Python>)
  python-shell-send-region(1 55 nil nil)
  python-shell-send-buffer(nil)
  elpy-shell--send-region-or-buffer-internal(nil)
  elpy-shell-send-region-or-buffer-and-step(nil)
  funcall-interactively(elpy-shell-send-region-or-buffer-and-step nil)
  call-interactively(elpy-shell-send-region-or-buffer-and-step)
  elpy-shell--send-with-step-go(elpy-shell-send-region-or-buffer-and-step nil nil nil)
  elpy-shell-send-region-or-buffer(nil)
  funcall-interactively(elpy-shell-send-region-or-buffer nil)
  call-interactively(elpy-shell-send-region-or-buffer nil nil)
  command-execute(elpy-shell-send-region-or-buffer)

Any clues?
Thanks for your help.

[galaunay]

If Emacs doesn't hang or throw any error during the first step (your point 1), I would think the code is correctly send to the shell, but there is something unexpected in the shell output that make Emacs parsing fail.
This is a very low-level problem and I can't see how elpy could be interfering with that...

I still have some ideas of things you can try to get us some more information:

• You can try opening your basic python script (with elpy-mode enabled), open a shell using M-x run-python and send the buffer to it with M-x python-shell-send-buffer.
• You can try re-opening the same basic script, executing M-: (remove-hook 'inferior-python-mode-hook 'elpy-shell--enable-output-filter), trying C-c C-c and see if it resolves the issue.
• You can try deactivating all elpy modules with M-: (setq elpy-modules '()) (as one of them could be at the origin of the problem), and try C-c C-c.

[mjappels]

Ok so having re-confirmed the problem persists, the results of your suggestions are the following (restarting emacs from fresh each time):

1. Running M-x run-python then python-shell-send-buffer works. In fact, once I am in the shell opened using run-python, I can C-c C-c the code and it executes successfully. The one oddity I have noticed is that any output produced appears without the Out [xx]: prompt. If I use the interpreter directly to print some output, it does. Not sure if this is expected behaviour, but thought it might be worth mentioning.

2. After executing the lisp code above, the minibuffer spits out the following message:

(#[0 "ÁÀ!�ÂÃ�\"���" [company-transformers make-local-variable remove company-sort-by-occurrence] 3])

Subsequently running C-c C-c opens the console, but fails to execute the code, and the console hangs.

3. Same result as 2, the message after deactivating the modules is nil, and attempting to run the code also hangs the console.

Does any of that make sense to you?

[galaunay]

The fact that everything works fine when you are using run-python instead of elpy-get-or-create-process seems to indicate that the problem comes from this function. At least it gives indication on where to look.

Could you try running this function from a freshly opened python script, and paste the result:

(defun elpy-debug ()
  (interactive)
  (let* ((bufname (format "*%s*" (python-shell-get-process-name nil)))
         (proc (get-buffer-process bufname))
         (default-directory (or (and elpy-shell-use-project-root
                                     (elpy-project-root))
                                default-directory)))
    (message "==================")
    (message "=== Elpy debug ===")
    (message "==================")
    (message "bufname: %s" bufname)
    (message "proc: %s" proc)
    (message "elpy-shell-use-project-root: %s" elpy-shell-use-project-root)
    (message "(elpy-project-root): %s" (elpy-project-root))
    (message "default-directory: %s" default-directory)
    (message "(python-shell-parse-command): %s" (python-shell-parse-command))
    ))

As a temporary workaround, putting the following code in your init.el (after loading elpy) may make elpy usable again:

(defun elpy-shell-get-or-create-process (&optional sit)
  "Get or create an inferior Python process for current buffer and return it.

If SIT is non-nil, sit for that many seconds after creating a
Python process. This allows the process to start up."
  (let* ((bufname (format "*%s*" (python-shell-get-process-name nil)))
         (proc (get-buffer-process bufname)))
    (if proc
        proc
      (run-python (python-shell-parse-command) nil t)
      (when sit (sit-for sit))
      (get-buffer-process bufname))))

[Gijs-Koot]

I also had this issue, and it was solved after including the defun elpy-shell-get-or-create-process as above.

[galaunay]

Thanks for the feedback.

I managed to reproduce this issue (from time to time).
For me it was linked to prompt detection, so setting (setq python-shell-prompt-detect-enabled nil) fixed it.

@Gijs-Koot, could you tell me if it solve your issue as well ? Maybe I could add a fix in Elpy so it wont bother anyone else.

[Gijs-Koot]

I've tried this by putting the (setq .. line in my init.el, and removing the function above, and that actually doesn't seem to solve the problem for me. I have no messages in the *Messages* buffer. I've reread the original issue, and it is exactly what I experience. Like @galaunay however, it's not always there, I have had previous environments where I didn't need this workaround.

[galaunay]

hum...
I am failing to reproduce it now.

It could be that we are sending code to the shell too quickly after its creation.
If it is the case, this updated implementation of elpy-shell-get-or-create-process should work:

(defun elpy-shell-get-or-create-process (&optional sit)
  "Get or create an inferior Python process for current buffer and return it.

If SIT is non-nil, sit for that many seconds after creating a
Python process. This allows the process to start up."
  (let* ((bufname (format "*%s*" (python-shell-get-process-name nil)))
         (proc (get-buffer-process bufname)))
    (if proc
        proc
      (unless (executable-find python-shell-interpreter)
        (error "Python shell interpreter `%s' cannot be found. Please set `python-shell-interpreter' to a valid python binary."
               python-shell-interpreter))
      (let ((default-directory
            Does it ?  (cond ((eq elpy-shell-starting-directory 'project-root)
                     (or (elpy-project-root)
                         default-directory))
                    ((eq elpy-shell-starting-directory 'current-directory)
                     default-directory)
                    ((stringp elpy-shell-starting-directory)
                     (file-name-as-directory
                      (expand-file-name elpy-shell-starting-directory)))
                    (t
                     (error "Wrong value for `elpy-shell-starting-directory', please check this variable documentation and set it to a proper value")))))
        (run-python (python-shell-parse-command) nil t))
      (when sit (sit-for sit))
      (when (elpy-project-root)
        (with-current-buffer bufname
          (add-hook 'python-shell-first-prompt-hook
                    (lambda ()
                      (python-shell-send-string-no-output
                       (format "import sys;sys.path.append('%s');del sys"
                               (elpy-project-root))))
                    nil t)))
      (get-buffer-process bufname))))

If it does for you I'll update the code.
Thanks for your help !

[Gijs-Koot]

Sorry for not getting back to this earlier, your message got buried. I looked at the referenced issue #1712 and it seems you have figured out and fixed this situation. Thank you for this work!

[galaunay]

I suppose I can close this then.

I suspected this to be at the origin of other problems, so I had to merge it quickly.
Let me know if the problem re-appears.