Skip to content
This repository

emacs freezes when tab is hit in ipython with latest python-mode #1015

Merged
merged 5 commits into from over 2 years ago

8 participants

Pete Aykroyd Fernando Perez deftpunk Jens H Nielsen Thomas Kluyver Andreas Röhler Isaac Pei Takafumi Arakaki
Pete Aykroyd

This relates to the issue I filed: #1010

python-mode has it's own completion function mapped to the tab key. When that mapping wins, the tab will cause that function to fire and it will hang emacs. I've used a defadvice to redirect every call to python-mode's completion function instead to ipython's.

Honestly, I'm not sure if this is the best way since python-mode seems to try to make all configuration buffer-local and this is more of a global solution. But to get ipython working with python-mode you current have to bypass that anyway.

Suggestions welcome.

(I've got @deftpunk's patch in here too because I needed that to get ipython working with the most recent python-mode. Sorry if that's messy.)

added some commits November 18, 2011
Pete Aykroyd merging deftpunk's patch for python-mode f54e9e3
Pete Aykroyd Ensures that ipython-complete is called.
python-mode defines a key-binding for the tab key for its own
completion function (py-shell-complete) if that one wins, then
py-shell-complete will hang emacs as it tries to communicate
with the python shell process which does not exist. This uses
def-advice to instead call ipython-complete when py-shell-complete
is called.
4a983ca
Pete Aykroyd Switches to use a better completion API and send the whole line.
* sends the whole line of code to get better results
* not updated for xemacs yet
65ce75f
Fernando Perez
Owner

@paykroyd, the changes look good, but I'm struggling to make it work easily and fully with python-mode-6.0.4. Bear with me, I'm pretty ignorant of elisp... :)

problem 1

I can get it to load ipython instead of python with C-c C-!, but only if I put

(setq py-shell-name "ipython")

before python-mode is loaded at all, by anything. It means it took quite a bit of trial and error moving this line around in my config files; ultimately I stuck it just at the very beginning of my file. This didn't use to be the case long ago, one could just

(require 'ipython)

and everything would work. It would be nice to understand what's going on here, so we can give users some guidance.

problem 2

All my attempts at configuring the arguments, to load with pylab mode active, failed. I put

(setq py-python-command-args '("--pylab" "--colors" "LightBG"))

everywhere in my emacs init files, from the start to the end, and no matter what I do, the value of that variable is reported as just -i. Furthermore, I see there's code in ipython.el to auto-detect the color scheme and set --colorsappropriately, but it doesn't seem to be working, because the variablepy-python-command-args` is also not being set automatically (I tested by not setting it myself, and nothing happened).

I temporarily disabled all other python-related files, and tested that the codepath for auto-detection of the background is being taken. In fact, py-python-command-args is (--colors=LightBG -i) as I verified with a print statement in the elisp. But strangely, once the emacs session starts, it's back to being only -i, so something is overwriting it later...

The only way I could make it work was by setting it interactively with M-x set-variable. I went as far as completely nuking (temporarily) my entire emacs configuration, leaving a bare .emacs file with nothing but this:

(add-to-list 'load-path "/home/fperez/.emacs.conf.d/lisp")
(add-to-list 'load-path "/home/fperez/.emacs.conf.d/lisp/python-mode")
(setq py-shell-name "ipython")
(require 'ipython)
(setq py-python-command-args '("--colors" "LightBG"))

and this did make ipython start, but still the colors didn't work right.

So we seem to have a problem where the order of loading of the files is getting things rewritten, no idea why.

In any case, if you can point me to a bare .emacs file with the minimum amount that makes it work, and I can reproduce it, I'd be thrilled to add these fixes. But today I've already spent ~4 hours fighting with emacs and getting nowhere... I hope we can find a solution for this, and really appreciate your help.

Fernando Perez
Owner

One more data point: with your fixes and the bare .emacs file above, %debug now works better, which is great! But it will only track files with %colors nocolor, it seems the ansi color escapes confuse the regexp matcher. I get

error in process filter: not: Symbol's value as variable is void: py-pydbtrack-stack-entry-regexp

and a bunch of naked ansi junk on the debug buffer:

> [0;32m/home/fperez/scratch/error.py[0m(45)[0;36mRampNum[0;34m()[0m
[0;32m     44 [0;31m    [0mstep[0m [0;34m=[0m [0;34m([0m[0mend[0m[0;34m-[0m[0mstart[0m[0;34m)[0m[0;34m/[0m[0;34m([0m[0msize[0m[0;34m-[0m[0;36m1[0m[0;34m-[0m[0mtmp[0m[0;34m)[0m[0;34m[0m[0m
[0m[0;32m---> 45 [0;31m    [0mresult[0m[0;34m[[0m[0;34m:[0m[0;34m][0m [0;34m=[0m [0marange[0m[0;34m([0m[0msize[0m[0;34m)[0m[0;34m*[0m[0mstep[0m [0;34m+[0m [0mstart[0m[0;34m[0m[0m
[0m[0;32m     46 [0;31m[0;34m[0m[0m
[0m
ipdb> 

If this can't be fixed that's OK, we can just advertise that to use %debug within emacs, people should simply temporarily turn coloring off.

Fernando Perez
Owner

One more: this small change should probably be applied:

diff --git a/docs/emacs/ipython.el b/docs/emacs/ipython.el
index 1b11e5f..f56a1fd 100644
--- a/docs/emacs/ipython.el
+++ b/docs/emacs/ipython.el
@@ -218,7 +218,7 @@ the second for a 'normal' command, and the third for a multiline command.")
           py-shell-input-prompt-2-regexp "^   [.][.][.]+: *" )
     ;; select a suitable color-scheme
     (unless (delq nil
-                  (mapcar (lambda (x) (eq (string-match "^--colors=*" x) 0))
+                  (mapcar (lambda (x) (eq (string-match "^--colors*" x) 0))
                           py-python-command-args))
       (push (format "--colors=%s"
                     (cond

b/c the = sign is now optional.

Pete Aykroyd

Good feedback. I had sort of forgotten that I had made some changes to my environment just to get it all working together. I'll try to address those problems.

I'm still testing the %debug stuff. I found that with %pdb on it would jump to the right file, but the location in that file was not correct. I'm still working on that.

Based on a note in ipython.el, I also modified the completion code to send the entire line as well the current item being completed and switched it to use get_ipython().complete(). That works nicely, it can now complete things import statements. I still need to make that change for the xemacs completion function.

Fernando Perez
Owner

Awesome! It looks like we're finally getting out of the rut we've been in with emacs support! Many thanks to you (and @defunkt) for pitching in, welcome to the team :)

added some commits November 21, 2011
Pete Aykroyd Gets ipython working with python-mode with little config.
* sets the default value for py-shell-name to the ipython-command
* fixes color scheme setting to set a default value
cbf9083
Pete Aykroyd Changes color config detection based on feed back from @fperez. 32a579d
Pete Aykroyd

@fperez, ok I think that I have the config problem fixed. I tested my changes with a vanilla install. One of the problems was that python-mode makes these configuration variables buffer-local. So setting them once isn't always effective.

Here is the complete .emacs configuration that I used to test:

(add-to-list 'load-path "~/.emacs.d/python-mode")
(add-to-list 'load-path "~/.emacs.d/ipython/docs/emacs")
(require 'ipython)

I think the color issues are sorted as well and I incorporated the change that you suggested for detecting whether a color option had been set.

There are still some issues to be sorted with pdb, I think, and with some making completion smoother, but maybe that's a different pull request?

Let me know if you have any issues.

Pete Aykroyd

Oh I forgot to mention, to test manually configuring the parameters to ipython, see the documentation change. Use setq-default instead of setq.

deftpunk

paykroyd: Given that python-mode makes some variables buffer-local, is it even necessary for ipython.el to do

(require 'python-mode) 

anymore? Is it possible to do

(setq-default somevar somevalue)

on all of those variables directly in ipython.el - this might help with python.el compatibility.

Pete Aykroyd

deftpunk: I think it still needs to require it because ipython.el modifies the py-mode-map as well as a few other variables defined in python-mode.el. I'm pretty green to elisp though, is there a better way?

Jens H Nielsen

Seems to work fine for me.

One bug: It used to be possible to complete from a buffer and not only the terminal inside emacs. When I do that now
I get a Buffer *.py has no process error

One other long standing bug that I noticed is that one cannot complete stuff that contains a - because ship-chars-backwards does not skip it. If one changes l.394 to this it works.
(beg (save-excursion (skip-chars-backward "a-z0-9A-Z_./\-" (point-at-bol))
The slash is to escape the - in the regex

Fernando Perez
Owner

Hi folks, I'll try to finish the merge for this one soon, but I wanted to point your attention to #22, which also deals with emacs-related issues. If you have a chance to look at that code and you find anything worth doing with it (presuming the author is willing to relicense it as BSD for ipython), that would be great. And if everything there has already been done, we can simply close that issue.

Fernando Perez fperez referenced this pull request from a commit November 24, 2011
Fernando Perez Add '-' to escaped characters for emacs completion.
After feedback from @jenshnielsen in #1015.
9eb1305
Fernando Perez fperez merged commit 32a579d into from November 24, 2011
Fernando Perez fperez closed this November 24, 2011
Fernando Perez
Owner

Guys, I went ahead and merged it because it's already a big improvement over the previous situation. Oddly, tab completion isn't working at all for me. But because everything else is already so much better, I'd rather merge it now. If you guys think you know what could be the problem, we can open another PR for a new fix. I tried with the absolutely minimal .emacs file as noted above by @paykroyd, and still completion doesn't work even for something like 'import'.

But in any case, thanks a lot for the work, things are already in much better shape than before!

Thomas Kluyver
Collaborator

Emacs users: can you have a look at the docs about ipython.el, and update anything that needs it? In particular, I hope the issue with Python 2.2 has been solved now. ;-)

The relevant section is here: https://github.com/ipython/ipython/blob/master/docs/source/config/editors.txt#L40

Pete Aykroyd

@fperez, just to clarify, you were trying to do an import in the ipython shell, right? Were there any messages in the Messages buffer after it failed? If you start to do an import and then type in M-x ipython-complete does it work?

Thanks.

Fernando Perez
Owner

@paykroyd: yes, this is trying to complete by typing 'imp', which should unambigously complete to 'import'.

I tried both ways, with and with M-x ipython-complete, and both cases led to the same message in the *Messages* buffer:

Can't find completion for "imp" based on line imp [2 times]

HTH

Pete Aykroyd

@fperez, that's odd. I was hoping for an error message. This is the line of code that runs in ipython:

print(';'.join(get_ipython().complete('%s', '%s')[1])) #PYTHON-MODE SILENT\n

Where the first string is the current word ("imp") and the next is the whole line of code. Does that API call make sense? Are you on emacs or xemacs? I just found a bug in the xemacs hanlding.

Fernando Perez
Owner
Fernando Perez fperez referenced this pull request from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Andreas Röhler

Hi all,
seems something to do still at the python-mode.el side.
Linked a bug-report to this

https://bugs.launchpad.net/python-mode/+bug/934022

Cheers,
Andreas

Andreas Röhler

Hi,

if these kind of things happen, please alert us at
http://launchpad.net/python-mode
Let's prevent the evil at it's source :)

Cheers,
Andreas

Isaac Pei

hello, is this matter resolved? the ipython.el is indeed outdated, I tried multiple times in the past, and gave up for a while. just tried it again today on 2012-12-11, it froze my emacs 24.2 on windows ... it's a pity, maybe I'll go with vim then...

Takafumi Arakaki

My impression is that ipython.el is not maintained. Both python.el and python-mode.el support IPython now. So I guess there is no need for ipython.el now. If you are using Emacs 24.2, you have the new python.el which supports ipython and ipdb.

Thomas Kluyver
Collaborator
Viktor Haag ViktorHaag referenced this pull request in eschulte/emacs24-starter-kit April 05, 2013
Closed

starter-kit-python fails to load in Emacs 24.3.1 #37

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 5 unique commits by 1 author.

Nov 18, 2011
Pete Aykroyd merging deftpunk's patch for python-mode f54e9e3
Nov 19, 2011
Pete Aykroyd Ensures that ipython-complete is called.
python-mode defines a key-binding for the tab key for its own
completion function (py-shell-complete) if that one wins, then
py-shell-complete will hang emacs as it tries to communicate
with the python shell process which does not exist. This uses
def-advice to instead call ipython-complete when py-shell-complete
is called.
4a983ca
Pete Aykroyd Switches to use a better completion API and send the whole line.
* sends the whole line of code to get better results
* not updated for xemacs yet
65ce75f
Nov 21, 2011
Pete Aykroyd Gets ipython working with python-mode with little config.
* sets the default value for py-shell-name to the ipython-command
* fixes color scheme setting to set a default value
cbf9083
Pete Aykroyd Changes color config detection based on feed back from @fperez. 32a579d
This page is out of date. Refresh to see the latest.

Showing 1 changed file with 41 additions and 26 deletions. Show diff stats Hide diff stats

  1. 67  docs/emacs/ipython.el
67  docs/emacs/ipython.el
@@ -17,7 +17,8 @@
17 17
 ;; can be used to convert bits of a ipython session into something that can be
18 18
 ;; used for doctests. To install, put this file somewhere in your emacs
19 19
 ;; `load-path' [1] and add the following line to your ~/.emacs file (the first
20  
-;; line only needed if the default (``"ipython"``) is wrong)::
  20
+;; line only needed if the default (``"ipython"``) is wrong or ipython is not 
  21
+;; in your `exec-path')::
21 22
 ;;
22 23
 ;;   (setq ipython-command "/SOME-PATH/ipython")
23 24
 ;;   (require 'ipython)
@@ -36,7 +37,7 @@
36 37
 ;; always in ``pylab`` mode with hardcoded light-background colors, you can
37 38
 ;; use::
38 39
 ;;
39  
-;; (setq py-python-command-args '("--pylab" "--colors=LightBG"))
  40
+;; (setq-default py-python-command-args '("--pylab" "--colors=LightBG"))
40 41
 ;;
41 42
 ;;
42 43
 ;; NOTE: This mode is currently somewhat alpha and although I hope that it
@@ -132,6 +133,12 @@
132 133
 (require 'shell)
133 134
 (require 'executable)
134 135
 (require 'ansi-color)
  136
+;; XXX load python-mode, so that we can screw around with its variables
  137
+;; this has the disadvantage that python-mode is loaded even if no
  138
+;; python-file is ever edited etc. but it means that `py-shell' works
  139
+;; without loading a python-file first. Obviously screwing around with
  140
+;; python-mode's variables like this is a mess, but well.
  141
+(require 'python-mode)
135 142
 
136 143
 (defcustom ipython-command "ipython"
137 144
   "*Shell command used to start ipython."
@@ -164,12 +171,8 @@ the second for a 'normal' command, and the third for a multiline command.")
164 171
 (if (not (executable-find ipython-command))
165 172
     (message (format "Can't find executable %s - ipython.el *NOT* activated!!!"
166 173
                      ipython-command))
167  
-    ;; XXX load python-mode, so that we can screw around with its variables
168  
-    ;; this has the disadvantage that python-mode is loaded even if no
169  
-    ;; python-file is ever edited etc. but it means that `py-shell' works
170  
-    ;; without loading a python-file first. Obviously screwing around with
171  
-    ;; python-mode's variables like this is a mess, but well.
172  
-    (require 'python-mode)
  174
+    ;; change the default value of py-shell-name to ipython
  175
+    (setq-default py-shell-name ipython-command)
173 176
     ;; turn on ansi colors for ipython and activate completion
174 177
     (defun ipython-shell-hook ()
175 178
       ;; the following is to synchronize dir-changes
@@ -218,21 +221,24 @@ the second for a 'normal' command, and the third for a multiline command.")
218 221
           py-shell-input-prompt-2-regexp "^   [.][.][.]+: *" )
219 222
     ;; select a suitable color-scheme
220 223
     (unless (delq nil
221  
-                  (mapcar (lambda (x) (eq (string-match "^--colors=*" x) 0))
  224
+                  (mapcar (lambda (x) (eq (string-match "^--colors*" x) 0))
222 225
                           py-python-command-args))
223  
-      (push (format "--colors=%s"
224  
-                    (cond
225  
-                     ((eq frame-background-mode 'dark)
226  
-                      "Linux")
227  
-                     ((eq frame-background-mode 'light)
228  
-                      "LightBG")
229  
-                     (t ; default (backg-mode isn't always set by XEmacs)
230  
-                      "LightBG")))
231  
-            py-python-command-args))
232  
-    (unless (equal ipython-backup-of-py-python-command py-python-command)
233  
-      (setq ipython-backup-of-py-python-command py-python-command))
234  
-    (setq py-python-command ipython-command))
235  
-
  226
+      (setq-default py-python-command-args
  227
+		    (cons (format "--colors=%s"
  228
+				  (cond
  229
+				   ((eq frame-background-mode 'dark)
  230
+				    "Linux")
  231
+				   ((eq frame-background-mode 'light)
  232
+				    "LightBG")
  233
+				   (t ; default (backg-mode isn't always set by XEmacs)
  234
+				    "LightBG")))
  235
+			  py-python-command-args)))
  236
+    (when (boundp 'py-python-command)
  237
+      (unless (equal ipython-backup-of-py-python-command py-python-command)
  238
+        (setq ipython-backup-of-py-python-command py-python-command)))
  239
+    (setq py-python-command ipython-command)
  240
+    (when (boundp 'py-shell-name)
  241
+      (setq py-shell-name ipython-command)))
236 242
 
237 243
 ;; MODIFY py-shell so that it loads the editing history
238 244
 (defadvice py-shell (around py-shell-with-history)
@@ -312,7 +318,7 @@ gets converted to:
312 318
         (replace-match "" t nil)))))
313 319
 
314 320
 (defvar ipython-completion-command-string
315  
-  "print(';'.join(get_ipython().Completer.all_completions('%s'))) #PYTHON-MODE SILENT\n"
  321
+  "print(';'.join(get_ipython().complete('%s', '%s')[1])) #PYTHON-MODE SILENT\n"
316 322
   "The string send to ipython to query for all possible completions")
317 323
 
318 324
 
@@ -388,6 +394,7 @@ in the current *Python* session."
388 394
            (beg (save-excursion (skip-chars-backward "a-z0-9A-Z_./" (point-at-bol))
389 395
                                 (point)))
390 396
            (end (point))
  397
+	   (line (buffer-substring-no-properties (point-at-bol) end))
391 398
            (pattern (buffer-substring-no-properties beg end))
392 399
            (completions nil)
393 400
            (completion-table nil)
@@ -399,7 +406,7 @@ in the current *Python* session."
399 406
                       (setq ugly-return (concat ugly-return string))
400 407
                       "")))))
401 408
       (process-send-string python-process
402  
-                            (format ipython-completion-command-string pattern))
  409
+                            (format ipython-completion-command-string pattern line))
403 410
       (accept-process-output python-process)
404 411
       (setq completions
405 412
             (split-string (substring ugly-return 0 (position ?\n ugly-return)) sep))
@@ -409,10 +416,10 @@ in the current *Python* session."
409 416
       (setq completion (try-completion pattern completion-table))
410 417
       (cond ((eq completion t))
411 418
             ((null completion)
412  
-             (message "Can't find completion for \"%s\"" pattern)
  419
+             (message "Can't find completion for \"%s\" based on line %s" pattern line)
413 420
              (ding))
414 421
             ((not (string= pattern completion))
415  
-             (delete-region beg end)
  422
+             (delete-region (- end (length pattern)) end)
416 423
              (insert completion))
417 424
             (t
418 425
              (message "Making completion list...")
@@ -421,6 +428,14 @@ in the current *Python* session."
421 428
              (message "Making completion list...%s" "done")))))
422 429
 )
423 430
 
  431
+;;; if python-mode's keybinding for the tab key wins then py-shell-complete is called
  432
+;;; instead of ipython-complete which result in hanging emacs since there is no shell
  433
+;;; process for python-mode to communicate with
  434
+(defadvice py-shell-complete
  435
+  (around avoid-py-shell-complete activate)
  436
+  (ipython-complete))
  437
+
  438
+
424 439
 ;;; autoindent support: patch sent in by Jin Liu <m.liu.jin@gmail.com>,
425 440
 ;;; originally written by doxgen@newsmth.net
426 441
 ;;; Minor modifications by fperez for xemacs compatibility.
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.