Mac OSX backend keyboard focus stays in terminal

[jiffyclub]

I'm using OS X Lion, Python 2.7.2 built by Homebrew with llvm-gcc, and matplotlib installed via pip install git+git://github.com/matplotlib/matplotlib.git. There are no errors or other alarms during installation.

When I make an interactive figure with the Mac OSX backend the keyboard focus does not go to the plot window, it stays in the terminal. Mouse focus does go to the window. Clicking on the save button brings up the save dialog and I can click on things there, but typing just sends text to the terminal, making it impossible to specify a file name.

This seems to be the same as an issue reported on the ipython repo last year: ipython/ipython#1188 (I am not using ipython, though.)

If I switch to the Tkagg backend things work fine.

[mdboom]

I'm not on a Mac so can't really test, but does #663 help?

[jiffyclub]

#663 does not change anything, but it also does not achieve its stated goal of raising the plot window so I'm not sure if this is being correctly tested.

[jiffyclub]

This issue is a little different if I am in interactive mode.

When not in interactive mode I get a plot via the blocking show() function and my typing shows up in the terminal and after I close the plot I get my Python session back.

If I am in interactive mode typing goes to the terminal even when the plot has been brought to the foreground. As soon as I click the save button typing stops going to the terminal and after I cancel the save and close the plot Python is completely unresponsive. I get this message after pressing control-C + control-D in the terminal (they don't do anything if not pressed in that order).

Exception KeyboardInterrupt in <module 'threading' from '/usr/local/Cellar/python/2.7.2/lib/python2.7/threading.pyc'> ignored

[dmcdougall]

I can't seem to reproduce this. If interactive mode is off, show brings up the plot and brings it to the foreground with focus. I think this is expected behaviour.

When interactive mode is on and I issue a plotting command, a figure pops up in the background automatically. A subsequent call to show does not even bring the plot to the foreground. Keyboard focus stays in the terminal with the plot in the background. My ipython session does not freeze.

@jiffyclub Am I missing something?

[jiffyclub]

I can't reproduce this now either, but I don't have the same setup I did 9 months ago. Homebrew has recently switched their Python installers so that they build as a Framework. I understand that having a Framework build is important for some graphics libraries and when I reported this issue I didn't have a Framework build. Now I do, and that may be making a difference.

[asmeurer]

FWIW, I can reproduce this using both the system python and Fink python. It also doesn't matter if I use Terminal or iTerm2. Furthermore, if I use mission control (or expose in older versions of Max OS X; this is a quite old bug), and choose the window, it stays in the back. I have to move the terminal out of the way and click on it to bring it to the front. Needless to say, this is quite annoying, as I like to run my terminal full screen.

[dmcdougall]

So, after some simple googling I found this.

It appears that there isn't a neat way to achieve a synergy with the the OS X window manager if python is not a framework installation.

Weird that you get this problem with the system installation of python, as I'd imagine that would be a framework install. How did you install matplotlib?

[asmeurer]

I don't remember. It was probably either a pip install or a setup.py install from git. The matplotlib.__version__ is 1.1.0.

So is there a workaround for Fink Python? Another backend maybe? Or maybe some way to use aquaterm, or something like that?

[jiffyclub]

@asmeurer, I found that switching to the TkAgg backend resolved the problems with keyboard focus and Python hanging. I'm not sure it will change anything to do with where in the stack windows appear, though.

[asmeurer]

TK runs in X11, which is not ideal. I found that qt works. It still by default opens behind the terminal, but I can at least select it with mission control. Of course, if you you could make it open in front of the terminal, that would be better.

I didn't test either extensively, so I don't know if some things might be broken.

[dmcdougall]

X11 is another beast, but that explains the difference in behaviour between the Tk/Mac backends for X11/Aqua.

I wonder if Qt is making private OS X API calls to achieve its behaviour. That would be incredibly silly, but it might be the case.

Regarding python installed via Fink, I do not think it is worth pursuing this issue. Fink has installed python in a non-standard way for OS X. The apple provided python build, the python.org build and the MacPorts all install python 'properly'. That is, it is installed as a framework.

@asmeurer Ok, that's fine. You have used two standard methods of installing matplotlib. To confirm, are you seeing this behaviour using the apple-provided python build in conjunction with matplotlib?

[asmeurer]

QT is installed globally/manually (not via Fink), so that might have something to do with it.

@asmeurer Ok, that's fine. You have used two standard methods of installing matplotlib. To confirm, are you seeing this behaviour using the apple-provided python build in conjunction with matplotlib?

Ah, just noticed that it is different. For the system python, I can select the window with mission control and bring it to the front. For Fink Python, if I do that, it just jumps right back behind the terminal. Additionally, if I manually bring it to the front by moving the terminal out of the way and clicking the window title, text entry still goes to the terminal.

Both start behind the terminal, though.

Also, I don't know if this makes a difference, but plt.plot() shows the plot right away for the system Python, but for Fink, I have to call plt.show() (this is with the regular Python interpreter, not IPython).

[asmeurer]

I can't get matplotlib master to build, so I can't see if #663 fixed the front issue.

[mdehoon]

It really sounds like your Python is not installed as a framework.
With recent versions of matplotlib, you should be able to do:

from matplotlib.backends import _macosx
_macosx.verify_main_display()
True

If it returns True, you have a framework install; if not, you don't. Can you give it a try?
Though this should have been checked automatically when you created the figure...

[pelson]

@mdehoon - can I just say how great it is to have you working on these OSX issues at the moment. I find it really re-assuring to have your vast OSX know-how and experience on board. Long may it continue! 😄

Cheers,

[asmeurer]

>>> from matplotlib.backends import _macosx
>>> _macosx.verify_main_display()
True
>>> import sys
>>> sys.executable
'/sw/bin/python'

[mdehoon]

Can you check if /sw/bin/python is the Python executable itself, or a link to somewhere else?
Doing 'ls -l /sw/bin/python' will tell you that.
On my system, I get

$ ls -l /usr/local/bin/python
lrwxr-xr-x 1 root wheel 60 Feb 3 2012 /usr/local/bin/python -> /Library/Frameworks/Python.framework/Versions/2.7/bin/python

so you can see it actually is a framework install.
If yours is not a framework install, then it explains why the keyboard focus doesn't work properly.
But in that case, the _macosx.verify_main_display function is not a reliable way to test if Python is a framework install, and we'll have to find some other approach to test for that.

[asmeurer]

OK, the actual executable is /sw/bin/python2.7. But that doesn't matter

>>> from matplotlib.backends import _macosx
>>> _macosx.verify_main_display()
True
>>> import sys
>>> sys.executable
'/sw/bin/python2.7'

$ls -l /sw/bin/python2.7
-rwxr-xr-x  1 root  admin   8.8K Jul 13 22:02 /sw/bin/python2.7*

Maybe the Fink guys have patched some stuff to make things work, but it isn't done very well.

[mdehoon]

OK, then at least you know why you experienced this bug: Your python is not installed as a framework.
If you install Python as a framework, the problem should go away.
We're still left with the question though why _macosx.verify_main_display() returns True, since that should not happen for a non-framework installation.

[asmeurer]

I can confirm that it is indeed not an issue for my system python install (which is a framework). It still opens behind the terminal by default, but I think I could live with that :)

One other difference: in the framework version, it runs in the rocket ship Python program. In the Fink version, no new program loads (in the Dock).

For a few reasons, I prefer to use Fink (actually, one of them is that I have bad luck getting libraries like matplotlib to compile, whereas Fink just works in that regard). So it would be great if we could get a workable solution.

By the way, do you guys have a guide on the recommended way to hack on matplotlib, especially the parts written in C somewhere? I'm used to coding in libraries that pure Python, and there, you just import the library from the git directory.

[mdehoon]

If the Python provided by Fink is not a framework installation, then there is nothing we can do to fix this problem. But you could try asking Fink to make Python available as a framework installation.

Btw, can you check what CGMainDisplayID() returns in verify_main_display in src/_macosx.m? To find out, you would have to compile matplotlib though.

[asmeurer]

I was going to play around with it. That's why I asked if you have any guides on how to build/test the code.

[mdehoon]

Just adding a line
printf("display = %d\n", display);
in verify_main_display is enough. Previously CGMainDisplayID returned 0 for a non-framework build. I am wondering what CGMainDisplayID returns in your case.

[asmeurer]

display = 69731264

[asmeurer]

I can't get it to compile for the system Python (unable to execute gcc-4.2: No such file or directory). Again, if you have some kind of guide on best practices for building/testing, I might be able to get something out of that. I can post a full compile log somewhere too if you want more info on the system version.

[dmcdougall]

@asmeurer Do you have XCode installed? If not, that could be the cause of gcc not being present. If you have XCode installed, try clang or clang++. I know OS X recently changed their default compiler.

[asmeurer]

It's not clear to me what even calls gcc. If the Makefile calls cc, that should be using clang. Anyway, it finds and uses gcc just fine when I install through fink (I have gcc installed through fink). The only issue is when installing with the system python.

When I install with fink it uses (I think) the fink gcc, but when it installs with the system python, it tries to use gcc-4.2, which isn't even installed on my system. It's very possible that my configuration messed up in some subtle way, but my prowess and patience for figuring these things out is quite limited.

[asmeurer]

Oh and yes, I do have XCode. I wouldn't get very far without it (certainly wouldn't be able to install Fink). Clang has been the default for quite some time actually.

[dmcdougall]

@asmeurer Try doing export CC=clang to point to clang. Or export CC=/path/to/your/fink/gcc.

[mdehoon]

@asmeurer You could also try "python setup.py build" as usual. The makefile is primarily intended to find dependencies, but the Mac OS X backend itself doesn't rely on any dependencies other the Python and Numpy, which are easy to find.

As a related point, how did you get "display = 69731264" if you didn't recompile matplotlib?

[asmeurer]

OK, I fixed it by creating a symbolic link for gcc-4.2 to clang in my PATH (aside: I noticed that the system Python build is linking against the 10.6 developer SDK (I am running 10.8). So that might have something to do with it. I think the last time Apple used gcc 4.2 was in 10.6).

For my system Python, I also get display = 69731264.

I also tried to play with this a little further. I changed the if statement at https://github.com/matplotlib/matplotlib/blob/master/src/_macosx.m#L5712 to if (1), to force it to think that my Fink Python is not a framework, but then I just get

/sw/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.pyc in new_figure_manager(num, *args, **kwargs)
    232     Create a new figure manager instance
    233     """
--> 234     if not _macosx.verify_main_display():
    235         import warnings
    236         warnings.warn("Python is not installed as a framework. The MacOSX "

RuntimeError: Failed to obtain the display ID of the main display

I would have expected it to just not choose the Mac OS X backend.

[asmeurer]

(Didn't notice your last comment):

I actually am using setup.py install, as I don't know of any other way to test this. It works just fine for fink (I did recompile). The issue is that the system python somehow picks up an older gcc from an older developer sdk that I don't have. Fortunately, clang and gcc have the same interface.

[mdehoon]

OK thanks. Apparently the return value of CGMainDisplayID() is not a reliable way to check whether Python is installed as a framework.
Could you try the following instead?
At the Python prompt, try

import MacOS
MacOS.linkmodel
'framework'

If your Python is installed as a framework, then MacOS.linkmodel should return 'framework'. If not, then it will return either 'static' or 'shared'. Then we can use that to check whether your Python is a framework installation.

You can also look in /sw/include/python2.5/pyconfig.h. It should have WITH_NEXT_FRAMEWORK undefined, as in
/* #undef WITH_NEXT_FRAMEWORK */

If so, then I can modify _macosx.m to check for WITH_NEXT_FRAMEWORK, and issue a warning message if Python is not a framework install.

[asmeurer]

Could you try the following instead?

Actually, I get that in my system Python, but for Fink Python, the MacOS module does not exist!

You can also look in /sw/include/python2.5/pyconfig.h. It should have WITH_NEXT_FRAMEWORK undefined, as in
/* #undef WITH_NEXT_FRAMEWORK */

I can confirm that that line does exist in that particular file.

If so, then I can modify _macosx.m to check for WITH_NEXT_FRAMEWORK, and issue a warning message if Python is not a framework install.

Any way it could disable the Mac OS X backend in that case, rather than issue a warning, so that it automatically chooses the next one (which seem to work fine)?

[mdehoon]

I have made a pull request to use WITH_NEXT_FRAMEWORK to check if Python was installed as a framework; see
#1613

I don't think that matplotlib should automatically choose a different backend if the one chosen in matplotlibrc does not work. For one thing, it is not clear which backend should be chosen instead. Also, if the Mac OS X backend fails because Python is not installed as a backend, then it's easy to select a different backend by modifying matplotlibrc.

Keep in mind that other backends may also fail with non-framework Pythons. This probably will be true even more so in the future, as backends and their underlying libraries switch to Cocoa.

[efiring]

Merged #1613, so I am closing this.

[s-shah]

Hi, I am new to python and have this exact same problem. I somewhat got lost in the conversation above. Python has been (or is) installed as a framework on the machine. What can I do to fix the RunTimeError?

[mdehoon]

@s-shah How do you know that your Python is installed as a framework?

[s-shah]

@mdehoon I typed the following:
Python 2.7.5 (default, Mar 9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import MacOS
MacOS.linkmodel
'framework'

[mdehoon]

OK, then I would suggest to remove the call to verify_main_display from backend_macosx.py.
Or use matplotlib version 1.3.1 or later, which doesn't use verify_main_display.