Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

saving a figure triggers (very) excessive IO activity #1530

Closed
jfmoulin opened this Issue · 14 comments

4 participants

@jfmoulin

Saving a figure from the GUI under GTK triggers a lot of IO activity. Up to 60Mb/sec on my system.
The file /.local/share/recently-used.xbel gets written to each time the session history is scrolled through with cursor keys.

I am running matplotlib 1.2.0 but this issue is, I believe, much older than that.

I reported the issue to IPython (see ipython/ipython#2608) first thinking this was a purely IPy problem but I could then reproduce the bug in a normal python session.

As suggested by rkern there, I posted a dump of strace and lsof at http://filebin.ca/NdspC6wfTV3

Another user also reported that saving from script triggers the same problem, but I could personnally not observe that with certainty.

@efiring efiring referenced this issue in ipython/ipython
Closed

slow handling of keyboard input #1184

@efiring
Owner

I think you have provided the essential clue to a problem I was running into some time ago (ipython/ipython#1184). Unfortunately, I don't see any good solution. The problem seems to be deeply embedded. Although everything I have found indicates it is a gtk problem, using the qt4 backend does not solve it; my guess is that the qt4 toolkit is conforming to some standard for activity reporting such that the underlying gtk window manager is being notified and is accessing the infamous recently-used.xbel file. Using the tk backend seems to avoid this; please try that on your system and see if it provides a workaround for you.

Solutions suggested on the web all involve trying to make recently-used.xbel inaccessible in some way; the problem is that the attempted access still triggers a gtk warning message every time.

@jfmoulin

I tried the tk backend and it indeed circumvents the problem. But I was not able to use the GUI to save the figure, I got an exception:
Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1413, in call
return self.func(*args)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/backend_tkagg.py", line 882, in save_figure
default_filetype_name = filetypes[default_filetype]
KeyError: 'auto'

Why on earth would we need the auto mode? Is a drop down list with e.g png on top not the best option?

@efiring
Owner

I can't reproduce that problem; for me, saving via the gui works fine, and the file selector comes up with a png default.

@mdboom
Owner

If you set savefig.format in matplotlibrc, indeed, I can reproduce this bug. Is that a setting you entered yourself? I don't think savefig.format == 'auto' is supposed to have any special meaning.

@jfmoulin

I had a look at my matplotlibrc. Everything is default (i.e the commented version)m except the backend selection which is
backend : GTKAgg

the savefig part is

# the default savefig params can be different from the display params
# Eg, you may want a higher resolution, or to make the figure
# background white
#savefig.dpi       : 100      # figure dots per inch
#savefig.facecolor : white    # figure facecolor when saving
#savefig.edgecolor : white    # figure edgecolor when saving
#savefig.extension : auto     # what extension to use for savefig('foo'), or 'auto'

I am positive I did not edit this.

If I now set it to png, this is what I get

>>> import pylab as p
/usr/local/lib/python2.7/dist-packages/matplotlib/__init__.py:695: UserWarning: savefig.extension is deprecated and replaced with savefig.format; please use the latter.
  warnings.warn(self.msg_depr % (key, alt))

The gui comes up with png extension though. I edited my matplotlibrc according to the new convention and warning is no longer raised. I guess installing the latest matplotlib did not overwrite my old matplotlibrc.

Now, as for the original bug, IO activity. I tried to reproduce the bug from script using savefig. And it seems I never trigger it (I created and saved some hundreds of pics in a loop). As soon as I use the gui, bingo...
The gui calls savefig too, that the gtk does some homekeeping and writes frequetly used once is ok, but why does the history browsing have to retrigger that? Looks more like a readline problem to me ...

(EDIT: @pelson fixed shouting, :smile:, by adding code blocks ``` code ```)

@jfmoulin

oops, sorry I should have previewed... I did not intend shouting at you ;0)

@mdboom
Owner

That's a pretty serious bug. What's happening here is that we deprecated savefig.extension -- if the user has a savefig.extension in their matplotlibrc, we set savefig.format to its value. Unfortunately the new savefig.format does not expect auto as a value. This is something one would only see if they had a matplotlibrc from an earlier version and then upgraded matplotlib without upgrading their matplotlibrc. I'll have to add a fix for this.

I'm still investigating why saving from Gtk+ has such high IO -- still unable to reproduce -- I might send you some further things to investigate if I think of anything.

@jfmoulin

Yes, please!
I did a test on a colleague's computer. He had matplotlib installed but has never used it (no comment).
I tried saving a fig with the gui and I could see that there was some IO when I scrolled the history (using iotop), nevertheless this did not really slow the shell down. I had a look at his /.local/share/recently-used.xbel and the file was brand new and only some kb. In comparison my recently-used was almost 1Mb. I cleared it and tried again. Now the bug is still to be detected but one could possibly not notice it without accurate monitoring because this does not slow down the machine much (still, this remains not overly nice).

@mdboom mdboom referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@mdboom
Owner

So does that suggest that this isn't a bug with matplotlib, but instead with Gtk/Gnome's recently used handling? We don't ever touch or deal with that file directly, but simply use the standard dialog that Gtk provides.

@jfmoulin

Well, it seems even worse than I thought.
In order to reproduce the IO problem, it is enough to press the save button and open the FileChooserDialog. Even after canceling the operation before actually saving, the damn recently_used file gets written to at each cursor key press!
So I guess it is really a GTK problem...
I'll try my luck and submit a bug report there.
Thanks for your input.

@mdboom mdboom referenced this issue from a commit in mdboom/matplotlib
@mdboom mdboom If the user has "savefig.extension : auto" (deprecated) in their (old…
…) matplotlibrc, it will be copied into "savefig.format", where "auto" is not a supported choice. This will fix it so it becomes "png" if it's currently set to "auto". This bug discovered as part of #1530, though it doesn't fix it.
f145b2d
@oliford oliford referenced this issue in ipython/ipython
Closed

IPython terminal issue with Qt4Agg on XP SP3 #481

@oliford

Did this get reported at Gtk? I can't see a link.

Two of us here also seeing this issue. After one opening of the save figure dialog, the recently-used.xbel file is continuously written upon keyboard input in the terminal.

There seems to be an old related bug in vim ( http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=456897 ).

Unfortunately, it seems they 'solved' it by not using the new GTK file selector but they do report this little gem:
"This is because, when using the newer chooser, Gtk writes out the 'Recently Used Files' list everytime the Gtk main loop exits.". Would that also be true in IPython?

@oliford

It seems it came up again later for vim, to which the gtk folks claimed they were misusing Gtk's main loop. If that is also true for IPython/PyGTK/matplotlib (I don't even understand what uses what), then we/they/you should perhaps weigh in on the debate here: https://bugzilla.gnome.org/show_bug.cgi?id=546691

@efiring
Owner

With or without ipython, we are calling main_quit() only when the last figure is closed in non-interactive mode. If main_quit is getting called with every cursor motion, it must be getting done inside of gtk itself.

We still have a major bug here. To reproduce (on linux, ubuntu Precise virtual machine, mpl master):

Bring up two terminal windows. In the first:

strace -o testgtk ipython --pylab=gtk

In the second:

grep recently-used testgtk | wc

Now, you can periodically use the second window to see what is causing accesses to recently-used. It turns out that if you don't bring up the gtk savefig file chooser at all, there are no accesses. But if you do bring it up, even without saving, then forever after, each keystroke in the ipython window triggers eleven accesses! So evidently the gtk file chooser is leaving something nasty running in the background.

@jfmoulin

Hello,

sorry for the late reaction.... yes I actually reported the bug at gnome under
https://bugzilla.gnome.org/show_bug.cgi?id=689292
but this does not seem to raise substantial interest.

@jfmoulin jfmoulin closed this
@jfmoulin jfmoulin reopened this
@efiring efiring closed this issue from a commit
@efiring efiring backend_gtk: don't hide FileChooserDialog; closes #1530
Hiding the FileChooserDialog was pointless, because a
new one is created each time the Save Figure button is
clicked in the Toolbar.  Hiding seems to have prevented
the widget from being destroyed based on refcount, and this
led to heavy access to the recently-used.xbel file with
every subsequent keystroke.
bb6e6a9
@efiring efiring closed this in bb6e6a9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.