savefig to sys.stdout fails with pdf backend

[KennethNielsen]

Matplotlib 0.99.1.1 and 1.1.1rc crahes when I try to save image data to sys.stdout with the pdf backend (but works with the png backend). The error can be reproduced with this code by commenting out the savefig png line and commenting in the savefig pdf line.

#!/bin/env python

import sys
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

plt.plot([1,2,3,4])
plt.ylabel('some numbers')

# This will work
plt.savefig(sys.stdout, format='png')

# This will fail
#plt.savefig(sys.stdout, format='pdf')

It produces some of the header and then fails with this traceback:

%PDF-1.4
%�� ��
Traceback (most recent call last):
  File "mpl_fail.py", line 17, in <module>
    plt.savefig(sys.stdout, format='pdf')
  File "/usr/lib/pymodules/python2.7/matplotlib/pyplot.py", line 471, in savefig
    return fig.savefig(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/figure.py", line 1185, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/backend_bases.py", line 2021, in print_figure
    **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/backend_bases.py", line 1789, in print_pdf
    return pdf.print_pdf(*args, **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_pdf.py", line 2180, in print_pdf
    file = PdfFile(filename)
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_pdf.py", line 399, in __init__
    self.writeObject(self.rootObject, root)
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_pdf.py", line 1279, in writeObject
    self.recordXref(object.id)
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_pdf.py", line 1276, in recordXref
    self.xrefTable[id][0] = self.fh.tell()
IOError: [Errno 29] Illegal seek

My first thought (well technically a colleague of mine) was that it was some sort of a permission problem with the mpl files, i.e. that the one the pdf backend lives in doesn't have permission to write to sys.stdout. We checked if this was a packaging problem, so we tested it both on a ubuntu and centos system, with the same result.

If you need any more information, please let me know.

[pelson]

This doesn't occur on master. Will be fixed by 1.2 (release candidate will be available soonish).

[KennethNielsen]

@pelson Hallo and thanks for replying. Did you check it against a older version of the code as well to see if you could reproduce it. Because if it is a file permission problem, then perhaps that wouldn't be replicated in a local copy of the code.

[pelson]

No I didn't. I will boot my virtual machine to give it a shot. (I can't build on mountain lion before a recent change to master).
Are you asking because you are still seeing it on master? or out of good citizenship? :-)

[KennethNielsen]

No I haven't seen it in master. I'm haven't contributed to matplotlib yet, so I don't have it set up to run from git yet. I only asked because of the suspicion that it might be a file permission error, and that may not even be correct. If your tests indicate that it works fine on osx (right), maybe I'll see if I can figure out how to set it up myself and check it on linux.

[pelson]

I'm glad you asked. I should have tried reproducing the problem first, and then seeing that it had actually been resolved.

I can confirm this issue on Ubuntu 12.04 master (just before 1.2.x), but I do not get the problem on OSX 10.8. Presumably to do with some differences in sys.stdout between the two operating systems...

@KennethNielsen: Thanks for the nudge.

[efiring]

Evidently the tell() method is supported on sys.stdout on OSX but not on linux. I would not have expected it to be supported on any operating system. It is called 4 places in backend_pdf.py, apparently for two different purposes. Removing it looks non-trivial.

[mdboom]

On my Fedora box it has a tell:

>ipython
Python 2.7.3 (default, Jul 24 2012, 10:05:38) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.1 -- An enhanced Interactive Python.

In [1]: import sys

In [2]: sys.stdout.tell
Out[2]: <function tell>

It should be noted that we don't explicitly support using stdout anyway -- it just works by accident in some places. The docstring says that the file may be a "file-like object", not an "output stream". ("file-like object" is a superset of "output stream"). I'm being obnoxiously pedantic, of course, but we only promise things work as promised... ;)

EDIT: Of course, having a tell isn't enough. It fails with an IOError when called. Sorry for the false alarm.

[efiring]

Your answer is reasonable, though; I don't see any urgency in making the pdf backend support streams.

[pelson]

I've put it on the wishlist. I haven't tagged it with "low hanging fruit" based on @efiring's comments. I believe the workaround is to write to a stringio and then pipe the output to stdout. @KennethNielsen: If you do this, please feel free to share the code.

[KennethNielsen]

@efiring Thanks for troubleshooting
@mdboom Thanks for the clarification. I think it would be a useful thing to add in the long run, for backend purposes, but obviously doesn't have the highest priority when it is not actually a bug.
@efiring I'm a little in doubt about what you mean. Using the StringIO object as a workaround in mpl or in my script? I can confirm that it works fine as a workaround in my script, I actually already did that, sorry I forgot to mention that. The whole file is a little large to paste (285 lines, I can attach it if you think it is prudent), but the relevant part of the code

if self.o['image_format'] == 'pdf':
    import StringIO
    out = StringIO.StringIO()
    self.fig.savefig(out, bbox_inches='tight', pad_inches=0.03,
                     format=self.o['image_format'])
    sys.stdout.write(out.getvalue())
else:
    self.fig.savefig(sys.stdout, bbox_inches='tight', pad_inches=0.03,
                     format=self.o['image_format'])

works just fine (the other values of self.o['image_format'] are eps, png and svg). As for making a change in mpl I guess a similar approach might fix it, but it will probably double the memory usage for the file. I'll look into it if I find the time.

[jkseppan]

The resulting pdf is actually broken on MacOS too - though probably most viewers can fix it. The offsets in the xref table depend on the results of tell(), which includes everything printed before the pdf file (try calling savefig(sys.stdout, format='png') several times - the table of zero-padded numbers keeps changing every time).

[jkseppan]

I presume the pull request fixed this?

[KennethNielsen]

Yes, as far as my testing of your branch is enough to confirm, then yes, it is fixed.

I unfortunately don't have the option to test matplotlib master in the "produktion" environment (where the original problem was) and will therefore have to wait until a packaged version containing the fix trickles down to CentOS (so probably in half a decade or so G).

Thanks again for your work.