Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Animation module errors out when using Python3 #1891

Closed
aignas opened this issue Apr 9, 2013 · 35 comments
Closed

Animation module errors out when using Python3 #1891

aignas opened this issue Apr 9, 2013 · 35 comments
Milestone

Comments

@aignas
Copy link
Contributor

aignas commented Apr 9, 2013

I used the double pendulum example and it did not work with the latest version of Matplotlib and Py3k on Arch Linux. However, it worked with Python2.7. I was using

#! /usr/bin/env python3

as my shebang in the script.

The exact script which I used is here: https://gist.github.com/gns-ank/5347084#file-double_pendulum-py

The error log:

Traceback (most recent call last):
  File "./double_pendulum_animated.py", line 88, in <module>
    ani.save('double_pendulum.mp4', fps=15)
  File "/usr/lib/python3.3/site-packages/matplotlib/animation.py", line 616, in save
    writer.grab_frame()
  File "/usr/lib/python3.3/site-packages/matplotlib/animation.py", line 200, in grab_frame
    dpi=self.dpi)
  File "/usr/lib/python3.3/site-packages/matplotlib/figure.py", line 1371, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/usr/lib/python3.3/site-packages/matplotlib/backend_bases.py", line 2101, in print_figure
    self.figure.dpi = origDPI
  File "/usr/lib/python3.3/site-packages/matplotlib/figure.py", line 375, in _set_dpi
    self.dpi_scale_trans.clear().scale(dpi, dpi)
  File "/usr/lib/python3.3/site-packages/matplotlib/transforms.py", line 1751, in clear
    self._mtx = np.identity(3)
  File "/usr/lib/python3.3/site-packages/numpy/core/numeric.py", line 1913, in identity
    from numpy import eye
  File "<frozen importlib._bootstrap>", line 1609, in _handle_fromlist
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Press ENTER or type command to continue
@mdboom
Copy link
Member

mdboom commented Apr 9, 2013

Hmm... I can't reproduce with matplotlib 1.2.1, python 3.3 and numpy 1.7 on a Fedora box.

The traceback leads us into numpy, of course, so perhaps therein lies the bug. Can you open up a Python 3 interpreter and type:

import numpy
numpy.identity(3)

@aignas
Copy link
Contributor Author

aignas commented Apr 10, 2013

I get the following:

Python 3.3.1 (default, Apr  6 2013, 19:03:55) 
[GCC 4.8.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.identity(3)
array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.]])
>>>

I have numpy 1.7.0, mpl 1.2.1.

@mdboom
Copy link
Member

mdboom commented Apr 10, 2013

Interesting. There's definitely something fishy going on. Maybe try to use the Python debugger to see what's going on here. This is not in matplotlib, but in code that is trying to import numpy.eye -- Something corrupted about your __pycache__ files, perhaps?

File "<frozen importlib._bootstrap>", line 1609, in _handle_fromlist
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

@aignas
Copy link
Contributor Author

aignas commented Apr 10, 2013

How should I check for corrupted 'pycache' files?

@mdboom
Copy link
Member

mdboom commented Apr 10, 2013

I'm not sure -- did you perhaps compile numpy and/or matplotlib with an earlier version of Python and then are now opening them with Python 3.3.1?

File "<frozen importlib._bootstrap>", line 1609, in _handle_fromlist

That line number 1609 doesn't make much sense with the version of _bootstrap.py in Python 3.3.1, since there doesn't seem to be any decoding happening on that line (and fromlist is just a regular list):

File "<frozen importlib._bootstrap>", line 1609, in _handle_fromlist

So perhaps you're somehow getting an older version of _bootstrap.py than the one that comes with Python 3.3.1?

I think the only way to really figure this out (as there's a lot of redflags here that something is usual about your environment), is to start this up in the debugger:

python3 -m pdb double_pendulum_animated.py

and when it fails, start poking around and seeing where this non UTF-8 data is coming from that it's trying (and failing) to decode.

If you haven't already done so, I'd try creating a new Python 3.3.1 environment and install everything (numpy, matplotlib etc) fresh into it after clearing the build directories.

@vlsd
Copy link
Contributor

vlsd commented May 20, 2013

I am having the exact same issue while trying to run some older code after an update. Any new info on this? Also, @gns-ank are you by chance running Arch Linux? I have a nagging feeling that this is a packaging bug.

Also, extra info: the error only happens when calling animation.save(); this might be a video codec problem. Using plt.show() or anything like that works without a snag. Any advice?

@wackywendell
Copy link
Contributor

I am also having the same issue, also on Arch Linux. By "same issue" I mean that if I run the same example code, I get the same error and traceback. I have tried recompiling matplotlib in its own directory to no avail.

However, I have noticed that with my own code, I get a very different traceback (but the same error):

Traceback (most recent call last):
  File "src/2Danimation.py", line 71, in <module>
    ani.save('test.mp4', fps=3, writer=writer)#, extra_args=['-vcodec', 'libx264'])
  File "/usr/lib/python3.3/site-packages/matplotlib/animation.py", line 616, in save
    writer.grab_frame()
  File "/usr/lib/python3.3/site-packages/matplotlib/animation.py", line 200, in grab_frame
    dpi=self.dpi)
  File "/usr/lib/python3.3/site-packages/matplotlib/figure.py", line 1371, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/usr/lib/python3.3/site-packages/matplotlib/backend_bases.py", line 2057, in print_figure
    bbox_inches = self.figure.get_tightbbox(renderer)
  File "/usr/lib/python3.3/site-packages/matplotlib/figure.py", line 1500, in get_tightbbox
    for ax in self.axes:
  File "/usr/lib/python3.3/site-packages/matplotlib/figure.py", line 367, in _get_axes
    return self._axstack.as_list()
  File "/usr/lib/python3.3/site-packages/matplotlib/figure.py", line 73, in as_list
    ia_list = [a for k, a in self._elements]
  File "/usr/lib/python3.3/site-packages/matplotlib/figure.py", line 73, in <listcomp>
    ia_list = [a for k, a in self._elements]
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

So... I don't know where this error is coming from, but it looks to me like the from numpy import eye is a red herring...

@trwaters
Copy link

trwaters commented Aug 2, 2013

This problem frustrated me after I made the effort to setup a virtualenv for python3.3 on both my mac and linux boxes. I ended up abandoning that altogether in favor of the environment used to do these types of animations,
http://jakevdp.github.io/blog/2012/08/18/matplotlib-animation-tutorial/
This guy, Jake, uses anaconda, found here https://store.continuum.io/
It is a complete scientific computing package similar to an EPD install, which uses python 2.7 and matplotlib 1.2.1
(I heard of anaconda through Jake's course advertisement on scientific computing in python, worth a look:
https://github.com/jakevdp/2013_fall_ASTR599
You will still probably need ffmpeg, as I didn't check to see if anaconda has a codec already. I got ffmpeg for my mac running Lion using sudo port install ffmpeg (which supposedly depends on lame, so first do sudo port install lame to be safe) and for Ubuntu using this PPA
https://launchpad.net/~jon-severinsson/+archive/ffmpeg

It would still be nice to see the problem resolved for Python 3 however.

@mdboom
Copy link
Member

mdboom commented Aug 2, 2013

It appears that the original bug is actually a bug in Numpy -- not it happens within importing numpy.eye. I'm going to close this, but feel free to reopen if I'm wrong on further investigation.

@mdboom mdboom closed this as completed Aug 2, 2013
@vlsd
Copy link
Contributor

vlsd commented Sep 10, 2013

@mdboom do you happen to know the corresponding bug in numpy or does that need to be filed?

@astrojuanlu
Copy link

@mdboom @vlsd I run into this too and I also wonder which is the corresponding NumPy bug, I'm running recent versions.

@mdboom
Copy link
Member

mdboom commented Nov 27, 2013

Re-reading this thread, it's possible it's not a bug in Numpy. @Juanlu001 -- can you provide the full traceback and exactly which version of matplotlib, Numpy, and Python you are running, on which platform, and how you set up the environment?

@astrojuanlu
Copy link

Sure:

The code is https://gist.github.com/Juanlu001/7674552 (swapping comments in lines 36 and 37 to run animation.save)

$ uname -a
Linux nebulae 3.12.1-1-ARCH #1 SMP PREEMPT Thu Nov 21 08:18:42 CET 2013 x86_64 GNU/Linux
$ python --version
Python 3.3.2
$ python -c "import numpy; print(numpy.version.version)"
1.8.0
$ python -c "import matplotlib; print(matplotlib.__version__)"
1.3.1
$ python mathgif.py 
Traceback (most recent call last):
  File "mathgif.py", line 69, in <module>
    main(traj, cline, size=(WIDTH, HEIGHT), nframes=NFRAMES, npoints=NPOINTS)
  File "mathgif.py", line 37, in main
    anim.save("mathgif.mp4", fps=10)
  File "/home/juanlu/.local/lib/python3.3/site-packages/matplotlib/animation.py", line 719, in save
    writer.grab_frame(**savefig_kwargs)
  File "/home/juanlu/.local/lib/python3.3/site-packages/matplotlib/animation.py", line 205, in grab_frame
    dpi=self.dpi, **savefig_kwargs)
  File "/home/juanlu/.local/lib/python3.3/site-packages/matplotlib/figure.py", line 1422, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/home/juanlu/.local/lib/python3.3/site-packages/matplotlib/backend_bases.py", line 2225, in print_figure
    self.figure.dpi = origDPI
  File "/home/juanlu/.local/lib/python3.3/site-packages/matplotlib/figure.py", line 385, in _set_dpi
    self.dpi_scale_trans.clear().scale(dpi, dpi)
  File "/home/juanlu/.local/lib/python3.3/site-packages/matplotlib/transforms.py", line 1786, in clear
    self._mtx = np.identity(3)
  File "/home/juanlu/.local/lib/python3.3/site-packages/numpy/core/numeric.py", line 2053, in identity
    from numpy import eye
  File "<frozen importlib._bootstrap>", line 1612, in _handle_fromlist
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

@felixonmars
Copy link
Contributor

Well, I tried a lot and still can't figure out with the problem. I've tried to rebuild both numpy and matplotlib against current python (3.3.3) too.

Here are something from pdb:

(Pdb) args
module = <module 'numpy' from '/usr/lib/python3.3/site-packages/numpy/__init__.py'>
fromlist = ('eye',)
import_ = <built-in function __import__>

@astrojuanlu
Copy link

I just created a clean venv to test this, and I still get the same error. These were the steps I followed:

$ python -m venv clean_mpl
$ cd clean_mpl/
$ source bin/activate
$ python ez_setup.py # After downloading
$ python get-pip.py  # After downloading
$ pip install matplotlib
$ pip install scipy
$ python double_pendulum.py 
Traceback (most recent call last):
  File "double_pendulum.py", line 88, in <module>
    ani.save('double_pendulum.mp4', fps=15)
  File "/home/juanlu/Development/Python/test/clean_mpl/lib/python3.3/site-packages/matplotlib/animation.py", line 719, in save
    writer.grab_frame(**savefig_kwargs)
  File "/home/juanlu/Development/Python/test/clean_mpl/lib/python3.3/site-packages/matplotlib/animation.py", line 205, in grab_frame
    dpi=self.dpi, **savefig_kwargs)
  File "/home/juanlu/Development/Python/test/clean_mpl/lib/python3.3/site-packages/matplotlib/figure.py", line 1422, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/home/juanlu/Development/Python/test/clean_mpl/lib/python3.3/site-packages/matplotlib/backend_bases.py", line 2225, in print_figure
    self.figure.dpi = origDPI
  File "/home/juanlu/Development/Python/test/clean_mpl/lib/python3.3/site-packages/matplotlib/figure.py", line 385, in _set_dpi
    self.dpi_scale_trans.clear().scale(dpi, dpi)
  File "/home/juanlu/Development/Python/test/clean_mpl/lib/python3.3/site-packages/matplotlib/transforms.py", line 1786, in clear
    self._mtx = np.identity(3)
  File "/home/juanlu/Development/Python/test/clean_mpl/lib/python3.3/site-packages/numpy/core/numeric.py", line 2053, in identity
    from numpy import eye
  File "<frozen importlib._bootstrap>", line 1610, in _handle_fromlist
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
$ pip freeze
matplotlib==1.3.1
nose==1.3.0
numpy==1.8.0
pyparsing==2.0.1
python-dateutil==2.2
scipy==0.13.2
six==1.4.1
tornado==3.1.1

I think this bug is worth reopening.

@felixonmars
Copy link
Contributor

I've reproduced the same error on a clean Ubuntu 13.10 install (with python 3.3.2 / virtualenv). So it's not an Arch Linux only problem :p

matplotlib==1.3.1
nose==1.3.0
numpy==1.8.0
pyparsing==2.0.1
python-dateutil==2.2
six==1.4.1
tornado==3.1.1

@tacaswell tacaswell reopened this Dec 18, 2013
tacaswell added a commit to tacaswell/matplotlib that referenced this issue Dec 19, 2013
@tacaswell
Copy link
Member

The tests in test_animation.test_save_smoketest recreate this problem on travis. The tests currently wrap the offending saves in a try-except block which raises a KnownFail on the unicode error so that master tests clean.

The issue in 3.3 only and only seems to hit the streaming saves, not the aggregate a bunch of files saves.

@jluttine
Copy link

I'm experiencing this bug. Is there any workaround? How can I save video files in Python 3.3 environment?

@tacaswell
Copy link
Member

The simple (but very slow) work around is to use the file-based writers (FFMpegFile iirc) which will create a zillion temporary files and then stitch them together.

@lguez
Copy link

lguez commented Feb 26, 2014

Same problem with Python 3.2.3, Numpy 1.6.1 and Matplotlib 1.3.0. The example in Matplotlib documentation:

http://matplotlib.org/examples/animation/basic_example_writer.html

for example, does not work. This is a quite bad bug.

@bytbox
Copy link
Contributor

bytbox commented Mar 14, 2014

In write_rgba of _backend_agg.cpp, the s# argument to CallFunction should be y#. I'll do a PR.

@tacaswell tacaswell added this to the v1.3.x milestone Mar 18, 2014
@tacaswell tacaswell removed this from the v1.5.x milestone Mar 18, 2014
@palmieri
Copy link

Hi all,
I am receiving a different error message when I am trying to use the save function in animate. I am using python3.2, numpy 1.6.1, and the last version of matplotlib (cloned from https://github.com/matplotlib/matplotlib), where I have fixed the error #2898 (the one suggested by bytbox). Nevertheless, I am still receiveng an error, related to numpy.

Here the traceback:

Traceback (most recent call last):
  File "draw_cd4_plan.py", line 143, in <module>
    anim.save('basic_animation.mp4', fps=30, extra_args=['-vcodec', 'libx264'])
  File "/usr/local/lib/python3.2/dist-packages/matplotlib-1.4.x-py3.2-linux-x86_64.egg/matplotlib/animation.py", line 725, in save
    writer.grab_frame(**savefig_kwargs)
  File "/usr/local/lib/python3.2/dist-packages/matplotlib-1.4.x-py3.2-linux-x86_64.egg/matplotlib/animation.py", line 210, in grab_frame
    dpi=self.dpi, **savefig_kwargs)
  File "/usr/local/lib/python3.2/dist-packages/matplotlib-1.4.x-py3.2-linux-x86_64.egg/matplotlib/figure.py", line 1470, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/usr/local/lib/python3.2/dist-packages/matplotlib-1.4.x-py3.2-linux-x86_64.egg/matplotlib/backend_bases.py", line 2197, in print_figure
    self.figure.dpi = origDPI
  File "/usr/local/lib/python3.2/dist-packages/matplotlib-1.4.x-py3.2-linux-x86_64.egg/matplotlib/figure.py", line 400, in _set_dpi
    self.dpi_scale_trans.clear().scale(dpi, dpi)
  File "/usr/local/lib/python3.2/dist-packages/matplotlib-1.4.x-py3.2-linux-x86_64.egg/matplotlib/transforms.py", line 1793, in clear
    self._mtx = np.identity(3)
  File "/usr/lib/python3/dist-packages/numpy/core/numeric.py", line 1859, in identity
    a = zeros((n,n), dtype=dtype)
IOError: obtaining file position failed

Any ideas??

@bytbox
Copy link
Contributor

bytbox commented Apr 16, 2014

@palmieri That's one of the errors I encountered when I was working on #2898. Are you sure you merged all of that PR? I believe you need ba5c886.

@palmieri
Copy link

Thanks @bytbox !!! It works I missed it!

@tacaswell tacaswell modified the milestones: v1.4.0, v1.3.x Apr 17, 2014
@tomasbedrich
Copy link

After applying @bytbox's patches to 1.3.x I am getting segaults. Python 3.4, numpy and matplotlib built from source. Any ideas?

@bytbox
Copy link
Contributor

bytbox commented May 11, 2014

@tomasbedrich can you say what lines of code are triggering the segfaults?

@tomasbedrich
Copy link

Unfortunately no, I'm on OS X 10.9 where valgrind doesn't work.

@bytbox
Copy link
Contributor

bytbox commented May 11, 2014

What I mean is, what line of python causes a segfault? Can I have a minimal example?

@tomasbedrich
Copy link

Here is minimal Python example:

#!/usr/bin/env python

from matplotlib import pyplot
from matplotlib import animation

figure, axis = pyplot.subplots()

x = list(range(10))
y = [y ** 2 for y in range(10)]
line, = pyplot.plot(x, y)

def animate(percentage):
    new_y = list(map(lambda a: a * percentage / 100, y))
    line.set_data(x, new_y)

animation = animation.FuncAnimation(figure, animate, frames=100, interval=20, blit=False, repeat=False)
animation.save("animation.gif", writer="imagemagick")  # segfault

I get the same result with writer="ffmpeg". Here is a part of stack trace from standard OS X error dialog which may be interesting:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   _backend_agg.so     0x000000010ef7eae1 RendererAgg::write_rgba(Py::Tuple const&) + 745 (file_compat.h:160)
1   _backend_agg.so     0x000000010ef87dc2 Py::PythonExtension<RendererAgg>::method_varargs_call_handler(_object*, _object*) + 288 (Objects.hxx:190)
2   org.python.python   0x000000010a4a569b PyEval_EvalFrameEx + 15942
...
...

@efiring
Copy link
Member

efiring commented May 11, 2014

If this means that the problem is in line 160 of file_compat.h with #2898 applied, then it looks straightforward: it could be fixed by replacing the Py_DECREF(ret) with Py_XDECREF(ret), or by changing the last few lines to look like

    if (npy_lseek(fd, orig_pos, SEEK_SET) != -1) {
        if (position == -1) {
            PyErr_SetString(PyExc_IOError, "obtaining file position failed");
            return -1;
        }

        /* Seek Python-side handle to the FILE* handle position */
        ret = PyObject_CallMethod(file, "seek", MPL_OFF_T_PYFMT "i", position, 0);
        if (ret == NULL) {
            return -1;
        }
        Py_DECREF(ret);
    }
    return 0;

In other words, move the Py_DECREF up by one block level. Surely that is where it belongs. I suspect this is what was intended, and the present location of Py_DECREF is just a simple typo.

@bytbox
Copy link
Contributor

bytbox commented May 11, 2014

(I'll fix and test this shortly. Thanks @efiring.)

@tomasbedrich
Copy link

I can confirm that @efiring's quick patch is working. Thank you.

@tacaswell
Copy link
Member

@efiring Closing this as I think that #2898 fixes this, please re-open if I am confused.

@wrkyle
Copy link

wrkyle commented Sep 3, 2015

I'm using python 3.4.0, with matplotlib 1.3.1 and am just trying to save my animation but get the errors described above. I see that this appears to be solved on your end but on my end I have no idea what to do to get this to function properly. Can anyone help me out?

@tacaswell
Copy link
Member

The simplest solution is to update to mpl 1.4.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests