Reset the available animation movie writer on rcParam change #5628

Merged
merged 7 commits into from Feb 7, 2016

Conversation

Projects
None yet
7 participants
Contributor

janschulz commented Dec 6, 2015

If one of the rcParams for a path to a program, which was called by a
movie writer, is changed, the the available movie writer in the
registry should be reevaluated if they are (still/became) available.

This also fixes the problem that you have to set the path to a movie
writer before importing mpl.animation, as before the state was
fixed on import time.

Closes: #5616

@janschulz janschulz commented on the diff Dec 9, 2015

lib/matplotlib/animation.py
if writerClass.isAvailable():
self.avail[name] = writerClass
return writerClass
return wrapper
+ def endsure_not_dirty(self):
+ """If dirty, reasks the writers if they are available"""
+ if self._dirty:
+ self.reset_available_writers()
+
+ def reset_available_writers(self):
+ """Reset the available state of all registered writers"""
+ self.avail = {}
+ for name, writerClass in self._registered.items():
+ if writerClass.isAvailable():
@janschulz

janschulz Dec 9, 2015

Contributor

BTW: should that be "is_available"? It's the only method in that class(es) which use CamelCase instead of underscores...

@tacaswell

tacaswell Dec 9, 2015

Owner

It is annoying, but I don't think worth breaking user code over.

@mdboom

mdboom Dec 9, 2015

Owner

At most, we can rename and add an alias isAvailable -> is_available with the @deprecated decorator.

Contributor

janschulz commented Dec 27, 2015

Apart from the possible deprecation (which should go to a different PR), IMO this is ready to go in...

@dopplershift dopplershift and 1 other commented on an outdated diff Dec 28, 2015

lib/matplotlib/animation.py
if writerClass.isAvailable():
self.avail[name] = writerClass
return writerClass
return wrapper
+ def endsure_not_dirty(self):
@dopplershift

dopplershift Dec 28, 2015

Contributor

should endsure be ensure?

@janschulz

janschulz Dec 28, 2015

Contributor

Omg, fix is on the way...

Contributor

janschulz commented Feb 6, 2016

Ping?

@dopplershift dopplershift and 1 other commented on an outdated diff Feb 6, 2016

lib/matplotlib/rcsetup.py
@@ -802,6 +802,21 @@ def validate_hist_bins(s):
raise ValueError("'hist.bins' must be 'auto', an int or " +
"a sequence of floats")
+def validate_animation_writer_path(p):
+ # Make sure it's a string and then figure out if the animations
+ # are already loaded and reset the writers (which will validate
+ # the path on next call)
+ if not isinstance(p, six.text_type):
+ raise ValueError("path must be a (unicode) string")
+ from sys import modules
+ # set dirty, so that the next call to the reistry will reevaluate
@dopplershift

dopplershift Feb 6, 2016

Contributor

... registry will re-evaluate

@janschulz

janschulz Feb 6, 2016

Contributor

done and repushed

Contributor

dopplershift commented Feb 6, 2016

Minor comment nit, otherwise this is 👍 from me.

janschulz added some commits Dec 6, 2015

@janschulz janschulz Reset the available animation movie writer on rcParam change
If one of the rcParams for a path to a program, which was called by a
movie writer, is changed, the the available movie writer in the
registry should be reevaluated if they are (still/became) available.

This also fixes the problem that you have to set the path to a movie
writer before importing mpl.animation, as before the state was
fixed on import time.
e3e2be9
@janschulz janschulz TST: Use a better command for annimation tests
python is blocking, leading to timeouts...
6a01829
Contributor

janschulz commented Feb 6, 2016

cleaned up the typo and repushed

@WeatherGod WeatherGod commented on an outdated diff Feb 6, 2016

lib/matplotlib/tests/test_animation.py
@@ -163,6 +166,30 @@ def animate(i):
frames=iter(range(5)))
+def test_movie_writer_registry():
+ ffmpeg_path = mpl.rcParams['animation.ffmpeg_path']
+ # Not sure about the first state as there could be some writer
+ # which set rcparams
+ #assert_false(animation.writers._dirty)
+ assert_true(len(animation.writers._registered) > 0)
+ animation.writers.list() # resets dirty state
+ assert_false(animation.writers._dirty)
+ mpl.rcParams['animation.ffmpeg_path'] = u"not_available_ever_xxxx"
+ assert_true(animation.writers._dirty)
+ animation.writers.list() # resets
+ assert_false(animation.writers._dirty)
+ assert_false(animation.writers.is_available("ffmpeg"))
+ # something which is garanteered to be available in path
@WeatherGod

WeatherGod Feb 6, 2016

Member

"garanteered"?

Member

WeatherGod commented Feb 6, 2016

Appveyor is having all sorts of failures, some of them having to do with animation stuff, so I am not entirely certain if it is the usual failures or not.

@janschulz janschulz Fix: rerun check for imagemagick in isAvailable
On windows the check ensures that the imagemagick path for 'convert' was
set to '' if it was just 'convert' to prevent clashes with the windows
built-in command 'convert' (doing filesystem conversion). Unfortunately,
the command only set this values in the current rcParams and rcParamsDefault,
but a use of `matplotlib.style.use("classic")` after importing mpl.animation
would again overwrite it with `convert`.

In this case, the image comparison decorator sets the style to classic and
appveyor went BOOM...
6c6cf91
Contributor

janschulz commented Feb 7, 2016

The imagemagic problem is that the workaround for masking convert on windows (which is always available to to the internal convert.exefrom windows) is undone by a later call to matplotlib.style.use("classic") in the image comparison decorator...

It seems that the workaround should also be applied in the is_available method...

And for your daily dose of WTF:

c:\data\external\R
λ C:\Users\jschulz\Dropbox\Programme\cmder\vendor\imagemagick\convert --version
Version: ImageMagick 7.0.0-0 Q16 x64 2016-01-23 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2015 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Visual C++: 180040629
Features: Cipher DPC HDRI
Delegates (built-in): bzlib cairo freetype jng jp2 jpeg lcms lqr openexr pangocairo png ps rsvg tiff webp xml zlib

c:\data\external\R
λ echo %errorlevel%
1
c:\data\external\R                         
λ convert                                  
Ein Dateisystem muss angegeben werden.     

c:\data\external\R                         
λ echo %errorlevel%                        
4                                          

and running the tests with mpl.verbose.set_level("debug") triggers a different bug (pobably in subprocess)?:

======================================================================
ERROR: matplotlib.tests.test_animation.test_save_animation_smoketest('mencoder_file', 'mp4')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\testing\decorators.py", line 152, in wrapped_callable

    func(*args, **kwargs)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\tests\test_animation.py", line 137, in check_save_animation
    anim.save(F.name, fps=30, writer=writer, bitrate=500)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\animation.py", line 893, in save
    writer.grab_frame(**savefig_kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\animation.py", line 174, in saving
    self.finish()
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\animation.py", line 453, in finish
    self._run()
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\animation.py", line 283, in _run
    creationflags=subprocess_creation_flags)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\subprocess.py", line 914, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\subprocess.py", line 1145, in _get_handles
    c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
io.UnsupportedOperation: fileno
Contributor

janschulz commented Feb 7, 2016

I also currently have a bug that mencoder returns 3:

======================================================================
ERROR: matplotlib.tests.test_animation.test_save_animation_smoketest('mencoder_file', 'mp4')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\testing\decorators.py", line 152, in wrapped_callable

    func(*args, **kwargs)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\tests\test_animation.py", line 140, in check_save_animation
    anim.save(F.name, fps=30, writer=writer, bitrate=500)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\animation.py", line 894, in save
    writer.grab_frame(**savefig_kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\animation.py", line 174, in saving
    self.finish()
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\animation.py", line 462, in finish
    + ' Try running with --verbose-debug')
RuntimeError: Error creating movie, return code: 3 Try running with --verbose-debug

I've actually no clue why that's happens :-/ Anyone with mencoder knowledge here?

Contributor

janschulz commented Feb 7, 2016

The error seems to be this (pull from self._proc._stderr_buff via the debugger):

[b"
WARNING: OUTPUT FILE FORMAT IS _AVI_. See -of help.
[mpeg4 @ 00000000019034e0]AVFrame.format is not set
[mpeg4 @ 00000000019034e0]AVFrame.width or height is not set

[mpeg4 @ 00000000019034e0]AVFrame.format is not set
[mpeg4 @ 00000000019034e0]AVFrame.width or height is not set

[mpeg4 @ 00000000019034e0]AVFrame.format is not set
[mpeg4 @ 00000000019034e0]AVFrame.width or height is not set

[mpeg4 @ 00000000019034e0]AVFrame.format is not set
[mpeg4 @ 00000000019034e0]AVFrame.width or height is not set

[mpeg4 @ 00000000019034e0]AVFrame.format is not set
[mpeg4 @ 00000000019034e0]AVFrame.width or height is not set
Assertion v>0 && v<=(1 ? 32 : 16) failed at libavutil/mem.c:233

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
"]
Contributor

janschulz commented Feb 7, 2016

It seems the problem is this: Assertion v>0 && v<=(1 ? 32 : 16) failed at libavutil/mem.c:233. I've no clue what that means but it seems that others also have no clue, at least I didn't find any proper answer by googling :-(

E.g. here: http://ehc.ac/p/mplayer-win32/bugs/179/ -> unanswered :-(

I use this build: http://mplayerwin.sourceforge.net/downloads.html (build mplayer-svn-37610-x86_64.7z)

-> not a mpl bug and so IMO if this passes (without mplayer/mencoder, which is not installed on appveyor) and no one finds another typo :-), this is IMO good to merge...

@janschulz janschulz FIX: add debugging output when the MovieWriter fails
This pulls out the stderr and stdout from the internal POpen buffers (tested
on py3.5 and not shown if any error occures while pulling out the messages...).

It also changes the exception message to mention mpl.verbose.set_level("helpful")
because adding a commandline switch is probably not helpful in todays world where
everybody uses the Jupyter notebook and can't add to the kernel commandline...

It would be nice if this verbose level could be set by nose / tests.py, but I
have found no way yet :-(
9b61b34

@janschulz janschulz and 1 other commented on an outdated diff Feb 7, 2016

lib/matplotlib/animation.py
@@ -432,9 +456,16 @@ def finish(self):
# Check error code for creating file here, since we just run
# the process here, rather than having an open pipe.
if self._proc.returncode:
+ try:
+ stdout = [s.decode() for s in self._proc._stdout_buff]
+ stderr = [s.decode() for s in self._proc._stderr_buff]
+ verbose.report("MovieWriter.finish: stdout: %s" % stdout, level='helpful')
+ verbose.report("MovieWriter.finish: stderr: %s" % stderr, level='helpful')
+ except Exception as e:
+ pass
@janschulz

janschulz Feb 7, 2016

Contributor

This is not my finest code, but it works on py35...

@tacaswell

tacaswell Feb 7, 2016

Owner

This is just to provide improved debugging feed back right?

@janschulz

janschulz Feb 7, 2016

Contributor

yes, when the called converter fails and the user adds the mpl.verbose.set_level("helpful") before calling the failing method again...

Unfortunately, it doesn't work for the unittests unless I add such a set_level call in the unittest file :-(

@tacaswell tacaswell and 1 other commented on an outdated diff Feb 7, 2016

lib/matplotlib/animation.py
ImageMagickBase._init_from_registry()
@writers.register('imagemagick')
-class ImageMagickWriter(MovieWriter, ImageMagickBase):
+class ImageMagickWriter(ImageMagickBase, MovieWriter, ):
@tacaswell

tacaswell Feb 7, 2016

Owner

Why is the class re-ordering needed?

@janschulz

janschulz Feb 7, 2016

Contributor

Hm, I did this to get ImageMagickWriter.isAvailable() picked up from ImageMagickBase and not from MovieWriter (both declare this class method, so one of them is first and this ordering made the ImageMagickBase the first, which then calls the same method in the other class via super(...)).

-> try and error...

@tacaswell

tacaswell Feb 7, 2016

Owner

can you add a line comment noting that this order is needed to make the MI work?

@janschulz

janschulz Feb 7, 2016

Contributor

done

Owner

tacaswell commented Feb 7, 2016

Looks like some long lines have crept in

======================================================================
FAIL: matplotlib.tests.test_coding_standards.test_pep8_conformance_installed_files
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/matplotlib/matplotlib/venv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/travis/build/matplotlib/matplotlib/lib/matplotlib/tests/test_coding_standards.py", line 249, in test_pep8_conformance_installed_files
    expected_bad_files=expected_bad_files)
  File "/home/travis/build/matplotlib/matplotlib/lib/matplotlib/tests/test_coding_standards.py", line 143, in assert_pep8_conformance
    assert_equal(result.total_errors, 0, msg)
AssertionError: 3 != 0 : Found code syntax errors (and warnings):
/home/travis/build/matplotlib/matplotlib/lib/matplotlib/animation.py:462:80: E501 line too long (90 > 79 characters)
/home/travis/build/matplotlib/matplotlib/lib/matplotlib/animation.py:463:80: E501 line too long (90 > 79 characters)
/home/travis/build/matplotlib/matplotlib/lib/matplotlib/animation.py:468:80: E501 line too long (81 > 79 characters)
Owner

tacaswell commented Feb 7, 2016

👍 modulo small comments / pep8

janschulz added some commits Feb 7, 2016

@janschulz janschulz Pep8 fixes 33f564a
@janschulz janschulz Add comment to explain the fix in ImageMagickBase
e783867
Contributor

janschulz commented Feb 7, 2016

Oh damn, I really should add a pep8 precommit / prepush hook in the matplotlib repo...

Ok, added comments and hopefully fixed the PEP8 thingies...

@tacaswell tacaswell added a commit that referenced this pull request Feb 7, 2016

@tacaswell tacaswell Merge pull request #5628 from JanSchulz/ani_writer
Reset the available animation movie writer on rcParam change
424556b

@tacaswell tacaswell merged commit 424556b into matplotlib:master Feb 7, 2016

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

mdboom removed the needs_review label Feb 7, 2016

Owner

efiring commented Oct 18, 2016

@tacaswell, it looks to me like this could be backported to v2.x, in the spirit of making 2.0 less likely to trip people up.

Owner

tacaswell commented Oct 19, 2016

I completely forget about this and was contemplating re-writing it to fix issues that Christoph reported.

@tacaswell tacaswell added a commit to tacaswell/matplotlib that referenced this pull request Oct 19, 2016

@tacaswell tacaswell Merge pull request #5628 from JanSchulz/ani_writer
Reset the available animation movie writer on rcParam change
70418b9

@NelleV NelleV added a commit that referenced this pull request Oct 19, 2016

@NelleV NelleV Merge pull request #7306 from tacaswell/backport_win_animation_reg
Merge pull request #5628 from JanSchulz/ani_writer
f91dff4
Owner

efiring commented Oct 22, 2016

backported to v2.x via #7306

QuLogic added the Animation label Oct 22, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment