ValueError: insecure string pickle

[corradio]

When I run python -c "import matplotlib.animation", I get:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/matplotlib/animation.py", line 454, in <module>
    class AVConvWriter(AVConvBase, FFMpegWriter):
  File "/usr/local/lib/python2.7/site-packages/matplotlib/animation.py", line 66, in wrapper
    if writerClass.isAvailable():
  File "/usr/local/lib/python2.7/site-packages/matplotlib/animation.py", line 266, in isAvailable
    creationflags=subprocess_creation_flags)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1334, in _execute_child
    child_exception = pickle.loads(data)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 966, in load_string
    raise ValueError, "insecure string pickle"
ValueError: insecure string pickle

I'm using Python 2.7.10, matplotlib 1.4.3. Can you help?

[tacaswell]

That is odd and I can not reproduce locally. Also pretty sure that we smoke test this on travis.

Can you do a bit more debugging and sort out what the value of the arguments being passed to subprocess.Popen ?

[corradio]

Great advice thanks! Had to install libav and mplayer, then it worked. Thanks again.

[mdboom]

I'm going to reopen this. It still might be nice to get a better error message (about the missing dependencies) rather than the "insecure string pickle" message. I'll see if I can reproduce in an environment without libav and mplayer and come up with a solution.

[mdboom]

I'm having trouble reproducing the problem as well -- even with the brew python on a Mac as you are doing. Still would be nice to make this more friendly, but not sure where to start.

[corradio]

Sorry I had a messed up configuration so I wouldn't be of much help as to how to reproduce. I only know that brew install libav and brew install mplayer fixed it :/

[Oxtay]

I'm having a similar issue. Installing mplayer and libav didn't work for me. So instead, I upgraded matplotlib to 1.5.0. Now this is the error I get:

RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be 
able to function correctly if Python is not installed as a framework. See the Python 
documentation for more information on installing Python as a framework on Mac OS X. 
Please either reinstall Python as a framework, or try one of the other backends. If you 
are Working with Matplotlib in a virtual enviroment see 'Working with Matplotlib in 
Virtual environments' in the Matplotlib FAQ

,which seems a known problem with virtual environments? The workaround suggested there aren't very clean solutions. Reverting back to version 1.4.3, and reverting every other package I had to previous working stage, I get the same error as above.

The only major thing that has changed is upgrading my OSX to El Capitan. I use a virtual environment and this is the error I get:

      8 
      9 from matplotlib.widgets import Button
---> 10 import matplotlib.animation as animation
     11 from mpl_toolkits.mplot3d.axes3d import Axes3D
     12 

/Users/user/.virtualenvs/amiigo/lib/python2.7/site-packages/matplotlib/animation.py in <module>()
    415 # Combine FFMpeg options with pipe-based writing
    416 @writers.register('ffmpeg')
--> 417 class FFMpegWriter(MovieWriter, FFMpegBase):
    418     def _args(self):
    419         # Returns the command line parameters for subprocess to use

/Users/user/.virtualenvs/amiigo/lib/python2.7/site-packages/matplotlib/animation.py in wrapper(writerClass)
     64     def register(self, name):
     65         def wrapper(writerClass):
---> 66             if writerClass.isAvailable():
     67                 self.avail[name] = writerClass
     68             return writerClass

/Users/user/.virtualenvs/amiigo/lib/python2.7/site-packages/matplotlib/animation.py in isAvailable(cls)
    264                              stdout=subprocess.PIPE,
    265                              stderr=subprocess.PIPE,
--> 266                              creationflags=subprocess_creation_flags)
    267             p.communicate()
    268             return True

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.pyc in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags)
    708                                 p2cread, p2cwrite,
    709                                 c2pread, c2pwrite,
--> 710                                 errread, errwrite)
    711         except Exception:
    712             # Preserve original exception in case os.close raises.

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.pyc in _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, to_close, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite)
   1332                     if e.errno != errno.ECHILD:
   1333                         raise
-> 1334                 child_exception = pickle.loads(data)
   1335                 raise child_exception
   1336 

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in loads(str)
   1380 def loads(str):
   1381     file = StringIO(str)
-> 1382     return Unpickler(file).load()
   1383 
   1384 # Doctest

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in load(self)
    856             while 1:
    857                 key = read(1)
--> 858                 dispatch[key](self)
    859         except _Stop, stopinst:
    860             return stopinst.value

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.pyc in load_string(self)
    964             if rep.startswith(q):
    965                 if len(rep) < 2 or not rep.endswith(q):
--> 966                     raise ValueError, "insecure string pickle"
    967                 rep = rep[len(q):-len(q)]
    968                 break

ValueError: insecure string pickle

[mdboom]

Oh, Macs . Does this work if you use pythonw? (Which is the usual way to get a GUI supported framework-like environment on a Mac).

[jenshnielsen]

With regards to the framework issue the only thing that has changed is that we now actually check for a framework build and error out. With 1.4.3 you would just get a broken half working implementation that cant be forgrounded and has non working buttons etc. I am sorry that the workarounds are non clean to you but I spent a fair amount of time looking for better ones and as far as I know there are none. This is really a bug in virtualenv and there is little we can do about it since the GUI frameworks just don't work in a non framework build. You can try a different backend which might work without a framework build (not wx thou which have a similar check build in)

From the trackeback it seems like you are running OSX system python. Personally I have never had anything but issues with that and would strongly recommend installing an alternative python and leave the system version alone.

[Oxtay]

@mdboom Macs, very true! Ever since upgrading to El Capitan, I've been regretting it.

@jenshnielsen Sorry for my lack of knowledge in this area, but I had a couple of questions. I actually believe I have a python framework and I'm running that version instead of system's python. I have installed python through homebrew. I also tried running out of the virtual environment today but still got the insecure string pickle.
I also went ahead and tried this as well, but in this case, I got a Could not import runpy module. I'm not sure what's causing it yet.

BTW, this is the error I'm getting. @mdboom you're right. Maybe I should use a different backend, but I wasn't sure how to enforce a certain backend:

      9 # Plotting
---> 10 import matplotlib.pyplot as plt
     11 import seaborn as sns
     12 

/usr/local/lib/python2.7/site-packages/matplotlib/pyplot.py in <module>()
    112 
    113 from matplotlib.backends import pylab_setup
--> 114 _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
    115 
    116 _IP_REGISTERED = None

/usr/local/lib/python2.7/site-packages/matplotlib/backends/__init__.pyc in pylab_setup()
     30     # imports. 0 means only perform absolute imports.
     31     backend_mod = __import__(backend_name,
---> 32                              globals(),locals(),[backend_name],0)
     33 
     34     # Things we pull in from all backends

/usr/local/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.py in <module>()
     22 
     23 import matplotlib
---> 24 from matplotlib.backends import _macosx
     25 
     26 

RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are Working with Matplotlib in a virtual enviroment see 'Working with Matplotlib in Virtual environments' in the Matplotlib FAQ

[jenshnielsen]

The main problem with system python is that it ships old versions of numpy and matplotlib (1.3rc... afaik) since this is a system file you cant uninstall it. Sometimes it interferes with the newly installed matplotlib.

Homebrew python don't use these files and thus gets round this issue. Homebrew python should be build as a framework and behave as expected. But when virtualenv creates a new environment it never creates a framework build even if created from a framework python.

You can enforce a backend in a couple of different ways see http://matplotlib.org/faq/usage_faq.html#what-is-a-backend

The pickle issue seems to be the same as #5386 could you try running from a python process?

import subprocess
p = subprocess.Popen(['foo'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate()

And see if that raises an insecure string pickle warning?

[jenshnielsen]

this issue for youtube download is similar ytdl-org/youtube-dl#6840

[Oxtay]

I tried running the above command from python command line. It wasn't a virtual environment, but I still get the insecure string pickle after the second line.

[jenshnielsen]

Thanks that confirms the observation from #5386 and the youtube-dl bug. I don't expect this bug to be limited to virtualenvs or any specific version of matplotlib. It does seem to be specific to el capitan thou. Basically when python is calling a non existing subprocess (such as the non existing program foo) it gets the error report back via a pickle string. For some reason this string gets terminated on el Capitan for some reason.

The other issue with the frameworks build is specific to virtualenvs and is really caused by how python in a virtual env can interact with OSX and is not specific to any resent version of OSX.

[icedawn]

brew install fontconfig

That fixed the issue for me.

[RafaelCosman]

brew install fontconfig doesn't solve this issue for me.

[icedawn]

I have a new machine setup, and I'm getting the problem again. And yes, the code ...

import subprocess
p = subprocess.Popen(['foo'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate()

gives me the insecure string pickle. Installing fontconfig via brew doesn't help. Also, trying pythonw vs python doesn't help.

[icedawn]

Ah, I found the problem (for my install):

brew install imagemagick

I dug into the errors python was showing ...

$ python -c "import matplotlib.animation" 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/matplotlib/animation.py", line 578, in <module>
    class ImageMagickWriter(MovieWriter, ImageMagickBase):
  File "/usr/local/lib/python2.7/site-packages/matplotlib/animation.py", line 74, in wrapper
    if writerClass.isAvailable():
  File "/usr/local/lib/python2.7/site-packages/matplotlib/animation.py", line 277, in isAvailable
    creationflags=subprocess_creation_flags)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1334, in _execute_child
    child_exception = pickle.loads(data)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1388, in loads
    return Unpickler(file).load()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 972, in load_string
    raise ValueError, "insecure string pickle"
ValueError: insecure string pickle
$

And noticed ImageMagick was being used. Given the comments in this thread about missing dependencies, I installed ImageMagick and the issue went away.

[jenshnielsen]

@icedawn Can you check if subprocess32 gives a more meaningful error message? You will need to pip install it first.

import subprocess32 as subprocess
p = subprocess.Popen(['foo'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate()

@RafaelCosman you are probably missing another dependency. Can you try to provide some more information about what you are trying to do?

[icedawn]

Yes @jenshnielsen ... subprocess32 generates a clear error message -- No such file or directory: 'foo'

$ python
Python 2.7.11 (default, Feb 16 2016, 17:51:10) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess32 as subprocess
>>> p = subprocess.Popen(['foo'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 825, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 1574, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'foo'
>>> p.communicate()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'p' is not defined
>>> 

[jenshnielsen]

#6576 tries to work around this issue (which is really a bug in python2.7) by using subprocess32 it would be great if anyone has the time to test it out

[jenshnielsen]

Looks like this will finally be fixed in python 2.712 https://hg.python.org/cpython/raw-file/v2.7.12rc1/Misc/NEWS

- Issue #26083: Workaround a subprocess bug that raises an incorrect
  "ValueError: insecure string pickle" exception instead of the actual
  exception on some platforms such as Mac OS X when an exception raised
  in the forked child process prior to the exec() was large enough that
  it overflowed the internal errpipe_read pipe buffer.

[puiseux]

Running on Capitan, i had the same error, when i import matplotlib.
I shutdown and restart my Mac, and the problem is solved...
just an warning :

/usr/local/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

[prateek-470]

I had the same error...resolved after restarting the mac.

[QuLogic]

2.0 will require subprocess32 to work around this bug (see #6576.)