Skip to content
This repository

Work around missing subprocess members on Google App Engine #1825

Merged
merged 8 commits into from about 1 year ago

6 participants

Matt Giuca Phil Elson Dara Adib Michael Droettboom Benjamin Root Derek Homeier
Matt Giuca

On Google App Engine, the subprocess module exists, but is empty. This patch introduces a replacement module, subprocess_fixed, which provides a stub for subprocess.Popen which is compatible with existing usage.

I noticed that cbook already has a bit of a hack to work around the fact that subprocess.check_output is missing in Python 2.6, so I've moved that into my subprocess_fixed module as well, so now it has two reasons to exist.

Calls to subprocess_fixed.Popen on Google App Engine always raise OSError, which most of the codebase is already prepared for (since that's what happens when the application is missing). I've added a few missing catches of OSError.

Partial fix for Issue #1823.

Phil Elson pelson commented on the diff
lib/matplotlib/cbook.py
((6 lines not shown))
1216 1215
     pid = os.getpid()
1217 1216
     if sys.platform == 'sunos5':
1218  
-        a2 = Popen('ps -p %d -o osz' % pid, shell=True,
1219  
-                   stdout=PIPE).stdout.readlines()
  1217
+        try:
  1218
+            a2 = Popen('ps -p %d -o osz' % pid, shell=True,
  1219
+                       stdout=PIPE).stdout.readlines()
  1220
+        except OSError:
  1221
+            raise NotImplementedError(
3
Phil Elson Collaborator
pelson added a note

I'm tempted to suggest a RuntimeError here. None-the-less, this is an API change which should be documented in docs/api/api_changes.rst

Done. Note that I am having trouble building the docs locally, so I'm hoping my ReST syntax is correct!

Also, I agree RuntimeError is more appropriate (since NotImplementedError implies that it is planning to be fixed one day), but I changed this to be compatible with the Windows implementation. Happy to change it if you want. For what it's worth, NotImplementedError is a subclass of RuntimeError.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Phil Elson pelson commented on the diff
lib/matplotlib/cbook.py
((28 lines not shown))
1822  
-    if 'stdout' in kwargs:
1823  
-        raise ValueError('stdout argument not allowed, it will be overridden.')
1824  
-    process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
1825  
-    output, unused_err = process.communicate()
1826  
-    retcode = process.poll()
1827  
-    if retcode:
1828  
-        cmd = kwargs.get("args")
1829  
-        if cmd is None:
1830  
-            cmd = popenargs[0]
1831  
-        raise subprocess.CalledProcessError(retcode, cmd, output=output)
1832  
-    return output
1833  
-
1834  
-
1835  
-# python2.7's subprocess provides a check_output method
1836  
-if hasattr(subprocess, 'check_output'):
1837  
-    check_output = subprocess.check_output
4
Phil Elson Collaborator
pelson added a note

This will also need to be documented in api_changes.

The fact that it has been removed from cbook? It was never documented in the first place.

If you think that people are depending on cbook.check_output, perhaps I should just put it back there as an alias (check_output = subprocess_fixed.check_output)?

Phil Elson Collaborator
pelson added a note

No, I'm happy for us to move it out - the subprocess compat module makes more sense than cbook. We just need to make sure any public (even if undocumented) API changes are put in the api_changes.rst file.

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/subprocess_fixed.py
... ...
@@ -0,0 +1,76 @@
  1
+"""
  2
+A replacement wrapper around the subprocess module, with a number of
  3
+work-arounds:
  4
+- Provides the check_output function (which subprocess only provides from Python
  5
+  2.7 onwards).
  6
+- Provides a stub implementation of subprocess members on Google App Engine
  7
+  (which are missing in subprocess).
4
Phil Elson Collaborator
pelson added a note

I'd suggest an enhancement to core cpython might be desirable here too. Have you looked into doing that?

Phil Elson Collaborator
pelson added a note

(that would probably only apply to python3.4 onwards though)

Michael Droettboom Owner
mdboom added a note

Is there a reason AppEngine has an empty subprocess module rather than one that provides exception-raising stubs?

@pelson: What enhancement are you suggesting for core cpython? The missing subprocess members is not a cpython problem, it's an App Engine problem. I don't know the rationale (why we don't supply stubs that raise OSError instead of them simply being missing), but this is fairly consistent across App Engine for modules that are unsupported. I've had a discussion with the Python team about changing this and it would be unlikely, because it could break existing users expectations (for example, they may be catching AttributeError). I think for now it would be easier to bring this into matplotlib, if it's not too much trouble. If App Engine ends up changing subprocess to include stubs, then it can be removed from Matplotlib.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Phil Elson pelson commented on the diff
lib/matplotlib/texmanager.py
... ...
@@ -63,8 +63,12 @@
63 63
 
64 64
 
65 65
 def dvipng_hack_alpha():
66  
-    p = Popen('dvipng -version', shell=True, stdin=PIPE, stdout=PIPE,
67  
-              stderr=STDOUT, close_fds=(sys.platform != 'win32'))
  66
+    try:
  67
+        p = Popen('dvipng -version', stdin=PIPE, stdout=PIPE, stderr=STDOUT,
  68
+                  close_fds=(sys.platform != 'win32'))
  69
+    except OSError:
  70
+        mpl.verbose.report('No dvipng was found', 'helpful')
  71
+        return False
3
Phil Elson Collaborator
pelson added a note

Is this a good idea? Previously an exception would have been raised, now False is returned. What impact does that have?

I'm happy to back off on this since it doesn't block texmanager from being imported, and it won't work anyway if dvipng is missing.

I don't think dvipng_hack_alpha is a command that should be used externally. It's used within the module to decide whether to change the output from dvipng. It looks like matplotlib detects the presence of dvipng on startup and if it isn't present, it disables the use of the tex module anyway. So this shouldn't really have an effect.

Did you want me to revert this, or document it, or just leave as-is?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/dviread.py
... ...
@@ -23,9 +23,9 @@
23 23
 import errno
24 24
 import matplotlib
25 25
 import matplotlib.cbook as mpl_cbook
  26
+import matplotlib.subprocess_fixed as subprocess
4
Phil Elson Collaborator
pelson added a note

I'm not a huge fan of the name subprocess_fixed. I wonder if it would be worth having matplotlib.compatibility.subprocess.

@mdboom - what are your thoughts on this?

Michael Droettboom Owner
mdboom added a note

Yes -- I like the idea of having a compat submodule containing stuff like this... we already have compatibility code scattered about, but I've seen the compat module approach in other projects and it's really nice. It makes it easy to clean it out later when the minimum requirements change etc.

Benjamin Root Collaborator
WeatherGod added a note

Moved matplotlib.subprocess_fixed to matplotlib.compat.subprocess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Phil Elson
Collaborator

Again, this is looking good @mgiuca-google - thanks for your hard work.

Matt Giuca

Hmm ... looks like Travis failed on Python 3 with this error:

$ python ../matplotlib/tests.py
Traceback (most recent call last):
  File "../matplotlib/tests.py", line 8, in <module>
    import matplotlib
  File "/home/travis/virtualenv/python3.2/lib/python3.2/site-packages/matplotlib-1.3.x-py3.2-linux-x86_64.egg/matplotlib/__init__.py", line 136, in <module>
    from matplotlib.compat import subprocess
  File "/home/travis/virtualenv/python3.2/lib/python3.2/site-packages/matplotlib-1.3.x-py3.2-linux-x86_64.egg/matplotlib/compat/subprocess.py", line 18, in <module>
    from . import subprocess
ImportError: cannot import name subprocess

The command "python ../matplotlib/tests.py" exited with 1.

I'm not sure why my code import subprocess was magically changed to from . import subprocess -- can you think of what is doing this and a way around it? In any case, I think the problem is that the name subprocess is now overloaded. From within the compat package, it could refer to either my overriding version, or the official subprocess library. This doesn't sound like a very good idea, but then if we can't call it "subprocess" then we're back to requiring all imports of this module to use "as".

I think this can be worked around using __import__ magic. Does that sound like a good idea, or would you rather just rename the module?

Phil Elson
Collaborator

I'm not sure why my code import subprocess was magically changed to from . import subprocess -- can you think of what is doing this and a way around it

Python 2's default is to use relative imports before absolute ones, so import subprocess in python 2 is the same as from . import subprocess in python 3 (assuming you have a submodule called subprocess).

In order to get the behaviour you desire, I think you can just add the line from __future__ import absolute_import into your subprocess compatibility module.

It does look a little like 2to3 is being a little over zealous here...

added some commits
Matt Giuca Added new module, matplotlib.subprocess_fixed, as a replacement for s…
…ubprocess.

cbook: Moved check_output to subprocess_fixed.
backend_pgf: Import check_output from subprocess_fixed instead of cbook.
0755aa6
Matt Giuca All modules now use matplotlib.subprocess_fixed instead of subprocess. 52c075f
Matt Giuca subprocess_fixed: Gracefully handle platforms without subprocess.Popen.
If subprocess.Popen is missing (for example, on Google App Engine), replaces it
with a dummy version that raises OSError. In these environments, calls to
subprocess will be handled properly as if the called app could not be found,
instead of raising AttributeError.
b29c93c
Matt Giuca texmanager: Ignore dvipng if it cannot be found, instead of raising O…
…SError.
a117e19
Matt Giuca cbook.report_memory: If ps not found, raises NotImplementedError, not…
… OSError.
ea342b6
Matt Giuca Moved matplotlib.subprocess_fixed to matplotlib.compat.subprocess. f41000d
Matt Giuca api_changes: Document the API changes.
- The subtle change to cbook.report_memory's exception types.
- Moving subprocess_fixed to compat.subprocess.
888fa20
Matt Giuca compat.subprocess: Turn on absolute importing.
This prevents the module from importing itself (as opposed to the global
subprocess module) on Python 3 after 2to3 conversion.
03cddc8
Matt Giuca

OK thanks for the explanation. What I don't get is if Python 2 has relative imports by default, why does it work? (It only stopped working in Python 3.)

Oh well, I've added the absolute import so we'll see what Travis says. Also rebased.

Phil Elson pelson merged commit 8ec9555 into from
Phil Elson pelson closed this
Matt Giuca mgiuca-google deleted the branch
Derek Homeier

This Popen call is broken with matplotlib.compat.subprocess, as its Popen (just like subprocess.Popen) does not accept a full command line but rather a list of [command, flags, args...].
So this needs to be changed to Popen('dvipng', '-version'], ...)

What do you mean "broken"? It worked for me last time I checked (from memory?), and as far as I know, it always has. From the Python docs (emphasis mine):

args should be a sequence of program arguments or else a single string. By default, the program to execute is the first item in args if args is a sequence. If args is a string, the interpretation is platform-dependent and described below.

It goes on to say that "it is recommended to pass args as a sequence", which I agree is good advice. But this shouldn't be broken.

It goes further on just into the next paragraph:

On Unix, if args is a string, the string is interpreted as the name or path of the program to execute. However, this can only be done if not passing arguments to the program.

The string is interpreted as a command line only with shell=True set, which is not the case here and also discouraged. Otherwise the Unix interpreter takes "dvipng -version" as the literal name of the executable, which of course does not exist:

matplotlib version 1.3.x
verbose.level helpful
interactive is True
platform is darwin
>>> p = matplotlib.compat.subprocess.Popen('ls')
>>> CHANGELOG       LICENSE         README.rst      agg24           distribute_setup.py lib         setup.cfg.template  src         ttconv
CONTRIBUTING.md     MANIFEST.in     TODO            boilerplate.py      distribute_setup.pyc    license.py      setup.py        tests.py        unit
CXX         Makefile        TODO_TESTS      build           doc         matplotlibrc.template   setupext.py     tools
INSTALL         README.osx      __pycache__     dist            examples        release         setupext.pyc        tox.ini

>>> p = matplotlib.compat.subprocess.Popen('ls -F')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sw/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/sw/lib/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
>>> p = matplotlib.compat.subprocess.Popen('ls -F'.split())
>>> CHANGELOG       LICENSE/        README.rst      agg24/          distribute_setup.py*    lib/            setup.cfg.template  src/            ttconv/
CONTRIBUTING.md     MANIFEST.in     TODO            boilerplate.py      distribute_setup.pyc    license.py      setup.py        tests.py*       unit/
CXX/            Makefile        TODO_TESTS      build/          doc/            matplotlibrc.template   setupext.py     tools/
INSTALL         README.osx      __pycache__/        dist/           examples/       release/        setupext.pyc        tox.ini

This is on MacOS X, but subprocess.Popen shows exactly the same behaviour on Debian 7, and according to the docs on every Unix system.

Cheers,

Derek

Owner

Confirmed. I've opened #1911 to track this.

Oh, I see, I removed shell=True in my patch. No idea why I did that or if it was an accident. Oops. I'll fix it (properly). Edit: Thanks for finding this, @dhomeier.

Collaborator

All fixed in master. Thanks everyone.

Dara Adib

Looks like 70f2296 added the line CalledProcessError = subprocess.CalledProcessError to compat/subprocess.py which causes an exception to raised since the attribute doesn't exist.

Michael Droettboom
Owner

@daradib: If you're a Google App Engine user, would you mind writing up a PR that works around the issue?

Dara Adib daradib referenced this pull request from a commit in daradib/matplotlib
Dara Adib Work around subprocess.CalledProcessError on Google App Engine
Set subprocess.CalledProcessError to None on restrictive environments
like Google App Engine because the exception cannot be raised and there
is no need to catch it. Builds upon #1825 which broke after #1857.
e99acd6
Dara Adib daradib referenced this pull request from a commit in daradib/matplotlib
Dara Adib Work around subprocess.CalledProcessError on Google App Engine
Set subprocess.CalledProcessError to None on restrictive environments
like Google App Engine because the exception cannot be raised and there
is no need to catch it. Fixes #1825 which broke after #1857.
15d09a5
Dara Adib

@mdboom Ok, sure. I submitted #2860 to the maintenance branch because it's a tiny bugfix. If I should set it to master, let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 8 unique commits by 1 author.

Mar 20, 2013
Matt Giuca Added new module, matplotlib.subprocess_fixed, as a replacement for s…
…ubprocess.

cbook: Moved check_output to subprocess_fixed.
backend_pgf: Import check_output from subprocess_fixed instead of cbook.
0755aa6
Matt Giuca All modules now use matplotlib.subprocess_fixed instead of subprocess. 52c075f
Matt Giuca subprocess_fixed: Gracefully handle platforms without subprocess.Popen.
If subprocess.Popen is missing (for example, on Google App Engine), replaces it
with a dummy version that raises OSError. In these environments, calls to
subprocess will be handled properly as if the called app could not be found,
instead of raising AttributeError.
b29c93c
Matt Giuca texmanager: Ignore dvipng if it cannot be found, instead of raising O…
…SError.
a117e19
Matt Giuca cbook.report_memory: If ps not found, raises NotImplementedError, not…
… OSError.
ea342b6
Matt Giuca Moved matplotlib.subprocess_fixed to matplotlib.compat.subprocess. f41000d
Matt Giuca api_changes: Document the API changes.
- The subtle change to cbook.report_memory's exception types.
- Moving subprocess_fixed to compat.subprocess.
888fa20
Matt Giuca compat.subprocess: Turn on absolute importing.
This prevents the module from importing itself (as opposed to the global
subprocess module) on Python 3 after 2to3 conversion.
03cddc8
This page is out of date. Refresh to see the latest.
7  doc/api/api_changes.rst
Source Rendered
@@ -32,6 +32,13 @@ Changes in 1.3.x
32 32
   by ``self.vline`` for vertical cursors lines and ``self.hline`` is added
33 33
   for the horizontal cursors lines.
34 34
 
  35
+* On POSIX platforms, the :func:`~matplotlib.cbook.report_memory` function
  36
+  raises :class:`NotImplementedError` instead of :class:`OSError` if the
  37
+  :command:`ps` command cannot be run.
  38
+
  39
+* The :func:`~matplotlib.cbook.check_output` function has been moved to
  40
+  `~matplotlib.compat.subprocess`.
  41
+
35 42
 Changes in 1.2.x
36 43
 ================
37 44
 
3  lib/matplotlib/__init__.py
@@ -124,7 +124,7 @@
124 124
                 '.'.join(str(x) for x in _required),
125 125
                 sys.version_info[0]))
126 126
 
127  
-import os, re, shutil, subprocess, warnings
  127
+import os, re, shutil, warnings
128 128
 import distutils.sysconfig
129 129
 import distutils.version
130 130
 
@@ -132,6 +132,7 @@
132 132
 # definitions, so it is safe to import from it here.
133 133
 from matplotlib.cbook import MatplotlibDeprecationWarning
134 134
 from matplotlib.cbook import is_string_like
  135
+from matplotlib.compat import subprocess
135 136
 
136 137
 try:
137 138
     reload
2  lib/matplotlib/animation.py
@@ -20,8 +20,8 @@
20 20
 import sys
21 21
 import itertools
22 22
 import contextlib
23  
-import subprocess
24 23
 from matplotlib.cbook import iterable, is_string_like
  24
+from matplotlib.compat import subprocess
25 25
 from matplotlib import verbose
26 26
 from matplotlib import rcParams
27 27
 
4  lib/matplotlib/backends/backend_pgf.py
@@ -7,7 +7,6 @@
7 7
 import shutil
8 8
 import tempfile
9 9
 import codecs
10  
-import subprocess
11 10
 import atexit
12 11
 import weakref
13 12
 
@@ -21,7 +20,8 @@
21 20
 from matplotlib import font_manager
22 21
 from matplotlib.ft2font import FT2Font
23 22
 from matplotlib.cbook import is_string_like, is_writable_file_like
24  
-from matplotlib.cbook import check_output
  23
+from matplotlib.compat import subprocess
  24
+from matplotlib.compat.subprocess import check_output
25 25
 
26 26
 
27 27
 ###############################################################################
2  lib/matplotlib/backends/backend_ps.py
@@ -88,7 +88,7 @@ def gs_version(self):
88 88
         except KeyError:
89 89
             pass
90 90
 
91  
-        from subprocess import Popen, PIPE
  91
+        from matplotlib.compat.subprocess import Popen, PIPE
92 92
         pipe = Popen(self.gs_exe + " --version",
93 93
                      shell=True, stdout=PIPE).stdout
94 94
         if sys.version_info[0] >= 3:
72  lib/matplotlib/cbook.py
@@ -17,7 +17,6 @@
17 17
 import locale
18 18
 import os
19 19
 import re
20  
-import subprocess
21 20
 import sys
22 21
 import threading
23 22
 import time
@@ -1212,19 +1211,34 @@ def restrict_dict(d, keys):
1212 1211
 
1213 1212
 def report_memory(i=0):  # argument may go away
1214 1213
     'return the memory consumed by process'
1215  
-    from subprocess import Popen, PIPE
  1214
+    from matplotlib.compat.subprocess import Popen, PIPE
1216 1215
     pid = os.getpid()
1217 1216
     if sys.platform == 'sunos5':
1218  
-        a2 = Popen('ps -p %d -o osz' % pid, shell=True,
1219  
-                   stdout=PIPE).stdout.readlines()
  1217
+        try:
  1218
+            a2 = Popen('ps -p %d -o osz' % pid, shell=True,
  1219
+                       stdout=PIPE).stdout.readlines()
  1220
+        except OSError:
  1221
+            raise NotImplementedError(
  1222
+                "report_memory works on Sun OS only if "
  1223
+                "the 'ps' program is found")
1220 1224
         mem = int(a2[-1].strip())
1221 1225
     elif sys.platform.startswith('linux'):
1222  
-        a2 = Popen('ps -p %d -o rss,sz' % pid, shell=True,
1223  
-                   stdout=PIPE).stdout.readlines()
  1226
+        try:
  1227
+            a2 = Popen('ps -p %d -o rss,sz' % pid, shell=True,
  1228
+                       stdout=PIPE).stdout.readlines()
  1229
+        except OSError:
  1230
+            raise NotImplementedError(
  1231
+                "report_memory works on Linux only if "
  1232
+                "the 'ps' program is found")
1224 1233
         mem = int(a2[1].split()[1])
1225 1234
     elif sys.platform.startswith('darwin'):
1226  
-        a2 = Popen('ps -p %d -o rss,vsz' % pid, shell=True,
1227  
-                   stdout=PIPE).stdout.readlines()
  1235
+        try:
  1236
+            a2 = Popen('ps -p %d -o rss,vsz' % pid, shell=True,
  1237
+                       stdout=PIPE).stdout.readlines()
  1238
+        except OSError:
  1239
+            raise NotImplementedError(
  1240
+                "report_memory works on Mac OS only if "
  1241
+                "the 'ps' program is found")
1228 1242
         mem = int(a2[1].split()[0])
1229 1243
     elif sys.platform.startswith('win'):
1230 1244
         try:
@@ -1795,45 +1809,3 @@ def get_instancemethod(self):
1795 1809
 else:
1796 1810
     def _putmask(a, mask, values):
1797 1811
         return np.copyto(a, values, where=mask)
1798  
-
1799  
-
1800  
-def _check_output(*popenargs, **kwargs):
1801  
-    r"""Run command with arguments and return its output as a byte
1802  
-    string.
1803  
-
1804  
-    If the exit code was non-zero it raises a CalledProcessError.  The
1805  
-    CalledProcessError object will have the return code in the
1806  
-    returncode
1807  
-    attribute and output in the output attribute.
1808  
-
1809  
-    The arguments are the same as for the Popen constructor.  Example::
1810  
-
1811  
-    >>> check_output(["ls", "-l", "/dev/null"])
1812  
-    'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'
1813  
-
1814  
-    The stdout argument is not allowed as it is used internally.
1815  
-    To capture standard error in the result, use stderr=STDOUT.::
1816  
-
1817  
-    >>> check_output(["/bin/sh", "-c",
1818  
-    ...               "ls -l non_existent_file ; exit 0"],
1819  
-    ...              stderr=STDOUT)
1820  
-    'ls: non_existent_file: No such file or directory\n'
1821  
-    """
1822  
-    if 'stdout' in kwargs:
1823  
-        raise ValueError('stdout argument not allowed, it will be overridden.')
1824  
-    process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
1825  
-    output, unused_err = process.communicate()
1826  
-    retcode = process.poll()
1827  
-    if retcode:
1828  
-        cmd = kwargs.get("args")
1829  
-        if cmd is None:
1830  
-            cmd = popenargs[0]
1831  
-        raise subprocess.CalledProcessError(retcode, cmd, output=output)
1832  
-    return output
1833  
-
1834  
-
1835  
-# python2.7's subprocess provides a check_output method
1836  
-if hasattr(subprocess, 'check_output'):
1837  
-    check_output = subprocess.check_output
1838  
-else:
1839  
-    check_output = _check_output
77  lib/matplotlib/compat/subprocess.py
... ...
@@ -0,0 +1,77 @@
  1
+"""
  2
+A replacement wrapper around the subprocess module, with a number of
  3
+work-arounds:
  4
+- Provides the check_output function (which subprocess only provides from Python
  5
+  2.7 onwards).
  6
+- Provides a stub implementation of subprocess members on Google App Engine
  7
+  (which are missing in subprocess).
  8
+
  9
+Instead of importing subprocess, other modules should use this as follows:
  10
+
  11
+from matplotlib.compat import subprocess
  12
+
  13
+This module is safe to import from anywhere within matplotlib.
  14
+"""
  15
+
  16
+from __future__ import absolute_import    # Required to import subprocess
  17
+from __future__ import print_function
  18
+
  19
+import subprocess
  20
+
  21
+__all__ = ['Popen', 'PIPE', 'STDOUT', 'check_output']
  22
+
  23
+
  24
+if hasattr(subprocess, 'Popen'):
  25
+    Popen = subprocess.Popen
  26
+    # Assume that it also has the other constants.
  27
+    PIPE = subprocess.PIPE
  28
+    STDOUT = subprocess.STDOUT
  29
+else:
  30
+    # In restricted environments (such as Google App Engine), these are
  31
+    # non-existent. Replace them with dummy versions that always raise OSError.
  32
+    def Popen(*args, **kwargs):
  33
+        raise OSError("subprocess.Popen is not supported")
  34
+    PIPE = -1
  35
+    STDOUT = -2
  36
+
  37
+
  38
+def _check_output(*popenargs, **kwargs):
  39
+    r"""Run command with arguments and return its output as a byte
  40
+    string.
  41
+
  42
+    If the exit code was non-zero it raises a CalledProcessError.  The
  43
+    CalledProcessError object will have the return code in the
  44
+    returncode
  45
+    attribute and output in the output attribute.
  46
+
  47
+    The arguments are the same as for the Popen constructor.  Example::
  48
+
  49
+    >>> check_output(["ls", "-l", "/dev/null"])
  50
+    'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'
  51
+
  52
+    The stdout argument is not allowed as it is used internally.
  53
+    To capture standard error in the result, use stderr=STDOUT.::
  54
+
  55
+    >>> check_output(["/bin/sh", "-c",
  56
+    ...               "ls -l non_existent_file ; exit 0"],
  57
+    ...              stderr=STDOUT)
  58
+    'ls: non_existent_file: No such file or directory\n'
  59
+    """
  60
+    if 'stdout' in kwargs:
  61
+        raise ValueError('stdout argument not allowed, it will be overridden.')
  62
+    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  63
+    output, unused_err = process.communicate()
  64
+    retcode = process.poll()
  65
+    if retcode:
  66
+        cmd = kwargs.get("args")
  67
+        if cmd is None:
  68
+            cmd = popenargs[0]
  69
+        raise subprocess.CalledProcessError(retcode, cmd, output=output)
  70
+    return output
  71
+
  72
+
  73
+# python2.7's subprocess provides a check_output method
  74
+if hasattr(subprocess, 'check_output'):
  75
+    check_output = subprocess.check_output
  76
+else:
  77
+    check_output = _check_output
2  lib/matplotlib/dviread.py
@@ -23,9 +23,9 @@
23 23
 import errno
24 24
 import matplotlib
25 25
 import matplotlib.cbook as mpl_cbook
  26
+from matplotlib.compat import subprocess
26 27
 import numpy as np
27 28
 import struct
28  
-import subprocess
29 29
 import sys
30 30
 
31 31
 if sys.version_info[0] >= 3:
3  lib/matplotlib/font_manager.py
@@ -43,7 +43,7 @@
43 43
             see license/LICENSE_TTFQUERY.
44 44
 """
45 45
 
46  
-import os, sys, subprocess, warnings
  46
+import os, sys, warnings
47 47
 try:
48 48
     set
49 49
 except NameError:
@@ -54,6 +54,7 @@
54 54
 from matplotlib import rcParams, get_configdir
55 55
 from matplotlib.cbook import is_string_like
56 56
 import matplotlib.cbook as cbook
  57
+from matplotlib.compat import subprocess
57 58
 from matplotlib.fontconfig_pattern import \
58 59
     parse_fontconfig_pattern, generate_fontconfig_pattern
59 60
 
2  lib/matplotlib/testing/compare.py
@@ -7,6 +7,7 @@
7 7
 from __future__ import division
8 8
 
9 9
 import matplotlib
  10
+from matplotlib.compat import subprocess
10 11
 from matplotlib.testing.noseclasses import ImageComparisonFailure
11 12
 from matplotlib.testing import image_util, util
12 13
 from matplotlib import _png
@@ -17,7 +18,6 @@
17 18
 import os
18 19
 import numpy as np
19 20
 import shutil
20  
-import subprocess
21 21
 import sys
22 22
 from functools import reduce
23 23
 
2  lib/matplotlib/testing/util.py
... ...
@@ -1,5 +1,5 @@
1  
-import subprocess
2 1
 import sys
  2
+from matplotlib.compat import subprocess
3 3
 
4 4
 
5 5
 class MiniExpect:
2  lib/matplotlib/tests/test_backend_pgf.py
@@ -2,12 +2,12 @@
2 2
 
3 3
 import os
4 4
 import shutil
5  
-import subprocess
6 5
 import numpy as np
7 6
 import nose
8 7
 from nose.plugins.skip import SkipTest
9 8
 import matplotlib as mpl
10 9
 import matplotlib.pyplot as plt
  10
+from matplotlib.compat import subprocess
11 11
 from matplotlib.testing.compare import compare_images, ImageComparisonFailure
12 12
 from matplotlib.testing.decorators import _image_directories
13 13
 
12  lib/matplotlib/texmanager.py
@@ -41,7 +41,6 @@
41 41
 import os
42 42
 import shutil
43 43
 import sys
44  
-from subprocess import Popen, PIPE, STDOUT
45 44
 
46 45
 from hashlib import md5
47 46
 
@@ -51,6 +50,7 @@
51 50
 from matplotlib import rcParams
52 51
 from matplotlib._png import read_png
53 52
 from matplotlib.cbook import mkdirs
  53
+from matplotlib.compat.subprocess import Popen, PIPE, STDOUT
54 54
 import matplotlib.dviread as dviread
55 55
 import re
56 56
 
@@ -63,8 +63,12 @@
63 63
 
64 64
 
65 65
 def dvipng_hack_alpha():
66  
-    p = Popen('dvipng -version', shell=True, stdin=PIPE, stdout=PIPE,
67  
-              stderr=STDOUT, close_fds=(sys.platform != 'win32'))
  66
+    try:
  67
+        p = Popen('dvipng -version', stdin=PIPE, stdout=PIPE, stderr=STDOUT,
  68
+                  close_fds=(sys.platform != 'win32'))
  69
+    except OSError:
  70
+        mpl.verbose.report('No dvipng was found', 'helpful')
  71
+        return False
68 72
     stdin, stdout = p.stdin, p.stdout
69 73
     for line in stdout:
70 74
         if line.startswith(b'dvipng '):
@@ -74,7 +78,7 @@ def dvipng_hack_alpha():
74 78
             version = version.decode('ascii')
75 79
             version = distutils.version.LooseVersion(version)
76 80
             return version < distutils.version.LooseVersion('1.6')
77  
-    mpl.verbose.report('No dvipng was found', 'helpful')
  81
+    mpl.verbose.report('Unexpected response from dvipng -version', 'helpful')
78 82
     return False
79 83
 
80 84
 
0  matplotlib/compat/__init__.py b/lib/matplotlib/compat/__init__.py
No changes.
1  setupext.py
@@ -494,6 +494,7 @@ def get_packages(self):
494 494
             'matplotlib',
495 495
             'matplotlib.backends',
496 496
             'matplotlib.backends.qt4_editor',
  497
+            'matplotlib.compat',
497 498
             'matplotlib.projections',
498 499
             'matplotlib.sphinxext',
499 500
             'matplotlib.testing',
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.