bundled six 1.9.0 causes ImportError: No module named 'winreg' in Pympler

[rathann]

To help us understand and resolve your issue please check that you have provided
the information below.

• Matplotlib version, Python version and Platform (Windows, OSX, Linux ...)

1.5.1, 2.7.11/3.5.1, Fedora rawhide

• How did you install Matplotlib and Python (pip, anaconda, from source ...)

From Fedora package.

• If possible please supply a Short, Self Contained, Correct, Example
  that demonstrates the issue i.e a small piece of code which reproduces the issue
  and can be run with out any other (or as few as possible) external dependencies.

Try building Pympler (http://pythonhosted.org/Pympler/) and running its tests (python3 setup.py test). On armv7hl, I get this exception:

Traceback (most recent call last):
  File "/builddir/build/BUILD/Pympler-0.4.3/test/tracker/test_classtracker.py", line 178, in test_snapshot_members
    self.tracker.create_snapshot(compute_total=True)
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/classtracker.py", line 567, in create_snapshot
    snapshot.asizeof_total = asizeof.asizeof(all=True, code=True)
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/asizeof.py", line 2191, in asizeof
    s = _asizer.asizeof(*t)
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/asizeof.py", line 1783, in asizeof
    s, _ = self._sizes(objs, None)
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/asizeof.py", line 1753, in _sizes
    s[i] = self._sizer(o, 0, sized)
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/asizeof.py", line 1729, in _sizer
    s += z(o, d, None)
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/asizeof.py", line 1729, in _sizer
    s += z(o, d, None)
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/asizeof.py", line 1729, in _sizer
    s += z(o, d, None)
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/asizeof.py", line 1729, in _sizer
    s += z(o, d, None)
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/asizeof.py", line 1728, in _sizer
    for o in r(obj, False):
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/asizeof.py", line 536, in _refs
    for _, o in _dir2(obj, **kwds):
  File "/builddir/build/BUILDROOT/python-Pympler-0.4.3-1.fc25.noarch/usr/lib/python3.5/site-packages/pympler/asizeof.py", line 347, in _dir2
    if hasattr(obj, slots):
  File "/usr/lib/python3.5/site-packages/matplotlib/externals/six.py", line 116, in __getattr__
    _module = self._resolve()
  File "/usr/lib/python3.5/site-packages/matplotlib/externals/six.py", line 113, in _resolve
    return _import_module(self.mod)
  File "/usr/lib/python3.5/site-packages/matplotlib/externals/six.py", line 80, in _import_module
    __import__(name)
ImportError: No module named 'winreg'

The bundled six.py has this:

class _MovedItems(_LazyModule):
    """Lazy loading of moved objects"""
    __path__ = []  # mark as package


_moved_attributes = [
...
    MovedModule("winreg", "_winreg"),
]

while python-six-1.10.0 (latest version) has this:

# Add windows specific modules.
if sys.platform == "win32":
    _moved_attributes += [
        MovedModule("winreg", "_winreg"),
    ]

I applied this patch:

diff -up matplotlib-1.5.1/setupext.py.six matplotlib-1.5.1/setupext.py
--- matplotlib-1.5.1/setupext.py.six    2016-01-10 23:20:20.000000000 +0100
+++ matplotlib-1.5.1/setupext.py    2016-05-18 13:44:21.534494158 +0200
@@ -1026,6 +1026,18 @@ class Qhull(SetupPackage):
             ext.sources.extend(glob.glob('extern/qhull/*.c'))


+class Six(SetupPackage):
+    name = "six"
+
+    def check(self):
+        try:
+            import six
+        except ImportError:
+            return 'not found. pip may install it below.'
+
+        return 'version %s' % six.__version__
+
+
 class TTConv(SetupPackage):
     name = "ttconv"

diff -up matplotlib-1.5.1/setup.py.six matplotlib-1.5.1/setup.py
--- matplotlib-1.5.1/setup.py.six   2016-01-10 23:20:20.000000000 +0100
+++ matplotlib-1.5.1/setup.py   2016-05-18 13:44:21.528494040 +0200
@@ -84,7 +84,7 @@ mpl_packages = [
     setupext.Delaunay(),
     setupext.QhullWrap(),
     setupext.Tri(),
-    setupext.Externals(),
+    setupext.Six(),
     'Optional subpackages',
     setupext.SampleData(),
     setupext.Toolkits(),

and replaced all imports from bundled six to system-wide six without any adverse effects. Please consider unbundling six and using system-wide version.

[QuLogic]

Fedora packages correctly replace internal six with the external one; are you not using those? This directory /usr/lib/python3.5/site-packages/matplotlib/externals/ should not exist in Fedora packages.

[rathann]

Yes, I've just implemented that in the latest build. If you're fine with Fedora carrying a downstream patch, then I won't push any further, but it would be nice to have official support for using system-wide six.

[tacaswell]

Why doesn't this issue show up everywhere else?

The discussion over bundling six (#4501) was protracted and I am not excited about re litigating that.

Might be worth updating to 1.10, but the master branch is going to go python3 only ~18mo so this is not a pressing issue.

A simpler patch downstream might be to replace the contents of external/six.py with

from six import *

[matthew-brett]

As the person who proposed and implemented the bundling - I don't feel strongly - I am happy to reverse it if you like.

[QuLogic]

from six import *

No, six does some complicated monkey-patching; it doesn't work if you do that. I tried to do that on a different package.

[tacaswell]

@matthew-brett What has changed your mind from last year?

[rathann]

@tacaswell :

Why doesn't this issue show up everywhere else?

Probably because I've been able to reproduce it only on armv7hl so far and only while running Pympler testsuite. Debian doesn't package Pympler, for example.

[matthew-brett]

@tacaswell - I think in retrospect I was wrong last year, pip was easily up to to the job, but in any case, after another year of reliable pip, it seems even harder to justify now.

[tacaswell]

if @matthew-brett is on board with reverting this change then 👍 on reverting for 2.0 (would rather not do it for 1.5.2 as that is mostly held for cleaning up the other issues from @rathann at this point).

[jenshnielsen]

Closed by #6556

[megies]

It seems this bug is affecting the matplotlib package in Ubuntu xenial which is a long term stable release, so this is something that can hurt people for years to come.
I'm not sure how to best fix this, as I don't know if Ubuntu packaging team would consider a patch or would prepare an updated package (not sure about their policy for non-security-related bugfixes). Any ideas? See obspy/obspy/issues/1906.

[tacaswell]

I am not convinced that obspy/obspy#1906 is the same issue. This is a bug in the vendored version of six, that issue looks like the OP has crossed installations.