Tests failing with latest backports.unittest_mock

[jaraco]

Following the report in jaraco/backports.functools_lru_cache#9, I went through the six backports modules I maintain and updated them accordingly. This effort led to the release of backports.unittest_mock 1.3, the presence of which is triggering three new test failures on Python 2.6 and pypy. Pinning to backports.unittest_mock 1.2 works around the issue.

[jaraco]

Looking first at the two failures causing a SandboxViolation, the traceback looks like this:

______________________ TestSetupRequires.test_setup_requires_override_nspkg ______________________

self = <setuptools.tests.test_easy_install.TestSetupRequires instance at 0x1022ab560>

    def test_setup_requires_override_nspkg(self):
        """
            Like ``test_setup_requires_overrides_version_conflict`` but where the
            ``setup_requires`` package is part of a namespace package that has
            *already* been imported.
            """
    
        with contexts.save_pkg_resources_state():
            with contexts.tempdir() as temp_dir:
                foobar_1_archive = os.path.join(temp_dir, 'foo.bar-0.1.tar.gz')
                make_nspkg_sdist(foobar_1_archive, 'foo.bar', '0.1')
                # Now actually go ahead an extract to the temp dir and add the
                # extracted path to sys.path so foo.bar v0.1 is importable
                foobar_1_dir = os.path.join(temp_dir, 'foo.bar-0.1')
                os.mkdir(foobar_1_dir)
                with tarfile_open(foobar_1_archive) as tf:
                    tf.extractall(foobar_1_dir)
                sys.path.insert(1, foobar_1_dir)
    
                dist = PRDistribution(foobar_1_dir, project_name='foo.bar',
                                      version='0.1')
                working_set.add(dist)
    
                template = DALS("""\
                        import foo  # Even with foo imported first the
                                    # setup_requires package should override
                        import setuptools
                        setuptools.setup(**%r)
    
                        if not (hasattr(foo, '__path__') and
                                len(foo.__path__) == 2):
                            print('FAIL')
    
                        if 'foo.bar-0.2' not in foo.__path__[0]:
                            print('FAIL')
                    """)
    
                test_pkg = create_setup_requires_package(
                    temp_dir, 'foo.bar', '0.2', make_nspkg_sdist, template)
    
                test_setup_py = os.path.join(test_pkg, 'setup.py')
    
                with contexts.quiet() as (stdout, stderr):
                    try:
                        # Don't even need to install the package, just
                        # running the setup.py at all is sufficient
>                       run_setup(test_setup_py, ['--name'])

setuptools/tests/test_easy_install.py:455: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
setuptools/sandbox.py:258: in run_setup
    raise
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/contextlib.py:34: in __exit__
    self.gen.throw(type, value, traceback)
setuptools/sandbox.py:198: in setup_context
    yield
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/contextlib.py:34: in __exit__
    self.gen.throw(type, value, traceback)
setuptools/sandbox.py:169: in save_modules
    saved_exc.resume()
setuptools/sandbox.py:144: in resume
    six.reraise(type, exc, self._tb)
setuptools/sandbox.py:157: in save_modules
    yield saved
setuptools/sandbox.py:198: in setup_context
    yield
setuptools/sandbox.py:255: in run_setup
    DirectorySandbox(setup_dir).run(runner)
setuptools/sandbox.py:285: in run
    return func()
setuptools/sandbox.py:253: in runner
    _execfile(setup_script, ns)
setuptools/sandbox.py:47: in _execfile
    exec(code, globals, locals)
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpcKmHT5/test_pkg/setup.py:4: in <module>
    ???
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py:113: in setup
    _setup_distribution = dist = klass(attrs)
setuptools/dist.py:318: in __init__
    self.fetch_build_eggs(attrs['setup_requires'])
setuptools/dist.py:375: in fetch_build_eggs
    replace_conflicting=True,
pkg_resources/__init__.py:851: in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
pkg_resources/__init__.py:1123: in best_match
    return self.obtain(req, installer)
pkg_resources/__init__.py:1135: in obtain
    return installer(requirement)
setuptools/dist.py:419: in fetch_build_egg
    from setuptools.command.easy_install import easy_install
setuptools/command/easy_install.py:52: in <module>
    from setuptools.package_index import (
setuptools/package_index.py:26: in <module>
    from setuptools import ssl_support
setuptools/ssl_support.py:45: in <module>
    from backports.ssl_match_hostname import CertificateError
.tox/py26/lib/python2.6/site-packages/_pytest/assertion/rewrite.py:115: in find_module
    os.mkdir(cache_dir)
setuptools/sandbox.py:312: in wrap
    path = self._remap_input(name, path, *args, **kw)
setuptools/sandbox.py:454: in _remap_input
    self._violation(operation, os.path.realpath(path), *args, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <setuptools.sandbox.DirectorySandbox instance at 0x10157a290>, operation = 'mkdir'
args = ('/Users/jaraco/Dropbox/code/main/setuptools/.tox/py26/lib/python2.6/site-packages/backports/__pycache__',)
kw = {}, SandboxViolation = <class 'setuptools.sandbox.SandboxViolation'>

    def _violation(self, operation, *args, **kw):
        from setuptools.sandbox import SandboxViolation
>       raise SandboxViolation(operation, args, kw)
E       SandboxViolation: SandboxViolation: mkdir('/Users/jaraco/Dropbox/code/main/setuptools/.tox/py26/lib/python2.6/site-packages/backports/__pycache__',) {}
E       
E       The package setup script has attempted to modify files on your system
E       that are not within the EasyInstall build area, and has been aborted.
E       
E       This package cannot be safely installed by EasyInstall, and may not
E       support alternate installation locations even if you run its setup
E       script by hand.  Please inform the package's author and the EasyInstall
E       maintainers to find out if a fix or workaround is available.

setuptools/sandbox.py:409: SandboxViolation

The first thing I notice is that the attempted import is backports.ssl_match_hostname:

setuptools/ssl_support.py:45: in <module>
    from backports.ssl_match_hostname import CertificateError
.tox/py26/lib/python2.6/site-packages/_pytest/assertion/rewrite.py:115: in find_module
    os.mkdir(cache_dir)

And that package isn't installed, so I would expect that call to raise an ImportError, but before that happens, the pytest assertion rewrite triggers a sandbox violation.

[jaraco]

The fact that code is being reached by the rewrite logic suggests that pytest thinks the module is a module under test requiring rewriting.

[jaraco]

The issue seems to be related to pytest thinking backports must be rewritten.

-> raise SandboxViolation(operation, args, kw)
(Pdb) u 
> /Users/jaraco/Dropbox/code/main/setuptools/setuptools/sandbox.py(454)_remap_input()
-> self._violation(operation, os.path.realpath(path), *args, **kw)
(Pdb) u
> /Users/jaraco/Dropbox/code/main/setuptools/setuptools/sandbox.py(312)wrap()
-> path = self._remap_input(name, path, *args, **kw)
(Pdb) u
> /Users/jaraco/Dropbox/code/main/setuptools/.tox/py26/lib/python2.6/site-packages/_pytest/assertion/rewrite.py(115)find_module()
-> os.mkdir(cache_dir)
(Pdb) name
'backports.ssl_match_hostname'
(Pdb) fn
'/Users/jaraco/Dropbox/code/main/setuptools/.tox/py26/lib/python2.6/site-packages/backports/ssl_match_hostname.py'
(Pdb) self._should_rewrite(name, fn_pypath, state)
True
(Pdb) self._must_rewrite
set(['setuptools.tests.fixtures', 'pytest_flake8', 'backports'])

[jaraco]

Backports ends up there when the backports.unittest_mock pytest11 entry point is loaded and considered for importhook, but the same thing happens on the other Pythons where the test isn't failing.

[jaraco]

Aah. The reason the other tests aren't failing is because only on Python 2.6 and (presumably) pypy as found on Travis does the fallback to backports.ssl_match_hostname happen in ssl_support. On Python 2.7 and later, the imports happen from the ssl module without error.

[jaraco]

So then the question becomes - why with backports.unittest_mock 1.2 does the issue not occur? With 1.2, the backports module doesn't appear in the AssertionRewritingHook._must_rewrite, probably because backports was declared as a namespace_package in setup.py, but with 1.3 it is not.

This leads me to believe that any package that installs backports.unittest_mock and imports any backports module in its tests or under its tests will fail with this same error. So then the question becomes - what is pytest doing and should backports be appearing in the must_rewrite? I'll file a ticket upstream for help.

[jaraco]

any package that installs backports.unittest_mock and imports any backports module in its tests or under its tests will fail with this same error

I was wrong. I tried to replicate the issue in a clean environment and could not. That's because there's a third factor - the sandboxing logic.

[jaraco]

Setuptools is getting out of the business of managing environments, so the sandboxing logic should be obsolete soon. I'm closing this.