scandir as dependency: UnicodeDecodeError

[jonatanblue]

If I set scandir as a dependency, then installation on a system that does not already have scandir fails with UnicodeDecodeError.

Versions

• OS Ubuntu 16.04
• Python 2.7.12
• setuptools 34.1.0
• pip 9.0.1

Steps to reproduce

Start with a clean environment that does not already have scandir installed. I'm using the Dockerimage ubuntu:16.04, with the following system configuration:

apt-get update && apt-get install python-pip -y
pip install -U pip setuptools

Create a test project and a setup.py file:

from setuptools import setup

setup(
    name = "failr",
    version = "0.0.1",
    install_requires = [
        "scandir",
    ],
)

Run install command:

# python setup.py install
running install
running bdist_egg
running egg_info
creating failr.egg-info
writing requirements to failr.egg-info/requires.txt
writing failr.egg-info/PKG-INFO
writing top-level names to failr.egg-info/top_level.txt
writing dependency_links to failr.egg-info/dependency_links.txt
writing manifest file 'failr.egg-info/SOURCES.txt'
reading manifest file 'failr.egg-info/SOURCES.txt'
writing manifest file 'failr.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib.linux-x86_64-2.7' does not exist -- no Python modules to install

creating build
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying failr.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying failr.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying failr.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying failr.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying failr.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/failr-0.0.1-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing failr-0.0.1-py2.7.egg
Copying failr-0.0.1-py2.7.egg to /usr/local/lib/python2.7/dist-packages
Adding failr 0.0.1 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/failr-0.0.1-py2.7.egg
Processing dependencies for failr==0.0.1
Searching for scandir
Reading https://pypi.python.org/simple/scandir/
Downloading https://pypi.python.org/packages/95/40/ddbcd295ee58d5c1126645890bcf87853e4075547308884e4f8ada27f195/scandir-1.4.tar.gz#md5=bdac33e1548a18598da6c141c84af407
Best match: scandir 1.4
Processing scandir-1.4.tar.gz
Writing /tmp/easy_install-0rkxe4/scandir-1.4/setup.cfg
Running scandir-1.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-0rkxe4/scandir-1.4/egg-dist-tmp-3CReGP
'test/testdir/subdir/unicodƏ.txt' not ANSI_X3.4-1968 encodable -- skipping
'test/testdir/subdir/unidirƏ/file1.txt' not ANSI_X3.4-1968 encodable -- skipping
warning: no files found matching 'README.md'
'test/testdir/subdir/unicodƏ.txt' not ANSI_X3.4-1968 encodable -- skipping
'test/testdir/subdir/unidirƏ/file1.txt' not ANSI_X3.4-1968 encodable -- skipping
zip_safe flag not set; analyzing archive contents...
Moving scandir-1.4-py2.7-linux-x86_64.egg to /usr/local/lib/python2.7/dist-packages
Adding scandir 1.4 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/scandir-1.4-py2.7-linux-x86_64.egg
Traceback (most recent call last):
  File "setup.py", line 7, in <module>
    "scandir",
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.run()
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 411, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 655, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 702, in install_item
    self.process_distribution(spec, dist, deps)
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 747, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 851, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1123, in best_match
    return self.obtain(req, installer)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1135, in obtain
    return installer(requirement)
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 674, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 638, in _tmpdir
    os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
  File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 2232, in rmtree
    return shutil.rmtree(path, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 241, in rmtree
    fullname = os.path.join(path, name)
  File "/usr/lib/python2.7/posixpath.py", line 73, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc6 in position 7: ordinal not in range(128)

Expected outcome

Package is installed.

Actual outcome

Installation fails with UnicodeDecodeError (see log above).

Useful information

If I re-run the setup.py install command above a second time, it works.

If I install scandir with pip (pip install scandir) before running the install command above, it works fine on the first attempt.

The character referenced in the error comes from the file name unicodƏ.txt (also seen in the log). When the join() function in posixpath.py tries to treat it as ASCII to convert to unicode it fails, because Ə is not an ASCII character.

[jaraco]

This looks like another manifestation of #706.

[jaraco]

If I re-run the setup.py install command above a second time, it works.

That's because scandir was successfully installed the first time, but failed at the cleanup stage, so the second attempt finds scandir already installed and bypassed the part that creates the error.

I'm unable to replicate the underlying issue except when unicode is passed to rmtree:

vagrant@vagrant:~$ wget https://files.pythonhosted.org/packages/95/40/ddbcd295ee58d5c1126645890bcf87853e4075547308884e4f8ada27f195/scandir-1.4.tar.gz -q -O - | tar xz
vagrant@vagrant:~$ LANG=C python -c "import shutil; shutil.rmtree(u'scandir-1.4')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 241, in rmtree
    fullname = os.path.join(path, name)
  File "/usr/lib/python2.7/posixpath.py", line 73, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc6 in position 7: ordinal not in range(128)
vagrant@vagrant:~$ wget https://files.pythonhosted.org/packages/95/40/ddbcd295ee58d5c1126645890bcf87853e4075547308884e4f8ada27f195/scandir-1.4.tar.gz -q -O - | tar xz
vagrant@vagrant:~$ LANG=C python -c "import shutil; shutil.rmtree('scandir-1.4')"

This suggests to me that the rmtree_safe isn't being activated in your environment.

And when I test in my local Linux workstation, I find that's the case for me as well:

vagrant@vagrant:~$ LANG=C python2.7 -m rwt -q setuptools -- -c "import setuptools.py27compat as compat, sys, platform; print(compat.rmtree_safe); print(sys.getfilesystemencoding())"
<function <lambda> at 0x7f6e286d85f0>
ANSI_X3.4-1968

rmtree_safe isn't being activated because sys.getfilesystemencoding()=='ascii' evaluates to False.

Maybe I should just take that restriction out and always cast to string on Linux and Python 2.

[jaraco]

Give 34.1.1 a try, releasing momentarily.

[jonatanblue]

Thanks! It now works fine. I tested both version 34.1.1 and 34.2.0.