sandbox.run_setup incorrectly sets `__file__` when setup_script is Unicode on Python 2

[sileht]

Hi,

Since setuptools 25.X, we can't build Cython if present in setup_requires list.

that works on py3 or with setuptool 24.X but not with setuptool 25.X and py2:

To reproduce you can do:

$ virtualenv venv
$ venv/bin/easy_install Cython
New python executable in /tmp/cdt.oxE640oU/venv/bin/python
Installing setuptools, pip, wheel...done.
Searching for Cython
Reading https://pypi.python.org/simple/Cython/
Downloading https://pypi.python.org/packages/c6/fe/97319581905de40f1be7015a0ea1bd336a756f6249914b148a17eefa75dc/Cython-0.24.1.tar.gz#md5=890b494a12951f1d6228c416a5789554
Best match: Cython 0.24.1
Processing Cython-0.24.1.tar.gz
Writing /tmp/easy_install-rWIiof/Cython-0.24.1/setup.cfg
Running Cython-0.24.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-rWIiof/Cython-0.24.1/egg-dist-tmp-8xzUmf
Unable to find pgen, not compiling formal grammar.
Traceback (most recent call last):
File "venv/bin/easy_install", line 11, in
sys.exit(main())
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 2271, in main
**kw
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 409, in run
self.easy_install(spec, not self.no_deps)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 664, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 694, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 875, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1114, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1100, in run_setup
run_setup(setup_script, args)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 249, in run_setup
raise
File "/usr/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 197, in setup_context
yield
File "/usr/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 168, in save_modules
saved_exc.resume()
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 143, in resume
six.reraise(type, exc, self._tb)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 156, in save_modules
yield saved
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 197, in setup_context
yield
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 246, in run_setup
DirectorySandbox(setup_dir).run(runner)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 276, in run
return func()
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 245, in runner
_execfile(setup_script, ns)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 47, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-rWIiof/Cython-0.24.1/setup.py", line 238, in
File "/tmp/easy_install-rWIiof/Cython-0.24.1/setup.py", line 166, in compile_cython_modules
File "/usr/lib/python2.7/distutils/extension.py", line 109, in init
"'sources' must be a list of strings"
AssertionError: 'sources' must be a list of strings

Cheers,

[sileht]

From distutils PoV, it received a list of unicode string instead of a list of str string. Cython provides a list str string to setuptools. So I suspect setuptools to convert the list from str to unicode and breaks Cython build.

[sileht]

After more testing the regression have been introduced by 25.1.2.

[sileht]

This have been introduced by 8579495

[vallsv]

Hi. Same problem here. On Windows x64, Python 2.7.

I note that the last project release i named 25.1.2 instead of v25.1.2. Maybe this last version (without the v) fix the problem? Yet, i dont try.

[ferrants]

Not the same exact error, but similar for me with v25.1.2. This happens when we setup a virtualenv, which grabs the latest setuptools. The error is related to Unicode in a dependent package:

Running pylint-1.5.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-zjeZ02/pylint-1.5.0/egg-dist-tmp-jXiSCj
warning: no files found matching '*.html' under directory 'doc'
Traceback (most recent call last):
  File "setup.py", line 79, in <module>
    version='0.2.14'
  File "/usr/local/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.run()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 409, in run
    self.easy_install(spec, not self.no_deps)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 645, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 696, in install_item
    self.process_distribution(spec, dist, deps)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 741, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 826, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1098, in best_match
    return self.obtain(req, installer)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1110, in obtain
    return installer(requirement)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 664, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 694, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 875, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1114, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1100, in run_setup
    run_setup(setup_script, args)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 249, in run_setup
    raise
  File "/usr/local/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 197, in setup_context
    yield
  File "/usr/local/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 168, in save_modules
    saved_exc.resume()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 143, in resume
    six.reraise(type, exc, self._tb)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 156, in save_modules
    yield saved
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 197, in setup_context
    yield
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 246, in run_setup
    DirectorySandbox(setup_dir).run(runner)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 276, in run
    return func()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 245, in runner
    _execfile(setup_script, ns)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 47, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-zjeZ02/pylint-1.5.0/setup.py", line 178, in <module>

  File "/tmp/easy_install-zjeZ02/pylint-1.5.0/setup.py", line 175, in install

  File "/usr/local/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/bdist_egg.py", line 161, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/bdist_egg.py", line 147, in call_command
    self.run_command(cmdname)
  File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/tmp/easy_install-zjeZ02/pylint-1.5.0/setup.py", line 105, in run

  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/usr/local/lib/python2.7/distutils/command/install_lib.py", line 109, in build
    self.run_command('build_py')
  File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python2.7/distutils/command/build_py.py", line 93, in run
    self.build_packages()
  File "/usr/local/lib/python2.7/distutils/command/build_py.py", line 372, in build_packages
    self.build_module(module, module_file, package)
  File "/usr/local/lib/python2.7/distutils/command/build_py.py", line 333, in build_module
    "'package' must be a string (dot-separated), list, or tuple")
TypeError: 'package' must be a string (dot-separated), list, or tuple

We're doing virtualenv --no-download until we have a resolution for this

[jaraco]

I traced the issue in the OP, and it seems to me here's what's happening:

• easy_install is creating a temporary directory in which to extract the tarball.
• sandbox invokes setup.py with __file__ set to the setup script
• Cython uses __file__ to construct arguments to Extension.
• If the temporary directory was created as a unicode string, so also will be the arguments to Extension.

Presumably, this issue might also occur if distutils were invoked from a directory containing non-ascii characters. I'll test that theory.

[jaraco]

On my mac, here's what I get:

$ mkdir ☃
$ cat > ☃/test.py
print repr(__file__)
$ python2.7 ☃/test.py
'\xe2\x98\x83/test.py'

On my Windows box, I see this instead:

C:\Users\jaraco> mkdir ☃


    Directory: C:\Users\jaraco


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----      18-Dec-2016     10:21                ☃


C:\Users\jaraco> notepad ☃/test.py
C:\Users\jaraco> py -2 ☃/test.py
C:\Program Files\Python27\python.exe: can't open file '?/test.py': [Errno 22] Invalid argument

In both cases, it seems Python is setting __file__ to a str, so setuptools should probably be doing the same.

Encoding with sys.getfilesystemencoding() seems to be the right thing to do to be consistent with what the interpreter does:

C:\Users\jaraco> py -2
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getfilesystemencoding()
'mbcs'
>>> u'\u2603'
u'\u2603'
>>> u'\u2603'.encode(sys.getfilesystemencoding())
'?'

$ python2.7
Python 2.7.12 (default, Dec 18 2016, 09:24:19) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> u'\u2603'.encode(sys.getfilesystemencoding())
'\xe2\x98\x83'