New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sandbox.run_setup incorrectly sets `__file__` when setup_script is Unicode on Python 2 #712

Closed
sileht opened this Issue Aug 2, 2016 · 7 comments

Comments

Projects
None yet
4 participants
@sileht
Contributor

sileht commented Aug 2, 2016

Hi,

Since setuptools 25.X, we can't build Cython if present in setup_requires list.

that works on py3 or with setuptool 24.X but not with setuptool 25.X and py2:

To reproduce you can do:

$ virtualenv venv
$ venv/bin/easy_install Cython
New python executable in /tmp/cdt.oxE640oU/venv/bin/python
Installing setuptools, pip, wheel...done.
Searching for Cython
Reading https://pypi.python.org/simple/Cython/
Downloading https://pypi.python.org/packages/c6/fe/97319581905de40f1be7015a0ea1bd336a756f6249914b148a17eefa75dc/Cython-0.24.1.tar.gz#md5=890b494a12951f1d6228c416a5789554
Best match: Cython 0.24.1
Processing Cython-0.24.1.tar.gz
Writing /tmp/easy_install-rWIiof/Cython-0.24.1/setup.cfg
Running Cython-0.24.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-rWIiof/Cython-0.24.1/egg-dist-tmp-8xzUmf
Unable to find pgen, not compiling formal grammar.
Traceback (most recent call last):
File "venv/bin/easy_install", line 11, in
sys.exit(main())
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 2271, in main
**kw
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 409, in run
self.easy_install(spec, not self.no_deps)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 664, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 694, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 875, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1114, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1100, in run_setup
run_setup(setup_script, args)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 249, in run_setup
raise
File "/usr/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 197, in setup_context
yield
File "/usr/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 168, in save_modules
saved_exc.resume()
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 143, in resume
six.reraise(type, exc, self._tb)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 156, in save_modules
yield saved
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 197, in setup_context
yield
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 246, in run_setup
DirectorySandbox(setup_dir).run(runner)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 276, in run
return func()
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 245, in runner
_execfile(setup_script, ns)
File "/tmp/cdt.oxE640oU/venv/local/lib/python2.7/site-packages/setuptools/sandbox.py", line 47, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-rWIiof/Cython-0.24.1/setup.py", line 238, in
File "/tmp/easy_install-rWIiof/Cython-0.24.1/setup.py", line 166, in compile_cython_modules
File "/usr/lib/python2.7/distutils/extension.py", line 109, in init
"'sources' must be a list of strings"
AssertionError: 'sources' must be a list of strings

Cheers,

@sileht

This comment has been minimized.

Contributor

sileht commented Aug 2, 2016

From distutils PoV, it received a list of unicode string instead of a list of str string. Cython provides a list str string to setuptools. So I suspect setuptools to convert the list from str to unicode and breaks Cython build.

@sileht

This comment has been minimized.

Contributor

sileht commented Aug 2, 2016

After more testing the regression have been introduced by 25.1.2.

@sileht

This comment has been minimized.

Contributor

sileht commented Aug 2, 2016

This have been introduced by 8579495

sileht added a commit to sileht/setuptools that referenced this issue Aug 2, 2016

Revert "Ensure that tmpdir is unicode. Fixes pypa#704."
This reverts commit 8579495.

As we can see on pypa#709,
this breaks many things (easy_install C extensions, all py3.5 tests,
run with LANG=C).

So instead of fixing in a hurry all new bugs due to this, I propose to
revert this commit until all downsides of this change have been
investigated.

Related bug: pypa#709
Related bug: pypa#710
Related bug: pypa#712
@vallsv

This comment has been minimized.

Contributor

vallsv commented Aug 2, 2016

Hi. Same problem here. On Windows x64, Python 2.7.

I note that the last project release i named 25.1.2 instead of v25.1.2. Maybe this last version (without the v) fix the problem? Yet, i dont try.

@ferrants

This comment has been minimized.

ferrants commented Aug 2, 2016

Not the same exact error, but similar for me with v25.1.2. This happens when we setup a virtualenv, which grabs the latest setuptools. The error is related to Unicode in a dependent package:

Running pylint-1.5.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-zjeZ02/pylint-1.5.0/egg-dist-tmp-jXiSCj
warning: no files found matching '*.html' under directory 'doc'
Traceback (most recent call last):
  File "setup.py", line 79, in <module>
    version='0.2.14'
  File "/usr/local/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.run()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 409, in run
    self.easy_install(spec, not self.no_deps)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 645, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 696, in install_item
    self.process_distribution(spec, dist, deps)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 741, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 826, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1098, in best_match
    return self.obtain(req, installer)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1110, in obtain
    return installer(requirement)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 664, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 694, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 875, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1114, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1100, in run_setup
    run_setup(setup_script, args)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 249, in run_setup
    raise
  File "/usr/local/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 197, in setup_context
    yield
  File "/usr/local/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 168, in save_modules
    saved_exc.resume()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 143, in resume
    six.reraise(type, exc, self._tb)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 156, in save_modules
    yield saved
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 197, in setup_context
    yield
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 246, in run_setup
    DirectorySandbox(setup_dir).run(runner)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 276, in run
    return func()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 245, in runner
    _execfile(setup_script, ns)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/sandbox.py", line 47, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-zjeZ02/pylint-1.5.0/setup.py", line 178, in <module>

  File "/tmp/easy_install-zjeZ02/pylint-1.5.0/setup.py", line 175, in install

  File "/usr/local/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/bdist_egg.py", line 161, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/bdist_egg.py", line 147, in call_command
    self.run_command(cmdname)
  File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/tmp/easy_install-zjeZ02/pylint-1.5.0/setup.py", line 105, in run

  File "/var/lib/jenkins/workspace/tools_test-jenkins-tools-install/my_env/lib/python2.7/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/usr/local/lib/python2.7/distutils/command/install_lib.py", line 109, in build
    self.run_command('build_py')
  File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python2.7/distutils/command/build_py.py", line 93, in run
    self.build_packages()
  File "/usr/local/lib/python2.7/distutils/command/build_py.py", line 372, in build_packages
    self.build_module(module, module_file, package)
  File "/usr/local/lib/python2.7/distutils/command/build_py.py", line 333, in build_module
    "'package' must be a string (dot-separated), list, or tuple")
TypeError: 'package' must be a string (dot-separated), list, or tuple

We're doing virtualenv --no-download until we have a resolution for this

jaraco added a commit that referenced this issue Dec 18, 2016

@jaraco

This comment has been minimized.

Member

jaraco commented Dec 18, 2016

I traced the issue in the OP, and it seems to me here's what's happening:

Presumably, this issue might also occur if distutils were invoked from a directory containing non-ascii characters. I'll test that theory.

@jaraco

This comment has been minimized.

Member

jaraco commented Dec 18, 2016

On my mac, here's what I get:

$ mkdir ☃
$ cat > ☃/test.py
print repr(__file__)
$ python2.7 ☃/test.py
'\xe2\x98\x83/test.py'

On my Windows box, I see this instead:

C:\Users\jaraco> mkdir ☃


    Directory: C:\Users\jaraco


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----      18-Dec-2016     10:21                ☃


C:\Users\jaraco> notepad ☃/test.py
C:\Users\jaraco> py -2 ☃/test.py
C:\Program Files\Python27\python.exe: can't open file '?/test.py': [Errno 22] Invalid argument

In both cases, it seems Python is setting __file__ to a str, so setuptools should probably be doing the same.

Encoding with sys.getfilesystemencoding() seems to be the right thing to do to be consistent with what the interpreter does:

C:\Users\jaraco> py -2
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getfilesystemencoding()
'mbcs'
>>> u'\u2603'
u'\u2603'
>>> u'\u2603'.encode(sys.getfilesystemencoding())
'?'
$ python2.7
Python 2.7.12 (default, Dec 18 2016, 09:24:19) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> u'\u2603'.encode(sys.getfilesystemencoding())
'\xe2\x98\x83'

@jaraco jaraco changed the title from since setuptools 25 setup(setup_requires=["Cython"]) won't work anymore to sandbox.run_setup incorrectly sets `__file__` when setup_script is Unicode on Python 2 Dec 18, 2016

@jaraco jaraco closed this in cd2a519 Dec 18, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment