TypeError: 'output_dir' must be a string or None since setuptools 40.7.0 #1653

ghost · 2019-01-28T01:12:21Z

We have a tricky regression in Gentoo with setuptools release 40.7.0. This affects many (all?) packages that involve build_ext. These errors aren't present in 40.6.3. I think I managed to have reproducible steps:

git clone https://github.com/python-pillow/Pillow.git
cd Pillow
virtualenv -p python2 env
. env/bin/activate
pip install -U setuptools
echo -e "[build]\nbuild-base = /tmp/foo" >> setup.cfg
mkdir /tmp/foo
python setup.py build
[...]
Traceback (most recent call last):
  File "setup.py", line 793, in <module>
    zip_safe=not (debug_build() or PLATFORM_MINGW), )
  File "/home/vdupras/src/Pillow/env/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.7/distutils/command/build.py", line 127, in run
    self.run_command(cmd_name)
  File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "setup.py", line 695, in build_extensions
    build_ext.build_extensions(self)
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 449, in build_extensions
    self.build_extension(ext)
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 499, in build_extension
    depends=ext.depends)
  File "/home/vdupras/src/Pillow/mp_compile.py", line 40, in _mp_compile
    output_dir, macros, include_dirs, sources, depends, extra_postargs)
  File "/usr/lib64/python2.7/distutils/ccompiler.py", line 329, in _setup_compile
    raise TypeError, "'output_dir' must be a string or None"
TypeError: 'output_dir' must be a string or None

Can you reproduce on your side?

It the fix isn't obvious, I can try myself on a PR soon.

(by the way, this is due to distutils's isinstance() call being made on str rather than basestring and only happens under py2. output_dir is unicode when it should be str)

The text was updated successfully, but these errors were encountered:

benoit-pierre · 2019-01-28T09:25:53Z

This is probably due to #1180.

reinout · 2019-01-28T13:27:45Z

I think something similar is the cause of a zc.buildout test error. It only fails on 2.7 (pypy and python 3 are file) and it happens when we test a package with an extension.

It has to do with include_dirs instead of output_dir, but it sounds similar enough that the traceback (from travis-ci) could help:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 2316, in main
**kw
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
  File ".../python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
  File ".../python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
  File ".../python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 418, in run
self.easy_install(spec, not self.no_deps)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 660, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 705, in install_item
dists = self.install_eggs(spec, download, tmpdir)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 890, in install_eggs
return self.build_and_install(setup_script, setup_base)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 1158, in build_and_install
self.run_setup(setup_script, setup_base, args)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/easy_install.py", line 1144, in run_setup
run_setup(setup_script, args)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 253, in run_setup
raise
  File ".../python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
yield
  File ".../python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 166, in save_modules
saved_exc.resume()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 141, in resume
six.reraise(type, exc, self._tb)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 154, in save_modules
yield saved
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
yield
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 250, in run_setup
_execfile(setup_script, ns)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/sandbox.py", line 45, in _execfile
exec(code, globals, locals)
  File "/tmp/tmpc2rzJKbuild/extdemo-1.5/setup.py", line 10, in <module>
ext_modules = [Extension('extdemo', ['extdemo.c'])],
  File ".../python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
  File ".../python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
  File ".../python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/bdist_egg.py", line 163, in run
self.run_command("egg_info")
  File ".../python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
  File ".../python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/egg_info.py", line 296, in run
self.find_sources()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/egg_info.py", line 303, in find_sources
mm.run()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/egg_info.py", line 534, in run
self.add_defaults()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/egg_info.py", line 570, in add_defaults
sdist.add_defaults(self)
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/py36compat.py", line 36, in add_defaults
self._add_defaults_ext()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
  File ".../python2.7/distutils/cmd.py", line 312, in get_finalized_command
cmd_obj.ensure_finalized()
  File ".../python2.7/distutils/cmd.py", line 109, in ensure_finalized
self.finalize_options()
  File ".../setuptools-40.7.0-py2.7.egg/setuptools/command/build_ext.py", line 133, in finalize_options
_build_ext.finalize_options(self)
  File ".../python2.7/distutils/command/build_ext.py", line 159, in finalize_options
self.include_dirs.append(py_include)
AttributeError: 'unicode' object has no attribute 'append'

self.include_dirs apparently is a unicode object instead of a list.

I haven't myself contributed to that part of the buildout tests, but from looking at it, we're passing the result of an os.path.join() as include_dirs in a dict to setuptools.command.setopt.edit_config(). (I don't know what happens in there, though).

reinout · 2019-01-28T13:59:47Z

I've looked at my local 2.7 distutils, and in distutils/command/build_ext.py around line 150 I see something like this:

    if self.include_dirs is None:
        self.include_dirs = self.distribution.include_dirs or []
    if isinstance(self.include_dirs, str):
        self.include_dirs = self.include_dirs.split(os.pathsep)

That self.distribution.include_dirs is probably the os.path.join() output buildout puts into include_dirs.
The isinstance normally detected this as str and called "split", turning it into the (expected) list.

Now the value is a unicode instance, so this is another case where string_types should be used.

I did a bit of grepping, and there are some 25 cases where I see isinstance(something, str). #1180 didn't get all of them. Note that it will be a pain to fix them all with monkey patching...

Brainstorming: would it be an idea to stick with old-style strings on 2.7? I don't know exactly what #1180 fixes, but perhaps encoding it as byte strings again (only on 2.7) could be OK?

ghost · 2019-01-28T14:05:00Z

Or revert the change, postpone it for v41 and make that release py3-only? isittimeyet?

bivald · 2019-01-28T16:11:46Z

I'm seeing this for pypy as well, installing packages such as grpcio

jaraco · 2019-01-28T21:54:46Z

I don't yet understand which part of the change in #1180 led to these failures. Reading through the code, it seems to me the only place that unicode would appear where previously str would be present is in the .cfg file parsing, and I don't see those config values set in setup.cfg... unless maybe the build tools are setting those values. Yes, that must be it...

Confirmed. As can be seen from the trace, the following is being set in setup.cfg:

[build]
build-base=/tmp/foo

And tracing the distutils code, I can see where output_dir is derived from build_temp and build_temp is derived from build_base.

.

jaraco · 2019-01-28T22:21:34Z

In the aforementioned commit, I have what I think is a suitable workaround. @hsoft or @reinout Can you possibly test with setuptools from https://m.devpi.net/jaraco/dev/setuptools/40.7.0.post20190128 and see if your builds work with that code present?

rouge8 · 2019-01-28T22:38:05Z

Confirmed that https://m.devpi.net/jaraco/dev/setuptools/40.7.0.post20190128 fixed the building an affected package for me on OS X with Python 2.7

gnossen · 2019-01-28T22:44:31Z

@jaraco I can confirm that the artifact linked above fixes the regression for grpc.

ghost · 2019-01-28T22:53:38Z

Yes, it fixes the problem demonstrated by the steps written in my initial comment.

jaraco · 2019-01-29T00:21:45Z

v40.7.1 is released now with the fix. Thanks for the report and sorry for any inconvenience.

reinout · 2019-01-29T08:18:55Z

Yeah, also the buildout tests run fine again.

@jaraco: nice fix!

Setuptools 40.7.0 introduced support for non-ascii chars in setup.cfg [1], which causes python2.7 based sphinx jobs to fail with str-unicode mismatch, e.g. with [2]: 'error: 'builder' must be a list of strings (got u'html,man') See issue [3]. [1] https://setuptools.readthedocs.io/en/latest/history.html#v40-7-0 [2] http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/keystone/stable/queens/build-openstack-sphinx-docs/0ea4957/job-output.txt.gz#_2019-01-28_06_18_25_401045 [3] pypa/setuptools#1653 Change-Id: I8921639cc4a1edfa0ac9d46a7cc03549aa6189aa

Setuptools 40.7.0 introduced support for non-ascii chars in setup.cfg [1], which causes python2.7 based sphinx jobs to fail with str-unicode mismatch, e.g. with [2]: 'error: 'builder' must be a list of strings (got u'html,man') See issue [3]. Conflicts: global-requirements.txt [1] https://setuptools.readthedocs.io/en/latest/history.html#v40-7-0 [2] http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/keystone/stable/queens/build-openstack-sphinx-docs/0ea4957/job-output.txt.gz#_2019-01-28_06_18_25_401045 [3] pypa/setuptools#1653 Change-Id: I8921639cc4a1edfa0ac9d46a7cc03549aa6189aa (cherry picked from commit 5989f40) (cherry picked from commit 71f6094)

Setuptools 40.7.0 introduced support for non-ascii chars in setup.cfg [1], which causes python2.7 based sphinx jobs to fail with str-unicode mismatch, e.g. with [2]: 'error: 'builder' must be a list of strings (got u'html,man') See issue [3]. Conflicts: global-requirements.txt [1] https://setuptools.readthedocs.io/en/latest/history.html#v40-7-0 [2] http://logs.openstack.org/periodic-stable/git.openstack.org/openstack/keystone/stable/queens/build-openstack-sphinx-docs/0ea4957/job-output.txt.gz#_2019-01-28_06_18_25_401045 [3] pypa/setuptools#1653 Change-Id: I8921639cc4a1edfa0ac9d46a7cc03549aa6189aa (cherry picked from commit 5989f40)

reinout mentioned this issue Jan 28, 2019

document query command buildout/buildout#472

Merged

This was referenced Jan 28, 2019

40.7.0 fails with "AttributeError: 'unicode' object has no attribute 'compiler_type'" #1655

Closed

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-t_utKw/pyodbc/ #1654

Closed

scollazo mentioned this issue Jan 28, 2019

Problem: pip problems when deploying SS with ansible archivematica/Issues#455

Closed

benoit-pierre mentioned this issue Jan 28, 2019

Revert "improve encoding handling for setup.cfg" #1656

Closed

gnossen mentioned this issue Jan 28, 2019

[Python install] "Failed building wheel for grpcio" when installing on Raspberry Pi Zero W grpc/grpc#17829

Closed

jaraco added a commit that referenced this issue Jan 28, 2019

Prefer native strings on Python 2 when reading config files. Fixes #1653

6e3550c

.

jaraco mentioned this issue Jan 28, 2019

Prefer native strings on Python 2 when reading config files. #1660

Merged

2 tasks

jaraco closed this as completed in 9150b6b Jan 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TypeError: 'output_dir' must be a string or None since setuptools 40.7.0 #1653

TypeError: 'output_dir' must be a string or None since setuptools 40.7.0 #1653

ghost commented Jan 28, 2019 •

edited by ghost

Loading

benoit-pierre commented Jan 28, 2019

reinout commented Jan 28, 2019

reinout commented Jan 28, 2019

ghost commented Jan 28, 2019

bivald commented Jan 28, 2019

jaraco commented Jan 28, 2019

jaraco commented Jan 28, 2019

rouge8 commented Jan 28, 2019

gnossen commented Jan 28, 2019

ghost commented Jan 28, 2019

jaraco commented Jan 29, 2019

reinout commented Jan 29, 2019

TypeError: 'output_dir' must be a string or None since setuptools 40.7.0 #1653

TypeError: 'output_dir' must be a string or None since setuptools 40.7.0 #1653

Comments

ghost commented Jan 28, 2019 • edited by ghost Loading

benoit-pierre commented Jan 28, 2019

reinout commented Jan 28, 2019

reinout commented Jan 28, 2019

ghost commented Jan 28, 2019

bivald commented Jan 28, 2019

jaraco commented Jan 28, 2019

jaraco commented Jan 28, 2019

rouge8 commented Jan 28, 2019

gnossen commented Jan 28, 2019

ghost commented Jan 28, 2019

jaraco commented Jan 29, 2019

reinout commented Jan 29, 2019

ghost commented Jan 28, 2019 •

edited by ghost

Loading