Installation of packages fails after 28.7.0 upgrade: Documentation object is not iterable #833

nicpottier · 2016-10-28T19:15:48Z

Getting a weird error: TypeError: 'Documentation' object is not iterable trying to install rjsmin in a brand new virtualenv with the latest setup tools. This is on Ubuntu 16.0.4 and my OS X machine does the same thing.

ubuntu@staging:~$ virtualenv env
New python executable in /home/ubuntu/env/bin/python
Installing setuptools, pip, wheel...done.
ubuntu@staging:~$ source env/bin/activate
(env) ubuntu@staging:~$ pip install rjsmin
Collecting rjsmin
  Using cached rjsmin-1.0.12.tar.gz
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/rjsmin.egg-info
    writing pip-egg-info/rjsmin.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/rjsmin.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/rjsmin.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/rjsmin.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-EeZDjY/rjsmin/setup.py", line 42, in <module>
        setup()
      File "/tmp/pip-build-EeZDjY/rjsmin/setup.py", line 33, in setup
        return run(script_args=args, ext=ext, manifest_only=_manifest)
      File "_setup/py2/setup.py", line 421, in run
        return _core.setup(**kwargs)
      File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 279, in run
        self.find_sources()
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
        mm.run()
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 533, in run
        self.add_defaults()
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 562, in add_defaults
        sdist.add_defaults(self)
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 35, in add_defaults
        self._add_defaults_data_files()
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 111, in _add_defaults_data_files
        dirname, filenames = item
    TypeError: 'Documentation' object is not iterable

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-EeZDjY/rjsmin/
(env) ubuntu@staging:~$ pip show setuptools

---
Metadata-Version: 2.0
Name: setuptools
Version: 28.7.0
Summary: Easily download, build, install, upgrade, and uninstall Python packages
Home-page: https://github.com/pypa/setuptools
Author: Python Packaging Authority
Author-email: distutils-sig@python.org
Installer: pip
License: UNKNOWN
Location: /home/ubuntu/env/lib/python2.7/site-packages
Requires: 
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  License :: OSI Approved :: MIT License
  Operating System :: OS Independent
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4
  Programming Language :: Python :: 3.5
  Topic :: Software Development :: Libraries :: Python Modules
  Topic :: System :: Archiving :: Packaging
  Topic :: System :: Systems Administration
  Topic :: Utilities
Entry-points:
  [console_scripts]
  easy_install = setuptools.command.easy_install:main
  easy_install-3.5 = setuptools.command.easy_install:main
  [distutils.commands]
  alias = setuptools.command.alias:alias
  bdist_egg = setuptools.command.bdist_egg:bdist_egg
  bdist_rpm = setuptools.command.bdist_rpm:bdist_rpm
  bdist_wininst = setuptools.command.bdist_wininst:bdist_wininst
  build_ext = setuptools.command.build_ext:build_ext
  build_py = setuptools.command.build_py:build_py
  develop = setuptools.command.develop:develop
  easy_install = setuptools.command.easy_install:easy_install
  egg_info = setuptools.command.egg_info:egg_info
  install = setuptools.command.install:install
  install_egg_info = setuptools.command.install_egg_info:install_egg_info
  install_lib = setuptools.command.install_lib:install_lib
  install_scripts = setuptools.command.install_scripts:install_scripts
  register = setuptools.command.register:register
  rotate = setuptools.command.rotate:rotate
  saveopts = setuptools.command.saveopts:saveopts
  sdist = setuptools.command.sdist:sdist
  setopt = setuptools.command.setopt:setopt
  test = setuptools.command.test:test
  upload = setuptools.command.upload:upload
  upload_docs = setuptools.command.upload_docs:upload_docs
  [distutils.setup_keywords]
  convert_2to3_doctests = setuptools.dist:assert_string_list
  dependency_links = setuptools.dist:assert_string_list
  eager_resources = setuptools.dist:assert_string_list
  entry_points = setuptools.dist:check_entry_points
  exclude_package_data = setuptools.dist:check_package_data
  extras_require = setuptools.dist:check_extras
  include_package_data = setuptools.dist:assert_bool
  install_requires = setuptools.dist:check_requirements
  namespace_packages = setuptools.dist:check_nsp
  package_data = setuptools.dist:check_package_data
  packages = setuptools.dist:check_packages
  python_requires = setuptools.dist:check_specifier
  setup_requires = setuptools.dist:check_requirements
  test_loader = setuptools.dist:check_importable
  test_runner = setuptools.dist:check_importable
  test_suite = setuptools.dist:check_test_suite
  tests_require = setuptools.dist:check_requirements
  use_2to3 = setuptools.dist:assert_bool
  use_2to3_exclude_fixers = setuptools.dist:assert_string_list
  use_2to3_fixers = setuptools.dist:assert_string_list
  zip_safe = setuptools.dist:assert_bool
  [egg_info.writers]
  PKG-INFO = setuptools.command.egg_info:write_pkg_info
  dependency_links.txt = setuptools.command.egg_info:overwrite_arg
  depends.txt = setuptools.command.egg_info:warn_depends_obsolete
  eager_resources.txt = setuptools.command.egg_info:overwrite_arg
  entry_points.txt = setuptools.command.egg_info:write_entries
  namespace_packages.txt = setuptools.command.egg_info:overwrite_arg
  requires.txt = setuptools.command.egg_info:write_requirements
  top_level.txt = setuptools.command.egg_info:write_toplevel_names
  [setuptools.installation]
  eggsecutable = setuptools.command.easy_install:bootstrap

The text was updated successfully, but these errors were encountered:

nicpottier · 2016-10-28T19:57:31Z

For the record, can make installation pass if I hack this into py36compat.py at line 110:

                    if not isinstance(item, tuple):
            continue

Obviously not the fix, but ya, setuptools doesn't seem to like the 'Documentation' object being passed in.

nicpottier · 2016-10-28T20:40:14Z

Also downgrading to 28.6.2 makes this work..

saily · 2016-10-28T21:30:21Z

same here, downgrading temporary solves the issue.

jaraco · 2016-10-29T02:22:10Z

@tweksteen

jaraco · 2016-10-29T02:23:04Z

@tweksteen, could you investigate. What is the cause? What's the proper fix?

pypa/setuptools#833

se-bastiaan · 2016-10-29T17:14:09Z

This issue also occurs when installing rcssmin. And issues in the repositories have been opened already: ndparker/rcssmin#6 & ndparker/rjsmin#9

DocCyblade · 2016-10-29T21:05:22Z

Working on another project noticed my builds were failing at rcssmin and then other packages. I tracked down the common error files and saw it was setuptools package, and found this issue.

I too can vouch that there are issues with this version 28.7.0, and the work around is using 28.6.1

- Change document storage location back to default (Issue #34) - Update DEMO data (Issue #37) - Workaround for Issue #38 - setuptools v28.7 issue (pypa/setuptools#833) - Use specific version of Mayan EDMS for build (v2.1.4)

tweksteen · 2016-10-30T22:43:54Z

@jaraco I don't understand why you reversed this patch when the correct thing is to fix the failing package itself. I've reproduced the install of rjsmin. The backtrace is similar to this:

Traceback (most recent call last):
  File "<string>", line 17, in <module>
  File "/home/twek/.virtualenvs/tmp-c712bdea02dc1ccb/build/rjsmin/setup.py", line 42, in <module>
    setup()
  File "/home/twek/.virtualenvs/tmp-c712bdea02dc1ccb/build/rjsmin/setup.py", line 33, in setup
    return run(script_args=args, ext=ext, manifest_only=_manifest)
  File "_setup/py2/setup.py", line 421, in run
    return _core.setup(**kwargs)
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "<string>", line 15, in replacement_run
  File "/home/twek/.virtualenvs/tmp-c712bdea02dc1ccb/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
    mm.run()
  [...]

If you read a bit the code of that specific package, you will see that it is strongly hooking onto the setup process. If you traceback the setup call, you'll find that data_files is populated using a call to the find_data function. A bit further down, if you have a look at what is returned by that function, you will find that it builds a list of Documentation object (an internal class define in this package).

Now, the standard Python documentation (3.7) mentions:
data_files specifies a sequence of (directory, files) pairs [...]
or even You can specify the data_files options as a simple sequence of files without specifying a target directory, but this is not recommended

At no point does it mention that it will accept random object... So yes, someone hacked onto the setup process and relied on the undocumented and inconsistent past behaviour of setuptools.

Moving forward, the correct solution is to create a bug for that project. But this is not a setuptools issue. setuptools should follow the standard behaviour described in the Python documentation.

jaraco · 2016-10-31T00:04:59Z

I reversed the patch because a regression was reported and I don't have enough time to dig in and analyze the underlying cause. Thanks for doing that. I had the feeling that multiple projects were already filing complaints, but now that I review the history, I see that you have indicated that the issue is probably isolated to one or a few projects and they may be abusing the API. I'm happy to bring the change back if that's the right behavior.

saily · 2016-10-31T09:23:27Z

@tweksteen if there's a bug people rely on it's not a bug, it's a feature (as linus would say) and focus should be to not break userspace, see: https://lkml.org/lkml/2012/12/23/75

jaraco · 2016-10-31T14:33:43Z

I wouldn't state it nearly as strongly as Linus did there. If this usage can be shown to be isolated to a single project (or just a few projects), and those projects can develop a solution avoid encountering the issue, then I'd have no problem re-introducing the behavior.

jaraco · 2016-10-31T14:53:22Z

The main thing I'd like to know - does the use of Documentation objects in data_files serve any purpose? That is, does that interface work for installing that documentation? I guess it must. I wonder how it is that Documentation objects are a suitable interface for other aspects of distutils/setuptools such that they don't cause problems during installation.

tweksteen · 2016-11-01T00:57:01Z

I've added further technical details on that issue on the rcssmin bug tracker (ndparker/rcssmin#6). The extra setup code in that project is hooking onto the install_data command and translating the options later on. This is where the Documentation objects are 'flatten' to a list of pair of strings. That translation does not happen when running the egg_info subcommand.

From a setuptools perspective, I'll create a new pull request with unit test to make sure that particular behaviour is tested. Additionally and since setuptools is a corner stone software of the Python community, we might want to setup larger integration tests where we install each of the top 100 Pypi packages (http://pypi-ranking.info/?) using the test version of setuptools. This will make sure this type of commit and rollback does not happen again. It will also give quantitative details on the impact of such change.

tweksteen · 2016-11-01T05:15:15Z

I see that there is already this kind of test in test_integration.py. Would it be worth expanding these to cover Pypi top downloaded packages?

joostrijneveld · 2016-11-02T14:02:42Z

If you make it the top 200 this would've been caught, as rcssmin is a dependency of django_compressor 😉

jaraco · 2016-11-02T14:50:53Z

Would it be worth expanding these to cover Pypi top downloaded packages?

Perhaps. The integration tests as currently implemented are already brittle and somewhat broken (virtualenvwrapper and novaclient are marked xfail). I'm sure the situation would get messy if a dynamic list of hundreds of packages were tested. If someone is able to maintain those tests, I'd be all for it.

We might want to add a command-line option to opt-in to those tests such that a developer running the tests locally wouldn't have to shoulder the burden of running those slow tests during development.

tweksteen · 2016-11-03T23:14:42Z

FYI, I have run the integration test on the top 300 packages of Pypi and django_compressor has been the only package failing the installation with the patched setuptools.

raphaelm added a commit to pretix/pretix that referenced this issue Oct 29, 2016

Revert to setuptools 28.6 on CI due to

d701133

pypa/setuptools#833

saily mentioned this issue Oct 29, 2016

Installation bug django-compressor (rcssmin) django-compressor/django-compressor#803

Closed

DocCyblade mentioned this issue Oct 29, 2016

Build Error: rcssmin DocCyblade/tkl-mayan-edms#38

Closed

jaraco closed this as completed in b18391c Oct 30, 2016

saily mentioned this issue Oct 30, 2016

Ability to add args on how virtualenv is created tox-dev/tox#393

Closed

tweksteen mentioned this issue Oct 30, 2016

Installing with new setuptools fails ndparker/rcssmin#6

Closed

tweksteen mentioned this issue Nov 3, 2016

Add test to verify the install of Pypi top packages #839

Merged

Installation of packages fails after 28.7.0 upgrade: Documentation object is not iterable #833

Installation of packages fails after 28.7.0 upgrade: Documentation object is not iterable #833

nicpottier commented Oct 28, 2016

nicpottier commented Oct 28, 2016

nicpottier commented Oct 28, 2016

saily commented Oct 28, 2016

jaraco commented Oct 29, 2016

jaraco commented Oct 29, 2016

se-bastiaan commented Oct 29, 2016

DocCyblade commented Oct 29, 2016 •

edited

tweksteen commented Oct 30, 2016

jaraco commented Oct 31, 2016

saily commented Oct 31, 2016 •

edited

jaraco commented Oct 31, 2016

jaraco commented Oct 31, 2016

tweksteen commented Nov 1, 2016

tweksteen commented Nov 1, 2016

joostrijneveld commented Nov 2, 2016

jaraco commented Nov 2, 2016

tweksteen commented Nov 3, 2016

Installation of packages fails after 28.7.0 upgrade: Documentation object is not iterable #833

Installation of packages fails after 28.7.0 upgrade: Documentation object is not iterable #833

Comments

nicpottier commented Oct 28, 2016

nicpottier commented Oct 28, 2016

nicpottier commented Oct 28, 2016

saily commented Oct 28, 2016

jaraco commented Oct 29, 2016

jaraco commented Oct 29, 2016

se-bastiaan commented Oct 29, 2016

DocCyblade commented Oct 29, 2016 • edited

tweksteen commented Oct 30, 2016

jaraco commented Oct 31, 2016

saily commented Oct 31, 2016 • edited

jaraco commented Oct 31, 2016

jaraco commented Oct 31, 2016

tweksteen commented Nov 1, 2016

tweksteen commented Nov 1, 2016

joostrijneveld commented Nov 2, 2016

jaraco commented Nov 2, 2016

tweksteen commented Nov 3, 2016

DocCyblade commented Oct 29, 2016 •

edited

saily commented Oct 31, 2016 •

edited