New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installation of packages fails after 28.7.0 upgrade: Documentation object is not iterable #833

Closed
nicpottier opened this Issue Oct 28, 2016 · 17 comments

Comments

Projects
None yet
7 participants
@nicpottier

nicpottier commented Oct 28, 2016

Getting a weird error: TypeError: 'Documentation' object is not iterable trying to install rjsmin in a brand new virtualenv with the latest setup tools. This is on Ubuntu 16.0.4 and my OS X machine does the same thing.

ubuntu@staging:~$ virtualenv env
New python executable in /home/ubuntu/env/bin/python
Installing setuptools, pip, wheel...done.
ubuntu@staging:~$ source env/bin/activate
(env) ubuntu@staging:~$ pip install rjsmin
Collecting rjsmin
  Using cached rjsmin-1.0.12.tar.gz
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/rjsmin.egg-info
    writing pip-egg-info/rjsmin.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/rjsmin.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/rjsmin.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/rjsmin.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-EeZDjY/rjsmin/setup.py", line 42, in <module>
        setup()
      File "/tmp/pip-build-EeZDjY/rjsmin/setup.py", line 33, in setup
        return run(script_args=args, ext=ext, manifest_only=_manifest)
      File "_setup/py2/setup.py", line 421, in run
        return _core.setup(**kwargs)
      File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 279, in run
        self.find_sources()
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
        mm.run()
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 533, in run
        self.add_defaults()
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 562, in add_defaults
        sdist.add_defaults(self)
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 35, in add_defaults
        self._add_defaults_data_files()
      File "/home/ubuntu/env/local/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 111, in _add_defaults_data_files
        dirname, filenames = item
    TypeError: 'Documentation' object is not iterable

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-EeZDjY/rjsmin/
(env) ubuntu@staging:~$ pip show setuptools

---
Metadata-Version: 2.0
Name: setuptools
Version: 28.7.0
Summary: Easily download, build, install, upgrade, and uninstall Python packages
Home-page: https://github.com/pypa/setuptools
Author: Python Packaging Authority
Author-email: distutils-sig@python.org
Installer: pip
License: UNKNOWN
Location: /home/ubuntu/env/lib/python2.7/site-packages
Requires: 
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  License :: OSI Approved :: MIT License
  Operating System :: OS Independent
  Programming Language :: Python :: 2.6
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4
  Programming Language :: Python :: 3.5
  Topic :: Software Development :: Libraries :: Python Modules
  Topic :: System :: Archiving :: Packaging
  Topic :: System :: Systems Administration
  Topic :: Utilities
Entry-points:
  [console_scripts]
  easy_install = setuptools.command.easy_install:main
  easy_install-3.5 = setuptools.command.easy_install:main
  [distutils.commands]
  alias = setuptools.command.alias:alias
  bdist_egg = setuptools.command.bdist_egg:bdist_egg
  bdist_rpm = setuptools.command.bdist_rpm:bdist_rpm
  bdist_wininst = setuptools.command.bdist_wininst:bdist_wininst
  build_ext = setuptools.command.build_ext:build_ext
  build_py = setuptools.command.build_py:build_py
  develop = setuptools.command.develop:develop
  easy_install = setuptools.command.easy_install:easy_install
  egg_info = setuptools.command.egg_info:egg_info
  install = setuptools.command.install:install
  install_egg_info = setuptools.command.install_egg_info:install_egg_info
  install_lib = setuptools.command.install_lib:install_lib
  install_scripts = setuptools.command.install_scripts:install_scripts
  register = setuptools.command.register:register
  rotate = setuptools.command.rotate:rotate
  saveopts = setuptools.command.saveopts:saveopts
  sdist = setuptools.command.sdist:sdist
  setopt = setuptools.command.setopt:setopt
  test = setuptools.command.test:test
  upload = setuptools.command.upload:upload
  upload_docs = setuptools.command.upload_docs:upload_docs
  [distutils.setup_keywords]
  convert_2to3_doctests = setuptools.dist:assert_string_list
  dependency_links = setuptools.dist:assert_string_list
  eager_resources = setuptools.dist:assert_string_list
  entry_points = setuptools.dist:check_entry_points
  exclude_package_data = setuptools.dist:check_package_data
  extras_require = setuptools.dist:check_extras
  include_package_data = setuptools.dist:assert_bool
  install_requires = setuptools.dist:check_requirements
  namespace_packages = setuptools.dist:check_nsp
  package_data = setuptools.dist:check_package_data
  packages = setuptools.dist:check_packages
  python_requires = setuptools.dist:check_specifier
  setup_requires = setuptools.dist:check_requirements
  test_loader = setuptools.dist:check_importable
  test_runner = setuptools.dist:check_importable
  test_suite = setuptools.dist:check_test_suite
  tests_require = setuptools.dist:check_requirements
  use_2to3 = setuptools.dist:assert_bool
  use_2to3_exclude_fixers = setuptools.dist:assert_string_list
  use_2to3_fixers = setuptools.dist:assert_string_list
  zip_safe = setuptools.dist:assert_bool
  [egg_info.writers]
  PKG-INFO = setuptools.command.egg_info:write_pkg_info
  dependency_links.txt = setuptools.command.egg_info:overwrite_arg
  depends.txt = setuptools.command.egg_info:warn_depends_obsolete
  eager_resources.txt = setuptools.command.egg_info:overwrite_arg
  entry_points.txt = setuptools.command.egg_info:write_entries
  namespace_packages.txt = setuptools.command.egg_info:overwrite_arg
  requires.txt = setuptools.command.egg_info:write_requirements
  top_level.txt = setuptools.command.egg_info:write_toplevel_names
  [setuptools.installation]
  eggsecutable = setuptools.command.easy_install:bootstrap
@nicpottier

This comment has been minimized.

nicpottier commented Oct 28, 2016

For the record, can make installation pass if I hack this into py36compat.py at line 110:

                    if not isinstance(item, tuple):
            continue

Obviously not the fix, but ya, setuptools doesn't seem to like the 'Documentation' object being passed in.

@nicpottier

This comment has been minimized.

nicpottier commented Oct 28, 2016

Also downgrading to 28.6.2 makes this work..

@saily

This comment has been minimized.

saily commented Oct 28, 2016

same here, downgrading temporary solves the issue.

@jaraco

This comment has been minimized.

Member

jaraco commented Oct 29, 2016

@jaraco

This comment has been minimized.

Member

jaraco commented Oct 29, 2016

@tweksteen, could you investigate. What is the cause? What's the proper fix?

raphaelm added a commit to pretix/pretix that referenced this issue Oct 29, 2016

@se-bastiaan

This comment has been minimized.

se-bastiaan commented Oct 29, 2016

This issue also occurs when installing rcssmin. And issues in the repositories have been opened already: ndparker/rcssmin#6 & ndparker/rjsmin#9

@DocCyblade

This comment has been minimized.

DocCyblade commented Oct 29, 2016

Working on another project noticed my builds were failing at rcssmin and then other packages. I tracked down the common error files and saw it was setuptools package, and found this issue.

I too can vouch that there are issues with this version 28.7.0, and the work around is using 28.6.1

DocCyblade pushed a commit to DocCyblade/tkl-mayan-edms that referenced this issue Oct 29, 2016

Ken Robinson
Fixes and updates
   - Change document storage location back to default (Issue #34)
   - Update DEMO data (Issue #37)
   - Workaround for Issue #38
      - setuptools v28.7 issue (pypa/setuptools#833)
   - Use specific version of Mayan EDMS for build (v2.1.4)

@jaraco jaraco closed this in b18391c Oct 30, 2016

@tweksteen

This comment has been minimized.

Contributor

tweksteen commented Oct 30, 2016

@jaraco I don't understand why you reversed this patch when the correct thing is to fix the failing package itself. I've reproduced the install of rjsmin. The backtrace is similar to this:

Traceback (most recent call last):
  File "<string>", line 17, in <module>
  File "/home/twek/.virtualenvs/tmp-c712bdea02dc1ccb/build/rjsmin/setup.py", line 42, in <module>
    setup()
  File "/home/twek/.virtualenvs/tmp-c712bdea02dc1ccb/build/rjsmin/setup.py", line 33, in setup
    return run(script_args=args, ext=ext, manifest_only=_manifest)
  File "_setup/py2/setup.py", line 421, in run
    return _core.setup(**kwargs)
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "<string>", line 15, in replacement_run
  File "/home/twek/.virtualenvs/tmp-c712bdea02dc1ccb/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
    mm.run()
  [...] 

If you read a bit the code of that specific package, you will see that it is strongly hooking onto the setup process. If you traceback the setup call, you'll find that data_files is populated using a call to the find_data function. A bit further down, if you have a look at what is returned by that function, you will find that it builds a list of Documentation object (an internal class define in this package).

Now, the standard Python documentation (3.7) mentions:
data_files specifies a sequence of (directory, files) pairs [...]
or even You can specify the data_files options as a simple sequence of files without specifying a target directory, but this is not recommended

At no point does it mention that it will accept random object... So yes, someone hacked onto the setup process and relied on the undocumented and inconsistent past behaviour of setuptools.

Moving forward, the correct solution is to create a bug for that project. But this is not a setuptools issue. setuptools should follow the standard behaviour described in the Python documentation.

@jaraco

This comment has been minimized.

Member

jaraco commented Oct 31, 2016

I reversed the patch because a regression was reported and I don't have enough time to dig in and analyze the underlying cause. Thanks for doing that. I had the feeling that multiple projects were already filing complaints, but now that I review the history, I see that you have indicated that the issue is probably isolated to one or a few projects and they may be abusing the API. I'm happy to bring the change back if that's the right behavior.

@saily

This comment has been minimized.

saily commented Oct 31, 2016

@tweksteen if there's a bug people rely on it's not a bug, it's a feature (as linus would say) and focus should be to not break userspace, see: https://lkml.org/lkml/2012/12/23/75

@jaraco

This comment has been minimized.

Member

jaraco commented Oct 31, 2016

I wouldn't state it nearly as strongly as Linus did there. If this usage can be shown to be isolated to a single project (or just a few projects), and those projects can develop a solution avoid encountering the issue, then I'd have no problem re-introducing the behavior.

@jaraco

This comment has been minimized.

Member

jaraco commented Oct 31, 2016

The main thing I'd like to know - does the use of Documentation objects in data_files serve any purpose? That is, does that interface work for installing that documentation? I guess it must. I wonder how it is that Documentation objects are a suitable interface for other aspects of distutils/setuptools such that they don't cause problems during installation.

@tweksteen

This comment has been minimized.

Contributor

tweksteen commented Nov 1, 2016

I've added further technical details on that issue on the rcssmin bug tracker (ndparker/rcssmin#6). The extra setup code in that project is hooking onto the install_data command and translating the options later on. This is where the Documentation objects are 'flatten' to a list of pair of strings. That translation does not happen when running the egg_info subcommand.

From a setuptools perspective, I'll create a new pull request with unit test to make sure that particular behaviour is tested. Additionally and since setuptools is a corner stone software of the Python community, we might want to setup larger integration tests where we install each of the top 100 Pypi packages (http://pypi-ranking.info/?) using the test version of setuptools. This will make sure this type of commit and rollback does not happen again. It will also give quantitative details on the impact of such change.

@tweksteen

This comment has been minimized.

Contributor

tweksteen commented Nov 1, 2016

I see that there is already this kind of test in test_integration.py. Would it be worth expanding these to cover Pypi top downloaded packages?

@joostrijneveld

This comment has been minimized.

joostrijneveld commented Nov 2, 2016

If you make it the top 200 this would've been caught, as rcssmin is a dependency of django_compressor 😉

@jaraco

This comment has been minimized.

Member

jaraco commented Nov 2, 2016

Would it be worth expanding these to cover Pypi top downloaded packages?

Perhaps. The integration tests as currently implemented are already brittle and somewhat broken (virtualenvwrapper and novaclient are marked xfail). I'm sure the situation would get messy if a dynamic list of hundreds of packages were tested. If someone is able to maintain those tests, I'd be all for it.

We might want to add a command-line option to opt-in to those tests such that a developer running the tests locally wouldn't have to shoulder the burden of running those slow tests during development.

@tweksteen

This comment has been minimized.

Contributor

tweksteen commented Nov 3, 2016

FYI, I have run the integration test on the top 300 packages of Pypi and django_compressor has been the only package failing the installation with the patched setuptools.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment