ez_setup.py failing sporadically when setuptools is already installed

[ghost]

Originally reported by: jurko (Bitbucket: jurko, GitHub: jurko)

---

When you run ez_setup.py to install setuptools into a Python environment where setuptools are not already installed - everything works great. But if you do that, and then rerun ez_setup.py, you sometimes (ranging from occasionally to almost always, depending on the exact Python & setuptools version used) get an ugly ZipImportError about the setuptools zipped egg file having a bad local header.

I encountered & tested this on Windows 7 SP1 x64 with different clean CPython installations and different setuptools versions.

Here are some of the collected test results:

#!text
CPython 2.6 (x86)
  - setuptools 3.0, 3.1, 3.2, 3.3
      - fails sporadically - sometimes often, sometimes rarely
  - setuptools 2.2 - works every time

CPython 3.3 (both x86 & x64)
  - setuptools 3.0, 3.1, 3.2, 3.3, 2.2, 2.1, 2.0, 1.4.2
      - very rarely works

Here's a Windows batch script I used to test this behaviour:

#!cmd
@setlocal

@set LIMIT=%~1
@if not defined LIMIT goto :param_processing_done
@set /A LIMIT=%LIMIT%
@if %LIMIT% LEQ 0 set LIMIT=
:param_processing_done

@set TOTAL=0
@set SUCCESS=0
:here_we_go_again
@set /A TOTAL=%TOTAL% + 1
@call :install && set /A SUCCESS=%SUCCESS% + 1
@set ERROR=%ERRORLEVEL%
@echo Success rate: %SUCCESS%/%TOTAL%
@if not defined LIMIT goto :here_we_go_again
@if %TOTAL% LSS %LIMIT% goto :here_we_go_again
@exit /b %ERROR%

:install
    ::call py266 ez_setup.py
    ::call py266_x86 ez_setup.py
    call py340 ez_setup.py
    ::call py340_x86 ez_setup_2.2.py
    @exit /b

Here are some captured console outputs containing the actual displayed exception tracebacks.

CPython 2.6.6 x86 / setuptools 3.3

#!text
Copying setuptools-3.3-py2.6.egg to c:\program files (x86)\python\python266\lib\site-packages
setuptools 3.3 is already the active version in easy-install.pth
Installing easy_install-script.py script to C:\Program Files (x86)\Python\Python266\Scripts
Installing easy_install.exe script to C:\Program Files (x86)\Python\Python266\Scripts
Installing easy_install.exe.manifest script to C:\Program Files (x86)\Python\Python266\Scripts
Installing easy_install-2.6-script.py script to C:\Program Files (x86)\Python\Python266\Scripts
Installing easy_install-2.6.exe script to C:\Program Files (x86)\Python\Python266\Scripts
Installing easy_install-2.6.exe.manifest script to C:\Program Files (x86)\Python\Python266\Scripts

Installed c:\program files (x86)\python\python266\lib\site-packages\setuptools-3.3-py2.6.egg
Processing dependencies for setuptools==3.3
Traceback (most recent call last):
  File "setup.py", line 203, in <module>
    dist = setuptools.setup(**setup_params)
  File "C:\Program Files (x86)\Python\Python266\lib\distutils\core.py", line 152, in setup
    dist.run_commands()
  File "C:\Program Files (x86)\Python\Python266\lib\distutils\dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "C:\Program Files (x86)\Python\Python266\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\setuptools\command\install.py", line 73, in run
    self.do_egg_install()
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\setuptools\command\install.py", line 96, in do_egg_install
    cmd.run()
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\setuptools\command\easy_install.py", line 358, in run
    self.easy_install(spec, not self.no_deps)
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\setuptools\command\easy_install.py", line 574, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\setuptools\command\easy_install.py", line 625, in install_item
    self.process_distribution(spec, dist, deps)
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\setuptools\command\easy_install.py", line 671, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\pkg_resources.py", line 633, in resolve
    requirements.extend(dist.requires(req.extras)[::-1])
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\pkg_resources.py", line 2291, in requires
    dm = self._dep_map
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\pkg_resources.py", line 2277, in _dep_map
    for extra,reqs in split_sections(self._get_metadata(name)):
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\pkg_resources.py", line 2715, in split_sections
    for line in yield_lines(s):
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\pkg_resources.py", line 1989, in yield_lines
    for ss in strs:
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\pkg_resources.py", line 2305, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\pkg_resources.py", line 1369, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\pkg_resources.py", line 1361, in get_metadata
    return self._get(self._fn(self.egg_info,name))
  File "c:\users\jurko\appdata\local\temp\tmp_xhasz\setuptools-3.3\pkg_resources.py", line 1425, in _get
    return self.loader.get_data(path)
zipimport.ZipImportError: bad local file header in c:\program files (x86)\python\python266\lib\site-packages\setuptools-3.3-py2.6.egg
Something went wrong during the installation.
See the error message above.

CPython 3.4 x86 / setuptools 2.0

#!text
Copying setuptools-2.0-py3.4.egg to c:\program files (x86)\python\python340\lib\site-packages
setuptools 2.0 is already the active version in easy-install.pth
Installing easy_install-script.py script to C:\Program Files (x86)\Python\Python340\Scripts
Installing easy_install.exe script to C:\Program Files (x86)\Python\Python340\Scripts
Installing easy_install.exe.manifest script to C:\Program Files (x86)\Python\Python340\Scripts
Installing easy_install-3.4-script.py script to C:\Program Files (x86)\Python\Python340\Scripts
Installing easy_install-3.4.exe script to C:\Program Files (x86)\Python\Python340\Scripts
Installing easy_install-3.4.exe.manifest script to C:\Program Files (x86)\Python\Python340\Scripts

Installed c:\program files (x86)\python\python340\lib\site-packages\setuptools-2.0-py3.4.egg
Processing dependencies for setuptools==2.0
Traceback (most recent call last):
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 2221, in _dep_map
    return self.__dep_map
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 2291, in __getattr__
    raise AttributeError(attr)
AttributeError: _Distribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 200, in <module>
    dist = setuptools.setup(**setup_params)
  File "C:\Program Files (x86)\Python\Python340\lib\distutils\core.py", line 149, in setup
    dist.run_commands()
  File "C:\Program Files (x86)\Python\Python340\lib\distutils\dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "C:\Program Files (x86)\Python\Python340\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\setuptools\command\install.py", line 73, in run
    self.do_egg_install()
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\setuptools\command\install.py", line 101, in do_egg_install
    cmd.run()
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\setuptools\command\easy_install.py", line 358, in run
    self.easy_install(spec, not self.no_deps)
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\setuptools\command\easy_install.py", line 574, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\setuptools\command\easy_install.py", line 625, in install_item
    self.process_distribution(spec, dist, deps)
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\setuptools\command\easy_install.py", line 671, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 572, in resolve
    requirements.extend(dist.requires(req.extras)[::-1])
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 2240, in requires
    dm = self._dep_map
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 2225, in _dep_map
    for extra,reqs in split_sections(self._get_metadata(name)):
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 2666, in split_sections
    for line in yield_lines(s):
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 1926, in yield_lines
    for ss in strs:
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 2254, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 1316, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 1313, in get_metadata
    return self._get(self._fn(self.egg_info,name)).decode("utf-8")
  File "C:\Users\Jurko\AppData\Local\Temp\tmp77hrjll5\setuptools-2.0\pkg_resources.py", line 1372, in _get
    return self.loader.get_data(path)
zipimport.ZipImportError: bad local file header in c:\program files (x86)\python\python340\lib\site-packages\setuptools-2.0-py3.4.egg
Something went wrong during the installation.
See the error message above.

CPython 3.4 x64 / setuptools 3.3

#!text
Copying setuptools-3.3-py3.4.egg to c:\program files\python\python340\lib\site-packages
setuptools 3.3 is already the active version in easy-install.pth
Installing easy_install-3.4-script.py script to C:\Program Files\Python\Python340\Scripts
Installing easy_install-3.4.exe script to C:\Program Files\Python\Python340\Scripts
Installing easy_install-script.py script to C:\Program Files\Python\Python340\Scripts
Installing easy_install.exe script to C:\Program Files\Python\Python340\Scripts

Installed c:\program files\python\python340\lib\site-packages\setuptools-3.3-py3.4.egg
Processing dependencies for setuptools==3.3
Traceback (most recent call last):
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 2273, in _dep_map
    return self.__dep_map
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 2344, in __getattr__
    raise AttributeError(attr)
AttributeError: _Distribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 203, in <module>
    dist = setuptools.setup(**setup_params)
  File "C:\Program Files\Python\Python340\lib\distutils\core.py", line 149, in setup
    dist.run_commands()
  File "C:\Program Files\Python\Python340\lib\distutils\dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "C:\Program Files\Python\Python340\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\setuptools\command\install.py", line 73, in run
    self.do_egg_install()
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\setuptools\command\install.py", line 96, in do_egg_install
    cmd.run()
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\setuptools\command\easy_install.py", line 358, in run
    self.easy_install(spec, not self.no_deps)
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\setuptools\command\easy_install.py", line 574, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\setuptools\command\easy_install.py", line 625, in install_item
    self.process_distribution(spec, dist, deps)
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\setuptools\command\easy_install.py", line 671, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 633, in resolve
    requirements.extend(dist.requires(req.extras)[::-1])
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 2291, in requires
    dm = self._dep_map
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 2277, in _dep_map
    for extra,reqs in split_sections(self._get_metadata(name)):
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 2715, in split_sections
    for line in yield_lines(s):
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 1989, in yield_lines
    for ss in strs:
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 2305, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 1369, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 1366, in get_metadata
    return self._get(self._fn(self.egg_info,name)).decode("utf-8")
  File "C:\Users\Jurko\AppData\Local\Temp\tmphxzo4q_i\setuptools-3.3\pkg_resources.py", line 1425, in _get
    return self.loader.get_data(path)
zipimport.ZipImportError: bad local file header in c:\program files\python\python340\lib\site-packages\setuptools-3.3-py3.4.egg
Something went wrong during the installation.
See the error message above.

Hope this helps.

Best regards,
Jurko Gospodnetić

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/168

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I've had this same issue. The issue can also occur outside of running ez_setup, but also when simply easy_installing setuptools or running setup.py from a tarball or zipball. I know I've had a conversation about this before, but I forget what the outcome is. I'll look around more later.

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Well, as easy as it is to reproduce this behaviour 😄, it seems weird that it persisted through so many Python & setuptools versions. Let me know if I can help in any way.

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by pje (Bitbucket: pje, GitHub: pje):

---

The cause of the error is usually a corrupt/invalid zipimport cache entry. If you replace the zipfile without removing the corresponding entry from the cache, then you're going to get an error if the zipfile's contents are not in the same locations as they were in the old zipfile.

Basically, any time you move, rename, delete, or overwrite a zipfile that could have previously been imported (or inspected via a zipimporter), you need to delete the corresponding entry in the zipimport cache.

[ghost]

Original comment by johnsmith (Bitbucket: johnsmith, GitHub: johnsmith):

---

@pje How do I remove the corresponding entry in the zipimport cache?

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Have you looked at zipimport._zip_directory_cache?

It seems to be a dictionary with zip file paths as keys.

#!python
>>> zipimport._zip_directory_cache.keys()
dict_keys(['C:\\Program Files\\Python\\Python340\\lib\\site-packages\\setuptools-3.3-py3.4.egg'])

zipimport module's docstring description mentions it:

#!python
>>> help(zipimport)
Help on built-in module zipimport:

NAME
    zipimport - zipimport provides support for importing Python modules from Zip archives.

MODULE REFERENCE
    http://docs.python.org/3.4/library/zipimport

    The following documentation is automatically generated from the Python
    source files.  It may be incomplete, incorrect or include features that
    are considered implementation detail and may vary between Python
    implementations.  When in doubt, consult the module reference at the
    location listed above.

DESCRIPTION
    This module exports three objects:
    - zipimporter: a class; its constructor takes a path to a Zip archive.
    - ZipImportError: exception raised by zipimporter objects. It's a
      subclass of ImportError, so it can be caught as ImportError, too.
    - _zip_directory_cache: a dict, mapping archive paths to zip directory
      info dicts, as used in zipimporter._files.

    It is usually not needed to use the zipimport module explicitly; it is
    used by the builtin import mechanism for sys.path items that are paths
    to Zip archives.

and it seems to be implemented in CPython's Modules\zipimport.c source file.

Hope this helps.

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by pje (Bitbucket: pje, GitHub: pje):

---

That's one cache. The other is sys.path_importer_cache. Both need to have any entries matching the zipfile removed. On top of that, more recent Pythons set __loader__ attributes on the modules loaded, which refer to the old zipimporter object... which means that if you try using those to load data or anything else from the old cache entries, you'll be hosed as well.

Just to make things even more fun, there is the fact that sys.path_importer_cache entries can have subpaths -- if /foo/bar/setuptools.egg is an entry, then you will also have /foo/bar/setuptools.egg/setuptools, /foo/bar/setuptools.egg/setuptools/command, etc. The simplest thing to do there is just clear sys.path_importer_cache, which I'm pretty sure I already put code in easy_install to do.

If the relevant zipimport entry is being removed and the importer cache is being cleared, that leaves stale __loader__ attributes as the likely culprit. So be sure to check all three.

[ghost]

Original comment by matthewbrett (Bitbucket: matthewbrett, GitHub: matthewbrett):

---

As a datapoint, I have this failure reliably on OSX.

It happening reliably on our automated builds starting on March 17 : http://nipy.bic.berkeley.edu/builders/numpy-bdist-whl-osx-2.7-downloaded

I can cause it reliably on my laptop with current tip of setuptools with:

mkvirtualenv test
# edit ez_setup.py to use version 3.3 as 3.4 is not present at given URL
# this works
python ez_setup.py
# this gives a zipfile error
python ez_setup.py

Setting the ez_setup.py version to 3.2 makes the error less likely, but I rerun ez_setup.py a few times, I get the error again. Running again with no other change most often works.

[ghost]

Original comment by mtimmsj (Bitbucket: mtimmsj, GitHub: Unknown):

---

I see this consistently if:

• I install a zipped egg using easy_install /path/to/eggfile.egg
• Later decided I want to install it unzipped and go back and do so without uninstalling it first - easy_install -Z /path/to/eggfile.egg
• And that egg defines dependancies in the metadata via install_requires = ['somepackage']

This always fails in pkg_resources.py at return self.loader.get_data(path) in _get.

[ghost]

Original comment by tal_weiss (Bitbucket: tal_weiss, GitHub: Unknown):

---

Why is this issue classified as Minor? Install needs to work perfectly, every time. It is part of our server deployment scripts.

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

When I ran into it and saw no one else reported it already, and that it must have been present for a long time now, I assumed it is not all that important...

And on the other hand, I figured it is quite easy to manually check if and what version of setuptools is already installed so a manual workaround (uninstall by deleting a few files) is not all that difficult.

Hmmm... I might try and take a look at this now. If anyone more knowledgeable about setuptools internals wants to join me - I'm available on Skype (username: jurkog).

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Ok, so far I've managed to get it reproduced reliably in the following scenario:

• Windows 7 SP1 x64
• bleeding-edge debug CPython build (x64)
• bleeding-edge setuptools checkout

Steps to reproduce:

• run python_d setup.py install from the setuptools checkout
• change setuptools's pkg_resources.py module, e.g. by adding an empty line at the start is enough or removing it if you already added it before
• run python_d setup.py install from the setuptools checkout again

Still working on it, but now at least I should be able to debug into the relevant zipimport C code...

If anyone more experienced with setuptools/zipimport internals wants to chime in - I'd be more than happy to accept any useful tips/thoughts/ideas... 😄

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by pje (Bitbucket: pje, GitHub: pje):

---

Does it make a difference what Python version is used? i.e. Do any older versions (e.g. 3.1, 3.2, 2.x) work correctly?

As I mentioned earlier, there is code in easy_install (i.e. the uncache_zipdir function) that was specifically added to fix this problem in earlier versions of setuptools, but I do not know if there are any 3.3 or 3.4 changes (e.g. due to importlib) that might have affected it. Knowing for sure whether the current codebase still works with Python 2.x or any later versions would help distinguish between a regression in setuptools and a change in Python breaking setuptools.

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Ok, just tested it and the described scenario fails consistently with:

• CPython 2.6.6
• CPython 2.7.6
• CPython 3.2.5
• CPython 3.3.3
• CPython 3.3.5
• CPython 3.4.0
• CPython 3.5.0.dev

With Python 3.1.3 setuptools' py313 setup.py install fails on its second run and reports the following exception (but that's most likely an unrelated issue):

D:\Workplace\Python SetupTools>py313 setup.py install
running install
Traceback (most recent call last):
  File "D:\Workplace\Python SetupTools\pkg_resources.py", line 2264, in version
    return self._version
  File "D:\Workplace\Python SetupTools\pkg_resources.py", line 2349, in __getattr__
    raise AttributeError(attr)
AttributeError: _version

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 217, in <module>
    dist = setuptools.setup(**setup_params)
  File "C:\Program Files\Python\Python313\lib\distutils\core.py", line 149, in setup
    dist.run_commands()
  File "C:\Program Files\Python\Python313\lib\distutils\dist.py", line 919, in run_commands
    self.run_command(cmd)
  File "C:\Program Files\Python\Python313\lib\distutils\dist.py", line 938, in run_command
    cmd_obj.run()
  File "D:\Workplace\Python SetupTools\setuptools\command\install.py", line 61, in run
    self.do_egg_install()
  File "D:\Workplace\Python SetupTools\setuptools\command\install.py", line 92, in do_egg_install
    easy_install = self.distribution.get_command_class('easy_install')
  File "D:\Workplace\Python SetupTools\setuptools\dist.py", line 364, in get_command_class
    self.cmdclass[command] = cmdclass = ep.load()
  File "D:\Workplace\Python SetupTools\pkg_resources.py", line 2093, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "D:\Workplace\Python SetupTools\setuptools\command\easy_install.py", line 44, in <module>
    from setuptools.package_index import PackageIndex
  File "D:\Workplace\Python SetupTools\setuptools\package_index.py", line 203, in <module>
    sys.version[:3], require('setuptools')[0].version
  File "D:\Workplace\Python SetupTools\pkg_resources.py", line 2266, in version
    for line in self._get_metadata(self.PKG_INFO):
  File "D:\Workplace\Python SetupTools\pkg_resources.py", line 2310, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "D:\Workplace\Python SetupTools\pkg_resources.py", line 1369, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "D:\Workplace\Python SetupTools\pkg_resources.py", line 1366, in get_metadata
    return self._get(self._fn(self.egg_info,name)).decode("utf-8")
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 10717: invalid continuation byte

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Reproducible using CPython 3.1.3 as well... once you manually avoid the unicode decoding issue (seems to balk when attempting to read the string Tarek Ziadé from setuptools.egg-info/PKG-INFO).

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Pull request #45 fixes the CPython 3.1.3 installation issue.

I'll do more research on the original ZipImportError issue tomorrow.

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by pje (Bitbucket: pje, GitHub: pje):

---

I've reviewed the code and the tracebacks; it seems to me the next critical point is to determine whether uncache_zipdir() is actually being called, and whether the path it's called with corresponds to the installation target. Perhaps setting a pdb breakpoint in the egg_distribution function or modifying it to dump the keys for both the zipimport cache and sys.path_importer_cache would be a good idea, to confirm there are no entries present for the corresponding path.

If there are entries, then there is something wrong with the uncache functions or they aren't being called. If there aren't entries corresponding to the cache, then something else is wrong -- like maybe there's some other cache not being considered. (Which seems unlikely.)

I also wonder what's the oldest version of setuptools itself that manifests this bug, since the uncaching code was added to fix this problem quite a long time ago. (Suggesting that this is a regression due to some other change in the setuptools code base, or perhaps a change in ez_setup.)

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Just did some bisecting over the old repo commits, and the problematic commit has been found on the setuptools-0.6 branch:

1. bug can be reproduced with revision f40b810acc5f6494735c912a625d647dc2a3c582 (number 1727 in my repo).
2. bug can not be reproduced with revision 88af28ece55fbb7ebc2208a375e4093fc27a24bd (number 1726 in my repo).

The commit in question was done by @pje with commit description:

Enable safe SSL dependency installs via "easy_install setuptools[ssl]"
(grafted from 695a82c1934090e7c56ba74e7b5a2bdc13071698)

All testing done using CPython 2.7.6. x64 on Windows 7 SP1 x64.

The behaviour so far has been 100% consistent, so this should at least give us a starting point.

I'll continue looking into this a bit later on when I get back home.

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

@pje - I've been debugging the issue using pdb and so far I think your cleanup hook is getting called correctly. Although I am not yet sure what should really be stored in those caches.

I'll some more debugging tonight and hopefully have some more exact results for you.

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by pje (Bitbucket: pje, GitHub: pje):

---

Based on that commit, it looks as though the problem was exposed because of the addition of a requirements file to setuptools' metadata. Before the commit, the file wouldn't have existed, so it wouldn't have called loader.get_data() during the process. So that at least clarifies the issue was never really fixed before. :-(

At this point, it seems to me that the egg_distribution() function is the critical point. It creates the zipimporter instance, and it is at that point that the presence or absence of the zip directory cache entry matters. Specifically, this line:

metadata = EggMetadata(zipimport.zipimporter(egg_path))

The value of egg_path must not have a corresponding entry in the zipimport cache at the time this executes, or it will see stale data (for the previous version of the zipfile), and get the bad local header error. I suspect the problem here is that somehow that entry isn't being removed.

The init function for a zipimporter checks for a cache entry, and if found, uses it. Otherwise it opens the zipfile on disk, reads its directory, and caches it. So this is the critical moment for whether bad data will be seen later, in the code that actually gets the error, as far as I can tell.

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Yup, that commit introduces the setuptools.egg-info/requires.txt file.

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

I found time to do some more debugging and I can now confirm that the reason is in fact an old zipimporter instance being used to load data. The old one contains cached information about the original egg's contents, which does not match the new egg. It then tries to look for a specific file in the egg at a position indicated by the cached content information, but that location does not point to a valid zip file entry (it points in the middle of some 'random' zip entry instead of a start of one) and so we get burned...

I can now reproduce the behaviour and inspect the buggy zipimporter instance under pdb.

I inspected the zipimport._zip_directory_cache & sys.path_importer_cache cache content at multiple points during my testing and it does get cleared 'well enough'. Their entries for the exact egg path get cleared, but, at least on Windows, they also contain additional entries with differently spelled paths (upper/lower letter case differences) that do not. However, I do not think what remains in them is the cause of the exact error I am getting. I tried clearing them completely and that did not resolve the issue, so I'll most likely just add that as a separate cleanup issue for later.

The problem is somewhere inside setuptools. As I see it, the stale zipimporter object has been cached in some setuptools pkg_resources.EggMetadata objects (their loader attribute). More research is needed to discover exactly where those pkg_resources.EggMetadata objects get created and stored.

During the installation procedure the following occurs in order (none of them recursively):

[1] dist\\setuptools-3.4.4dev-py3.4.egg egg gets built. We now have an old one installed and a new one built in the dist folder.

[2] command.easy_install.easy_install.egg_distribution() gets called for dist\\setuptools-3.4.4dev-py3.4.egg. Original egg has already been loaded (and cached) by Python. New egg now gets loaded as well but from its temporary dist location.

[3] The original egg file is removed. Output:

#!text
Removing c:\program files\python\python340\lib\site-packages\setuptools-3.4.4dev-py3.4.egg

[4] command.easy_install.uncache_zipdir() is called and clears zipimport._zip_directory_cache & sys.path_importer_cache Python caches.

[5] The new egg is copies into the old one's location. Output:

#!text
Copying setuptools-3.4.4dev-py3.4.egg to c:\program files\python\python340\lib\site-packages

[6] command.easy_install.easy_install.egg_distribution() is called for egg_path c:\\program files\\python\\python340\\lib\\site-packages\\setuptools-3.4.4dev-py3.4.egg. This loads the new egg and its data can be seen cached inside zipimport._zip_directory_cache.

[7] pkg_resources.NullProvider._get() is called for path c:\\program files\\python\\python340\\lib\\site-packages\\setuptools-3.4.4dev-py3.4.egg\\EGG-INFO\\entry_points.txt. The provider's loader is a zipimporter related to the newly loaded egg. All goes well, no caches are modified or eggs loaded/unloaded. Output:

#!text
Installing easy_install-3.4-script.py script to C:\Program Files\Python\Python340\Scripts
Installing easy_install-3.4.exe script to C:\Program Files\Python\Python340\Scripts
Installing easy_install-script.py script to C:\Program Files\Python\Python340\Scripts
Installing easy_install.exe script to C:\Program Files\Python\Python340\Scripts

Installed c:\program files\python\python340\lib\site-packages\setuptools-3.4.4dev-py3.4.egg

[8] pkg_resources.NullProvider._get() is called for path c:\\program files\\python\\python340\\lib\\site-packages\\setuptools-3.4.4dev-py3.4.egg\\EGG-INFO\\dependency_links.txt. The provider's loader is a zipimporter related to the newly loaded egg. All goes well, no caches are modified or eggs loaded/unloaded. Output:

#!text
Processing dependencies for setuptools==3.4.4dev

[9] pkg_resources.NullProvider._get() is called for path c:\\program files\\python\\python340\\lib\\site-packages\\setuptools-3.4.4dev-py3.4.egg\\EGG-INFO\\requires.txt. The provider's loader is an old stale zipimporter that should no longer exist. We crash and burn...

#!text
  File "D:\Workplace\Python SetupTools\pkg_resources.py", line 1433, in _get
    return self.loader.get_data(path)
zipimport.ZipImportError: bad local file header in c:\program files\python\python340\lib\site-packages\setuptools-3.4.4dev-py3.4.egg

Hope this helps.

Best regards,
Jurko Gospodnetić

P.S.

I'd appreciate some help from someone more knowledgeable about the internal setuptools workings, e.g.:

• what exactly those pkg_resources.EggMetadata & pkg_resources.NullProvider objects are,
• how they are supposed to be used
• when they are supposed to be created
• who is supposed to cache them
• what is special about how the requires.txt metadata file is read that makes its data be loaded using an old stale cached zipimporter while dependency_links.txt, for example, does not.

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Here's some more info...

The problem seems to come from the requires.txt metadata being read through a pkg_resources.Distribution instance that still has an old pkg_resources.EggMetadata object stored as its ._provider attribute, which in turn has an old zipimport.zipimporter object containing the original (and now no longer existing) egg's content information.

I'll work on tracking this down some more tomorrow... but I have a nagging feeling someone more familiar with the code-base would find all this trivial now that we know that this stale pkg_resources.EggMetadata instance is the problem. :-s

Hope this helps.

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by pje (Bitbucket: pje, GitHub: pje):

---

Yes, it's probably as simple as changing the code here to read:

ws = WorkingSet([])
ws.add(dist)
distros = ws.resolve(
        [requirement], self.local_index, self.easy_install
)

In other words, add the new distribution to the working set used to resolve the requirements, so that it takes precedence over the stale distribution in self.local_index.

In truth, that is actually just a workaround. Really, the problem here is that the earlier line here:

self.local_index.add(dist)

...is broken in the case where there was already a matching distribution, because adding a new distribution with the same version, location, etc. is ignored. Probably the correct fix is to do:

if dist in self.local_index[dist.key]:
    self.local_index.remove(dist)
self.local_index.add(dist)

in order to force the existing copy (if any) to be removed. That should get rid of the stale distribution along with its stale metadata+loader... though it still won't fix the global pkg_resources.working_set entry if there is one. But it should be enough to fix the current bug, anyway.

I'm a bit surprised this whole problem hasn't surfaced before with anything besides setuptools, but I suppose it can really only happen when it's an egg that's loaded by the build/install process, so it's probably limited to self-updates of build tools, and only ones that have requirements of their own defined. So, if this is fixed, it should also solve any similar problems encountered by tools that build on top of setuptools.

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Yup, I can confirm it. The stale pkg_resources.Distribution object comes from the pkg_resources.Environment instance used in easy_install.easy_install.process_distribution(). If I remove the stale distribution from that pkg_resources.Environment before processing the project's requirements, everything works fine.

I'll see how to implement this cleanly and hopefully have a patch for you soon.

It seems better to me to 'clear all the old cached data' when the egg gets replaced (i.e. as has already been done for zipimport._zip_directory_cache & sys.path_importer_cache) instead of doing it lazily like this - only when the new egg gets processed. @pje & @jaraco - what do you guys think?

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

I have prepared some debugging code that uses the CPython garbage collector to detect and report remaining stale zipimport.zipimporter instances. I'll keep it active in my local code-base and, with time, this'll hopefully catch any remaining potential stale references that setuptools could set up but does not in my current use case. Any suggestions on how code such as this should be best inserted into the code-base as a debugging tool available to others as well? Is there some 'debug-mode' flag in setuptools that can be used for this?

So far I've used this code to track down all remaining stale zipimport.zipimporter instances in my use-case and where they are being held.

Basically they are all held in the easy_install command object via three different Environment instances:

1. PthDistributions instance - stored in easy_install.pth_path
2. PackageIndex instance - stored in easy_install.package_index
3. Environment instance - stored in easy_install.local_index

I have not yet tracked down exactly who uses them, what for, and whether that usage may end up using a stale zipimport.zipimporter as we encountered with easy_install.local_index usage in easy_install.process_distribution().

Any hints/ideas on whether they should be cleared + when & how to do that best?

Also, I have not found any problematic references in the 'master working set'. What does that entity represent anyway? It's global and may contain zipimport.zipimporter references so my gut tells me that it should be cleared as well but so far I haven't encountered it as a problem anywhere.

Hope this helps.

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by mtimmsj (Bitbucket: mtimmsj, GitHub: Unknown):

---

I quickly added:

#!python
if dist in self.local_index[dist.key]:
    self.local_index.remove(dist)
self.local_index.add(dist)

The failure I was seeing when installing an egg with the -Z option over a previous egg that was installed without unzipping previously without an uninstall first went away.

[ghost]

Original comment by pje (Bitbucket: pje, GitHub: pje):

---

Actively removing them from the environments sounds like a good idea. The global working set is pkg_resources.working_set, and the stale egg would be in there if it was previously imported from, or had its metadata queried previously. Replacing a distribution in a working set is trickier than removing it from an Environment, though: you have to do something like:

if ws.by_key.get(dist.key)==dist:
    ws.by_key[dist.key] = dist

However, this won't address the problem of what happens if this is an install run in a setuptools.sandbox -- the sandboxing saves the original working set state and restores it afterward. It might be better to update the stale distribution's metadata provider (its _provider attribute) so that the existing instance won't be broken any more -- that would carry through outside the sandbox and fix that as well.

All of this is, in my opinion, optional at this point. The simple fix for now is just to cleanse the environment objects, which can be handled by making a function for removing a dist from an environment, and then calling it on the various environments, at the same point where the other uncaching is taking place.

Until there's a set of steps to actually generate any problems with the working set and/or sandbox situation, the more advanced fixes are probably overkill.

Heck, it might be a good idea to submit a bug report for zipimport, so that in the event of bad local data, it first checked to see if the stat info had changed and fix its internal cache automatically... oh wait... I just thought of something.

A Way To Fix zipimport's Internal Caches

Suppose that, when we remove the entries from the zip cache, we actually set them aside for a moment, then create a new zipimporter for the file. Then, we clear the old entry, and update it with the new entry. Because the cache entries are shared between zipimporters, updating the old entry would automatically fix every existing zipimporter for that zipfile. Something like:

old_entry = zipdircache.pop(path_to_zip)  # remove from cache
zipimport.zipimporter(path_to_zip)    # create new entry
old_entry.clear()
old_entry.update(zipdircache[path_to_zip])  # force-update older zipimporters

(But done for each path in the cache that is the "same" path as path_to_zip, not just the one entry.)

This approach should fix all the use cases, without needing to track down and replace individual distribution objects in every environment and working set, because the way the cache works, the dictionary representing a given zipfile is shared between all existing zipimporter objects. So rewriting the old entry should fix the old zipimporters.

It would require a bit more elaborate change to the uncaching code, as it would need to first pull out any matching entries for the zipfile, saving them in a list, then it would need to create a zipimporter instance for the path, then pull out the new entry and update the old one(s). All in all, this is probably the "right" way to do it, as it should take care of all updating-an-egg-in-place scenarios properly.

(But it's still not that terribly high of a priority, and I'd still be satisfied with the approach of, "take it out of the local_index and wait for actual bug reports for the rest".)

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

pull request #48 includes a quick-fix for this problem + improves the currently implemented zipimporter cache clearing a bit (catches some more stale zipimporter instances to clear).

I hope to look into the suggested cleaner solution soon...

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

For anyone interested - the debug_references branch in the jurko/setuptools repo contains some additional debugging code that makes setuptools display some more information on all the remaining stale zipimporter instances detected in the system.

Hope this helps.

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by philip_thiem (Bitbucket: philip_thiem, GitHub: Unknown):

---

Be sure to do a double check with pypy\win32. Its implementation of _zip_directory_cache is subtly different and it has to do with paths. That and http://bugs.python.org/issue2953 were the primary reasons EggMetaData and ZipProvider does not use it for directory manifests anymore. Looking that this, it doesn't appear to be a python import, the get_data appears to bypassing the manifest and uses the zipimporter loader. Also, https://bitbucket.org/tarek/distribute/issue/27. Correction NullProvider._get() not get_data

[ghost]

Original comment by philip_thiem (Bitbucket: philip_thiem, GitHub: Unknown):

---

Ugh, sorry about that previous post, I've corrected it. After refreshing my memory, pypy uses "/" internally for path separation and cpython contains the results of os.normpath.

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

Why sorry? It gave us additional background info on the problem, so I think we should be thankful. 😄

And.... it's all a mess... and all because zipimporters cache their zip archive directory information, do so as a performance optimization and provide no official way to clear this cache... yuch...

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

zipimporters cache their zip archive directory information, do so as a performance optimization and provide no official way to clear this cache

If that's true, and that blocks this issue, I suggest:

1. File a ticket upstream (CPython) with this defect.
2. Push for a fix in the upstream implementation.
3. Backport the fix into a third party library.
4. Have setuptools use that library if it's available (or fail with a nicer error message if not).

I'll take a look at the pull request 48 later. Do I understand correctly that it implements pje's suggestion and perhaps a bit more?

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

As for the upstream ticket - I have a feeling no one is actively maintaining the zipimport.c module there. I've seen several related issue open already:

• http://bugs.python.org/issue2953
• http://bugs.python.org/issue14678

#!text
I'll take a look at the pull request 48 later. Do I understand correctly
that it implements pje's suggestion and perhaps a bit more?

Yup. Commit 041d281 is the original minimalistic quick-fix that corrects the reproducible failing use-case we had in this issue.

The same pull request also contains two other commits:

• 79778a6 - cleans up some related code
• 86aeadd - makes easy_install.uncache_zipdir() remove more stale zipimporter instances.

Each of them should be easy enough to review by itself, but if having them all in a single pull request presents a problem, I can easily refactor them into separate pull requests.

Hope this helps.

Best regards,
Jurko Gospodnetić

[ghost]

Original comment by jurko (Bitbucket: jurko, GitHub: jurko):

---

An updated zipimport module implementation would allow us to use importlib.invalidate_caches() introduced in Python 3.3:

#!text
Invalidate the internal caches of finders stored at sys.meta_path. If a
finder implements invalidate_caches() then it will be called to perform
the invalidation. This function should be called if any modules are
created/installed while your program is running to guarantee all
finders will notice the new module’s existence.

[ghost]

Original comment by philip_thiem (Bitbucket: philip_thiem, GitHub: Unknown):

---

Yeah these issues are pretty old. Will need the workaround while 2.6 is supported (when fixed and backported to 2.7+ and 3.2+), if zip-importer cache must be fiddled with.

On a side note, I've been working on memoizing build_zipmanifest from #154. I've had some success just cutting the loader out of the equation. It seems that EggMetaData is only used when the is not a directory, if I use the associated filename in the zipmanifest I was able to simply read from the zipfile. After fixing 154, should be able to use the zipinfo objects directly. However, if one was replacing a dependency in a chain, wouldn't one would still need to update the cache or would there be a process boundary? So I'm not sure that would be desirable in the first place (or if thirdparties would be using it) as a more unified attack approach.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

quick-fix #168: avoid using stale cached zipped egg dist info in easy_install

When installing a zipped egg, into a Python environment with a same named zipped
egg already installed, the installation can fail with a zipimport.ZipImportError
complaining about a 'bad local header' encountered in the new zip archive.

This can occur if setuptools loads the original egg for some reason and the two
zip archives have different content. Then if Python attempts to read a file from
the new archive, it will expect it in a location pointed to by the original
archive's directory. This will report an error if zipimport does not encounter
the expected local file start header in the given location.

The mismatch between the two archives can be reproduced by installing the same
setuptools version (prior to this commit and after commit
f40b810acc5f6494735c912a625d647dc2a3c582 that first introduced the requires.txt
metadata information file into the setuptools project) twice from its sources -
which can randomly fail due to the scenario described above. That will package
the zipped egg archive twice, with each of the archives containing slightly
different Python modules. In case this causes any of the compressed modules to
have different size (easy_install.pyc is often the culprit here), then
attempting to read any later file in the zip archive will fail (requires.txt
package metadata file is often the culprit here). A similar scenario can be
reproduced more consistently by manually modifying the setuptools
easy_install.py source file before building the new egg, e.g. by adding some
additional empty lines to its start.

The underlying reason for this problem is setuptools using zipimporter instances
with cached zip archive content directory information from the older zip
archive, even after the old archive has been replaced.

This patch cleans up only one such old zipimporter instance - one referenced via
easy_install command's local_index attribute. That is the one that has been
causing all the currently reported/reproduced installation failures.

A clean solution needs to make certain there are no more zipimporter instances
with stale archive content directory caches left behind after replacing a zipped
egg archive with another. There are currently at least the following known
potential sources for such stale zipimporter instances (all holding references
to Distribution instances that can then hold a reference to a zipimporter
related to their zipped egg archive):
easy_install command attributes:
local_index (Environment with a list of Distributions)
package_index (PackageIndex with a list of Distributions)
pth_file (PthDistributions with a list of Distributions)
global pkg_resources.working_set object (holds a list of Distributions)
imported module's loader attribute (zipimporter instance)
zipimport._zip_directory_cache
sys.path_importer_cache

Further debugging & development note: A complete list of all the currently
active stale zipimporter instances can be read using CPython's gc module and its
object reference introspection functionality (gc.get_objects() &
gc.get_referrers()) from inside the uncache_zipdir() method in the setuptools
easy_install.py module. That is the method called just after the old arhive has
been replaced by the new one and all the stale zipimporter instances were
supposed to have been released.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Following merging of Pull Request 48, I've created #202 to follow up with the suggestions of a more robust cache invalidation.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Big thanks to @pje and @jurko for all the help on this issue!

[ghost]

Original comment by languitar (Bitbucket: languitar, GitHub: languitar):

---

What is the state of this issue with respect to released versions of setuptools? In which version is the fix included?

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

It was fixed in 3.5.2. See the history for other details about released versions.