installing coverage with python 3.5.2 on windows fails with UnicodeDecodeError

[tkhyn]

Hi. I've just noticed that installing coverage on windows using python 3.5 and easy_install raised a UnicodeDecodeError. I am not able to determine if it comes from coverage (bad encoding of package info? is that possible?) or setuptools itself. I have included what I have used to make the installation complete, but I have no idea if it would be the solution ...

Please tell me if it's something I should rather report on the coverage side.

To reproduce, use this on a Windows machine (I've tried on a Debian VM and it worked) with python 3.5.2 and setuptools 28.6.0:

easy_install coverage

Expected: coverage is installed correctly

Happens:

Searching for coverage
Reading https://pypi.python.org/simple/coverage/
Best match: coverage 4.2
Downloading https://pypi.python.org/packages/67/2c/94a667aed32f91fa7b9279e224da65afa32b666ace5f07bc79f9b38adb06/coverage-4.2.win-amd64-py3.5.exe#md5=2532041888c8a35d961314996619fca3
Processing coverage-4.2.win-amd64-py3.5.exe
creating 'C:\Users\Thomas\AppData\Local\Temp\easy_install-vpji91r0\coverage-4.2-py3.5-win-amd64.egg' and adding 'C:\Users\Thomas\AppData\Local\Temp\easy_install-vpji91r0\coverage-4.2-py3.5-win-amd64.egg.tmp' to it
creating C:\Python\3.5\Lib\site-packages\coverage-4.2-py3.5-win-amd64.egg
Extracting coverage-4.2-py3.5-win-amd64.egg to C:\Python\3.5\Lib\site-packages
Adding coverage 4.2 to easy-install.pth file
Installing coverage-3.5-script.py script to d:\dev\.env\venv\buildout3\Scripts
Traceback (most recent call last):
  File "C:\Python\3.5\Lib\runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python\3.5\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python\3.5\easy_install.exe\__main__.py", line 9, in <module>
  File "C:\Python\3.5\Lib\site-packages\setuptools\command\easy_install.py", line 2245, in main
    distclass=DistributionWithoutHelpCommands, **kw
  File "C:\Python\3.5\Lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Python\3.5\Lib\distutils\dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "C:\Python\3.5\Lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\Python\3.5\Lib\site-packages\setuptools\command\easy_install.py", line 380, in run
    self.easy_install(spec, not self.no_deps)
  File "C:\Python\3.5\Lib\site-packages\setuptools\command\easy_install.py", line 629, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "C:\Python\3.5\Lib\site-packages\setuptools\command\easy_install.py", line 661, in install_item
    self.process_distribution(spec, dist, deps)
  File "C:\Python\3.5\Lib\site-packages\setuptools\command\easy_install.py", line 686, in process_distribution
    self.install_egg_scripts(dist)
  File "C:\Python\3.5\Lib\site-packages\setuptools\command\easy_install.py", line 562, in install_egg_scripts
    dist.get_metadata('scripts/' + script_name)
  File "C:\Python\3.5\Lib\site-packages\pkg_resources\__init__.py", line 1622, in get_metadata
    return self._get(self._fn(self.egg_info, name)).decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte

Workaround: change pkg_resources__init__.py in get_metadata from:

def get_metadata(self, name):
    if not self.egg_info:
        return ""
    value = self._get(self._fn(self.egg_info, name))
    return value.decode('utf-8') if six.PY3 else value

to:

def get_metadata(self, name):
    if not self.egg_info:
        return ""
    value = self._get(self._fn(self.egg_info, name))
    try:
        return value.decode('utf-8')
    except UnicodeDecodeError:
        return value

[jaraco]

The UnicodeDecodeError is coming from the fact that there are binary executable files in the egg, but the code seems to be assuming the metadata is always text. In this case, the script_name is coverage-3.5.exe. I'm not sure this is supposed to work; how can setuptools know if an arbitrary file (metadata) is meant to be a binary file or text?

To make matters more complicated, easy_install is (soft) deprecated in favor of pip, eggs are deprecated in favor of wheels (not yet implemented in setuptools), and distlib is the preferred mechanism for generating executable wrappers, but not yet used by setuptools.

What happens if you instead install the package using pip? My guess is it will ignore the .exe installer and instead grab the source package and rebuild from that. If that works for you, that would be my recommendation.

[jaraco]

Looks like there's a failing test capturing this failure, indicated in #594.

[tkhyn]

@jaraco thanks for the hints and info. Sorry for not seeing the existing issue, I did try and search for similar problems though.

It indeed works using pip (as it fetches the wheel distribution).

The thing is I'm using buildout to install the scripts for nearly all my projects, and buildout / zc.recipe.egg rely on easy_install. That's the reason why I have opened the issue. I never use easy_install in virtualenvs, but I don't have the choice in my buildout envs.

Would the proposed workaround of catching the UnicodeDecodeError exception instead of checking the python version in get_metadata have side-effects?

[jaraco]

Would the proposed workaround have side-effects?

Yes, probably. It would be bad form to allow a function to return both bytes and text objects. Actually, that's what's wrong with the existing implementation. It's relying on the lenience of Python 2 to allow bytes and text objects to be interchangeable.

Looking further down the chain, there's more and more about install_script that expects the result of that get_metadata call to be text (even ascii text). Did you find applying that workaround would work for you?

[tkhyn]

Yes, that workaround made things work for me on Python 3.5.x and all the packages in my buildout environments were installed normally without errors.

Sorry I really don't know enough about setuptools' internals to emit an informed opinion on how get_metadata should behave and offer a bullet-proof fix, all I can say is the workaround works for me!