distutils’ build_py fails when package string is unicode

[patrickandrew]

BPO	13943
Nosy	@tarekziade, @merwok, @Lothiraldan, @zooba
Files	patch-Lib-distutils-command-build_py.py: build_py package name support patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/merwok'
closed_at = <Date 2021-02-03.18:30:36.302>
created_at = <Date 2012-02-04.20:41:07.685>
labels = ['easy', 'type-bug', 'library']
title = 'distutils\xe2\x80\x99 build_py fails when package string is unicode'
updated_at = <Date 2021-02-03.18:30:36.301>
user = 'https://bugs.python.org/patrickandrew'

bugs.python.org fields:

activity = <Date 2021-02-03.18:30:36.301>
actor = 'steve.dower'
assignee = 'eric.araujo'
closed = True
closed_date = <Date 2021-02-03.18:30:36.302>
closer = 'steve.dower'
components = ['Distutils']
creation = <Date 2012-02-04.20:41:07.685>
creator = 'patrick.andrew'
dependencies = []
files = ['24422']
hgrepos = []
issue_num = 13943
keywords = ['easy']
message_count = 7.0
messages = ['152643', '152671', '152715', '207230', '207236', '207840', '386427']
nosy_count = 5.0
nosy_names = ['tarek', 'eric.araujo', 'Boris.FELD', 'patrick.andrew', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13943'
versions = ['Python 2.7']

[patrickandrew]

When a package list is built using Unicode strings, distutils fails to build the package with a TypeError.

This patch alternatively checks for 'unicode' as the instance type and also prints the type in the type error for easier debugging.

[merwok]

Thanks for the report and patch. I think distutils was not written with Unicode in mind, or maybe even before Python had a unicode type. Technically, http://docs.python.org/distutils/setupscript#additional-meta-data says that unicode is not allowed for metadata fields (nothing is said about py_modules, packages and the like), but we’ve fixed a couple of bugs related to unicode, so I think this is a reasonable request. Can you post (part of) your failing setup script?

[patrickandrew]

From py-logilab-common 0.57.1 port for FreeBSD. No patches applied:

package init file './test/__init__.py' not found (or not a regular file)
Traceback (most recent call last):
  File "setup.py", line 170, in <module>
    install()
  File "setup.py", line 166, in install
    **kwargs
  File "/usr/local/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python2.7/distutils/command/build.py", line 127, in run
    self.run_command(cmd_name)
  File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python2.7/distutils/command/build_py.py", line 93, in run
    self.build_packages()
  File "/usr/local/lib/python2.7/distutils/command/build_py.py", line 372, in build_packages
    self.build_module(module, module_file, package)
  File "/usr/local/lib/python2.7/distutils/command/build_py.py", line 333, in build_module
    "'package' must be a string (dot-separated), list, or tuple")
TypeError: 'package' must be a string (dot-separated), list, or tuple
*** Error code 1

Stop in /usr/src/ports/devel/py-logilab-common.

This package's setup.py is auto-generating the packages list with the current working directory.

def get_packages(directory, prefix):
    """return a list of subpackages for the given directory"""
    result = []
    for package in os.listdir(directory):
        absfile = join(directory, package)
        if isdir(absfile):
            if exists(join(absfile, '__init__.py')) or \
                   package in ('test', 'tests'):
                if prefix:
                    result.append('%s.%s' % (prefix, package))
                else:
                    result.append(package)
                result += get_packages(absfile, result[-1])
    return result

...

packages = [modname] + get_packages(os.getcwd(), modname)

kwargs['packages'] = packages

setup(...
    **kwargs)

where modname is imported from __pkginfo__.py:
distname = 'logilab-common'
modname = 'common'

What's interesting is there is no explicit unicode string definition within this package list gerneration, yet the final packages list looks like:

['logilab.common', u'logilab.common.test', u'logilab.common.test.data', u'logilab.common.test.data.find_test', u'logilab.common.ureports']

[Lothiraldan]

I've the same problem today with package https://pypi.python.org/pypi/httpretty/0.7.1 but only when I try to install one of my project which requires httpretty, if I try to install it directly it works like a charm.

pip install httpretty -> works
pip install mypkg -> doesn't works

Looks like HTTPretty is using __file__ variable in setup.py (https://github.com/gabrielfalcao/HTTPretty/blob/master/setup.py#L35) and pip seems to pass the file as unicode:

http://0bin.net/paste/dQfsSAmguWNYyY7w#0O/gcrWA44wKicfTdsGT4KqRYhbZLyhN9BUXNQD1XZA=

At the last line: "__file__=u'/home/lothiraldan/.virtualenvs/test/build/httpretty/setup.py'"

[merwok]

It’s strange that this would happen when installing as a dependency and not when installing directly. Pip can change faster than stdlib is released, could you report the bug to them and see if it’s possible to pass __file__ as a byte string?

[Lothiraldan]

An issue has been opened in pip repository: pypa/pip#1441

[zooba]

Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools