Port distutils as found in Python 2.7 to Python 3.x.

[jaraco]

BPO	19544
Nosy	@akuchling, @jaraco, @vstinner, @tarekziade, @ned-deily, @merwok
Files	ticket-list.txt python_2.7_distutil_commits.html

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/jaraco'
closed_at = <Date 2013-11-16.00:51:55.685>
created_at = <Date 2013-11-10.16:59:52.486>
labels = ['type-bug', 'library']
title = 'Port distutils as found in Python 2.7 to Python 3.x.'
updated_at = <Date 2013-12-30.21:09:27.773>
user = 'https://github.com/jaraco'

bugs.python.org fields:

activity = <Date 2013-12-30.21:09:27.773>
actor = 'python-dev'
assignee = 'jaraco'
closed = True
closed_date = <Date 2013-11-16.00:51:55.685>
closer = 'jaraco'
components = ['Distutils']
creation = <Date 2013-11-10.16:59:52.486>
creator = 'jaraco'
dependencies = []
files = ['32568', '32569']
hgrepos = []
issue_num = 19544
keywords = []
message_count = 20.0
messages = ['202534', '202571', '202572', '202574', '202580', '202585', '202587', '202590', '202592', '202597', '202966', '202973', '202975', '202979', '202981', '202986', '202989', '202997', '207109', '207113']
nosy_count = 7.0
nosy_names = ['akuchling', 'jaraco', 'vstinner', 'tarek', 'ned.deily', 'eric.araujo', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19544'
versions = ['Python 3.4']

[jaraco]

Following from bpo-7457, in which a single feature was identified to have gone missing in 29a3eda89995, this ticket captures the need to bring the Python 3 codebase up to match Python 2.7.

[akuchling]

I went through Python 2.7's Misc/NEWS file and collected the entries for Distutils-related issues that were applied. Perhaps we can check the individual entries on this list, and see which ones are still present in Python 3.x and which ones got reverted.

[ned-deily]

I wouldn't trust the NEWS items. I think the only reliable thing to do is diff each file, unfortunately.

[akuchling]

Issue bpo-11104 also made some functionality work in both 2.7 and 3.2, though it's not clear that the problem stemmed from the distutils2 revert.

[jaraco]

After spending several hours spelunking, we identified what we believe are the tickets that were backed out in the aforementioned reversion.

bpo-1180
bpo-6516
bpo-7457
bpo-6466
bpo-6286

Additionally, bpo-6377 (renaming .compiler to .compiler_obj) was reverted, but it likely should not be re-applied.

Attached is an export of the etherpad (http://beta.etherpad.org/p/python_2.7_distutil_commits) which we used to keep track of the changes and show our work.

We will flag the above tickets and address each individually.

[python-dev]

New changeset e19441e540ca by Jason R. Coombs in branch '3.3':
bpo-19544 and Issue bpo-7457: Restore the read_pkg_file method to distutils.dist.DistributionMetadata accidentally removed in the undo of distutils2.
http://hg.python.org/cpython/rev/e19441e540ca

New changeset 28059d8b395b by Jason R. Coombs in branch 'default':
Merge with 3.3 for Issue bpo-19544 and Issue bpo-7457
http://hg.python.org/cpython/rev/28059d8b395b

[akuchling]

Patches for the default branch have been added to bpo-1180 (option to ignore ~/.pydistutils.cfg) and bpo-6516 (setting the owner/group in Distutils-built tarballs). Please double-check those patches; I can apply them.

[python-dev]

New changeset 5e98c4e9c909 by Jason R. Coombs in branch '3.3':
Issue bpo-19544 and Issue bpo-6286: Restore use of urllib over http allowing use of http_proxy for Distutils upload command, a feature accidentally lost in the rollback of distutils2.
http://hg.python.org/cpython/rev/5e98c4e9c909

New changeset b1244046f37a by Jason R. Coombs in branch 'default':
Merge with 3.3 for Issue bpo-19544 and Issue bpo-6286. Merge is untested. I was unable to test due to bab0cbf86835.
http://hg.python.org/cpython/rev/b1244046f37a

[ned-deily]

b1244046f37a appears to have broken buildbots. See, for example:

http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/2984

[jaraco]

Thanks Ned. I did see that and have pushed 394ed9deebd4. I believe that corrects the only test failure.

[python-dev]

New changeset b9c9c4b2effe by Andrew Kuchling in branch 'default':
Issue bpo-19544 and Issue bpo-6516: Restore support for --user and --group parameters to sdist command as found in Python 2.7 and originally slated for Python 3.2 but accidentally rolled back as part of the distutils2 rollback. Closes Issue bpo-6516.
http://hg.python.org/cpython/rev/b9c9c4b2effe

[python-dev]

New changeset b08868fd5994 by Christian Heimes in branch 'default':
Issue bpo-19544 and Issue bpo-6516: quick workaround for failing builds
http://hg.python.org/cpython/rev/b08868fd5994

[python-dev]

New changeset 015463176d2e by Victor Stinner in branch 'default':
Issue bpo-19544, bpo-6516: no need to catch AttributeError on import pwd/grp
http://hg.python.org/cpython/rev/015463176d2e

[vstinner]

At changeset 015463176d2e2530e4f07cfbe97e41abac540a57, test_make_distribution_owner_group() was failing on some buildbots. test_distutils works fine on my Linux box, I ran the test as my haypo user and as root.

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%203.x/builds/5780/steps/test/logs/stdio

======================================================================
FAIL: test_make_distribution_owner_group (distutils.tests.test_sdist.SDistTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/distutils/tests/test_sdist.py", line 477, in test_make_distribution_owner_group
    self.assertEquals(member.gid, os.getgid())
AssertionError: 0 != 1002

http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/470/steps/test/logs/stdio

======================================================================
FAIL: test_make_distribution_owner_group (distutils.tests.test_sdist.SDistTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/distutils/tests/test_sdist.py", line 477, in test_make_distribution_owner_group
    self.assertEquals(member.gid, os.getgid())
AssertionError: 0 != 20

[python-dev]

New changeset c35311fcc967 by Andrew Kuchling in branch 'default':
Issue bpo-19544 and Issue bpo-1180: Restore global option to ignore ~/.pydistutils.cfg in Distutils, accidentally removed in backout of distutils2 changes.
http://hg.python.org/cpython/rev/c35311fcc967

[vstinner]

At changeset 015463176d2e2530e4f07cfbe97e41abac540a57, test_make_distribution_owner_group() was failing on some buildbots.

The problem is that tempfile.mkdtemp() creates a directory with the
group 0. Files created in this directory also have the group 0.

The test uses os.getpid() to get the group identifier.

The test should use os.stat(self.tmp_dir).st_gid (and st_uid) instead
of os.getgid(), or the tmp_dir directory. Or the group of the
temporary directory should be changed to os.getgid().

I don't know if distutils should create tarball with the group 0 or
the group of the current user.

[jaraco]

Thanks Victor.

Yes, it appears that there's yet another unported issue bpo-7408, a follow-up to bpo-6516.

[jaraco]

I believe all identified issues have been ported/fixed.

[python-dev]

New changeset 83f12a9593db by Zachary Ware in branch 'default':
Issue bpo-19544, bpo-6516: check ZLIB_SUPPORT, not zlib (which might not be bound)
http://hg.python.org/cpython/rev/83f12a9593db

[python-dev]

New changeset 2a872126f4a1 by Zachary Ware in branch 'default':
Issue bpo-19544, bpo-6516: check ZLIB_SUPPORT, not zlib (which might not be bound)
http://hg.python.org/cpython/rev/2a872126f4a1