distutils make_zipfile uses random order

[SomberNight]

BPO	40963
Nosy	@rhettinger, @merwok, @zooba, @dstufft, @SomberNight

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-02-03.18:08:04.642>
created_at = <Date 2020-06-12.17:30:20.940>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.7', 'library']
title = 'distutils make_zipfile uses random order'
updated_at = <Date 2021-02-03.18:08:04.641>
user = 'https://github.com/SomberNight'

bugs.python.org fields:

activity = <Date 2021-02-03.18:08:04.641>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2021-02-03.18:08:04.642>
closer = 'steve.dower'
components = ['Distutils']
creation = <Date 2020-06-12.17:30:20.940>
creator = 'ghost43'
dependencies = []
files = []
hgrepos = []
issue_num = 40963
keywords = []
message_count = 3.0
messages = ['371400', '371515', '386270']
nosy_count = 5.0
nosy_names = ['rhettinger', 'eric.araujo', 'steve.dower', 'dstufft', 'ghost43']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40963'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

[SomberNight]

I am trying to generate .zip sdists for a project in a reproducible manner, using setuptoools.
The generated zips differ in the order of packed files.

The root cause of the non-determinicity is using os.walk() in make_zipfile here:

cpython/Lib/distutils/archive_util.py

Line 174 in 0d3350d

for dirpath, dirnames, filenames in os.walk(base_dir):

For a potential fix, see pypa/setuptools@2968882

I guess https://bugs.python.org/issue30693 is sort of related. The change made there is necessary, and was sufficient to make the tars reproducible but not the zips.

(sidenote: Is it acceptable to sign the PSF CLA with a pseudonym?)

[rhettinger]

+1 for sorting filenames prior to insertion

[zooba]

Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools