Distutils: ZIP files don't include directory entries

[serhiy-storchaka]

BPO	34738
Nosy	@merwok, @serhiy-storchaka, @dstufft, @miss-islington
PRs	bpo-34738: Add directory entries in ZIP files created by distutils. #9419 [3.7] bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419) #10941 [3.6] bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419) #10942 [2.7] bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419). #10950

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-12-05.22:10:17.851>
created_at = <Date 2018-09-19.13:47:27.502>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = "Distutils: ZIP files don't include directory entries"
updated_at = <Date 2018-12-05.22:10:17.851>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-12-05.22:10:17.851>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-12-05.22:10:17.851>
closer = 'serhiy.storchaka'
components = ['Distutils']
creation = <Date 2018-09-19.13:47:27.502>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 34738
keywords = ['patch']
message_count = 9.0
messages = ['325761', '325776', '325777', '325778', '331140', '331150', '331156', '331162', '331179']
nosy_count = 5.0
nosy_names = ['eric.araujo', 'mrabarnett', 'serhiy.storchaka', 'dstufft', 'miss-islington']
pr_nums = ['9419', '10941', '10942', '10950']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34738'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[serhiy-storchaka]

ZIP files created by distutils don't include entries for directories. For example, if the directory structure is

dist/
    [file1](https://bugs.python.org/file1).py
    sub/
        [file2](https://bugs.python.org/file2).py

the ZIP file will contain entries 'dist/file1.py' and 'dist/sub/file2.py', but not 'dist/' and 'dist/sub/'.

This only when use the zipfile module. If it is not available for some reasons, the zip command is used. In that case directory entries are created. The are created also in TAR files.

All other methods of packing a tree to a ZIP in the stdlib (shutil.make_archive(), zipfile CLI, zipapp, Tools/msi/make_zip.py) create directory entries too.

zipimport doesn't support implicit namespace packages if directory entries are not added (bpo-14905).

The proposed PR makes distutils adding directory entries in a ZIP file.

[mrabarnett]

I don't see a problem with this. If the zip file has 'dist/file1.py' then you know to create a directory when unzipping. If you want to indicate that there's an empty directory 'foo', then put 'foo/' in the zip file.

[serhiy-storchaka]

There are two problems:

1. This is inconsistent with TAR files and the zip utility.

2. This is not compatible with zipimport.

[merwok]

IMO inconsistency with tarfile is not very relevant.

About the zip utility: are you saying that the unix unzip command is not able to inspect or extract distutils sdists?

About zipimport, the question is: are sdists meant to be directly importable (IMO no).

That said, if it is a minor change with no downsides, I wouldn’t oppose it. I just wanted to be clear on the reasons / benefits.

[serhiy-storchaka]

The inconsistency is not just with the tarfile module, but with other method of creating ZIP archives.

The unix unzip command creates directories if needed, but I am not sure about other third-party utilities. Since all know ZIP creators creates entries for directories, I would not be surprised if there are ZIP readers that doesn't work without explicit entries for directories. Zipimport is an example of this.

Matthew mentioned other case. Empty directories currently are added to the TAR archive, but are omitted in the ZIP archive (unless you use an external zip tool). TAR archives are common on Unix, ZIP archives are common on Windows, and we can get different source distributions for Unix and Windows. You need to add a fake empty file to be sure that the directory will be added in the distribution. Some old DOS archives needed this trick, but the ZIP format supports empty directories.

The downside of adding entries for directories is that this increases the size of the archive. But the difference is pretty small.

[serhiy-storchaka]

New changeset 67a93b3 by Serhiy Storchaka in branch 'master':
bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419)
67a93b3

[miss-islington]

New changeset 53bed18 by Miss Islington (bot) in branch '3.7':
bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419)
53bed18

[serhiy-storchaka]

New changeset e0c2046 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419) (GH-10942)
e0c2046

[serhiy-storchaka]

New changeset b2742ba by Serhiy Storchaka in branch '2.7':
[2.7] bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419). (GH-10950)
b2742ba