Expand zipimport to include other compression methods

[rhettinger]

BPO	17004
Nosy	@rhettinger, @gpshead, @pitrou, @briancurtin, @ericsnowcurrently, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-01-20.18:41:42.824>
labels = ['type-feature', 'library', '3.11']
title = 'Expand zipimport to include other compression methods'
updated_at = <Date 2022-04-06.03:00:34.782>
user = 'https://github.com/rhettinger'

bugs.python.org fields:

activity = <Date 2022-04-06.03:00:34.782>
actor = 'yan12125'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2013-01-20.18:41:42.824>
creator = 'rhettinger'
dependencies = []
files = []
hgrepos = []
issue_num = 17004
keywords = []
message_count = 11.0
messages = ['180307', '180310', '180311', '180313', '180314', '180323', '180324', '180347', '220589', '267527', '325729']
nosy_count = 8.0
nosy_names = ['rhettinger', 'gregory.p.smith', 'pitrou', 'nadeem.vawda', 'brian.curtin', 'eric.snow', 'serhiy.storchaka', 'superluser']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17004'
versions = ['Python 3.11']

Linked PRs

• gh-61206: Support zstandard compression in the zipimport module #138254

[rhettinger]

Only a little of the existing logic is tied to the zipfile format. Consider adding support for xz, tar, tar.gz, tar.bz2, etc.

In particular, xz has better compression, resulting in both space savings and faster load times.

[serhiy-storchaka]

tar.* is not a good choice because it doesn't allow random access. Bare tar better than zip only in case when you need to save additional file attributes (Unix file access mode, times, owner, group, links). ZIP format supports all this too, but not zipfile module yet.

Adding bz2 or lzma compression to ZIP file shouldn't be too hard.

[serhiy-storchaka]

Here are some tests.

time 7z a -tzip -mx=0 python-0.zip $(find Lib -type f -name '*.py') >/dev/null
time 7z a -tzip python.zip $(find Lib -type f -name '*.py') >/dev/null
time 7z a -tzip -mx=9 python-9.zip $(find Lib -type f -name '*.py') >/dev/null
time 7z a -tzip -mm=bzip2 python-bzip2.zip $(find Lib -type f -name '*.py') >/dev/null
time 7z a -tzip -mm=bzip2 -mx=9 python-bzip2-9.zip $(find Lib -type f -name '*.py') >/dev/null
time 7z a -tzip -mm=lzma python-lzma.zip $(find Lib -type f -name '*.py') >/dev/null
time 7z a -tzip -mm=lzma -mx=9 python-lzma-9.zip $(find Lib -type f -name '*.py') >/dev/null
time 7z t python-0.zip >/dev/null
time 7z t python.zip >/dev/null
time 7z t python-9.zip >/dev/null
time 7z t python-bzip2.zip >/dev/null
time 7z t python-bzip2-9.zip >/dev/null
time 7z t python-lzma >/dev/null
time 7z t python-lzma.zip >/dev/null
time 7z t python-lzma-9.zip >/dev/null
wc -c python*.zip

Results:

         pack* unpack   size
         time   time    (MB)

store 0.5 0.2 19.42
deflate 6 0.4 4.59
deflate-max 40 0.4 4.52
bzip2 6 2.1 4.45
bzip2-max 79 2.0 4.39
lzma 37 0.7 4.42
lzma-max 62 0.7 4.39

*) For pack time I take user time because 7-zip well parallelize deflate and bzip2 compression.

As you can see, a size difference between maximal compression with different methods only 3%. lzma decompress almost twice slower then deflate, and bzip2 decompress 5 times slower. Python files are too small to get benefit from advanced compression.

[pitrou]

Here are some tests.

I think you want to put pyc files in the zip file as well.

[rhettinger]

xz will likely be the best win -- it is purported to compress smaller than bz2 while retaining the decompression speed of zip.

As Antoine says, the usual practice is to add py, pyc, and pyo files to the compressed library; otherwise, there is an added cost with Python tries to write a missing pyc/pyo file.

[serhiy-storchaka]

Well.

./python -m compileall $(find Lib -type f -name '*.py')
./python -O -m compileall $(find Lib -type f -name '*.py')

Tests:

FILES="$(find Lib -name '*.py' -o -name '*.py[co]')"
time 7z a -tzip -mx=0 python-0.zip $FILES >/dev/null
time 7z a -tzip python.zip $FILES >/dev/null
time 7z a -tzip -mx=9 python-9.zip $FILES >/dev/null
time 7z a -tzip -mm=bzip2 python-bzip2.zip $FILES >/dev/null
time 7z a -tzip -mm=bzip2 -mx=9 python-bzip2-9.zip $FILES >/dev/null
time 7z a -tzip -mm=lzma python-lzma.zip $FILES >/dev/null
time 7z a -tzip -mm=lzma -mx=9 python-lzma-9.zip $FILES >/dev/null
time 7z t python-0.zip >/dev/null
time 7z t python.zip >/dev/null
time 7z t python-9.zip >/dev/null
time 7z t python-bzip2.zip >/dev/null
time 7z t python-bzip2-9.zip >/dev/null
time 7z t python-lzma.zip >/dev/null
time 7z t python-lzma-9.zip >/dev/null
wc -c python*.zip

Results:

         pack  unpack   size
         time   time    (MB)

store 1.6 0.5 65.4
deflate 19 0.9 17.5
deflate-max 134 0.9 17.2
bzip2 21 4.2 16.5
bzip2-max 294 4.1 16.3
lzma 120 2.3 15.9
lzma-max 204 2.3 15.8

All numbers are about 3x larger. lzma-max is 8% less than deflate-max but 2.5 times slower. Bzip2 is out of the game.

[pitrou]

Agreed it doesn't look very promising.

[brettcannon]

So this seems like a confluence of both supporting compressed files for loading source code as well as supporting new archive formats (e.g. xz vs. tar); zip just happens to do both implicitly. And there is also the question of if you explicitly plan to do this in C code or in pure Python as I plan to introduce a pure Python version of zipimport into importlib for 3.4 so that it can use zipfile directly and thus all of its full support of zipfile abilities.

And there doesn't have to be any performance cost in trying to write bytecode files; it's very simple to have a loader which simply skips that step entirely.

[ericsnowcurrently]

related: issue bpo-17630 and issue bpo-5950

[yan12125]

+1 for that. I like XZ support so that our application size can be reduced.

[serhiy-storchaka]

zipimport has been rewritten in pure Python (bpo-25711). Now it is easier to add support of other compression methods. Although I don't think that reducing the size by 3-8% is worth complicating the code.

If you still need this, I think that the simplest way is importing the zipfile module and monkey patching the simple ZIP file implementation in the zipimport module with zipfile-based implementation. This can be made only after importing zipfile itself, i.e. in case of zipping the stdlib, the zipfile module and its dependencies should be stored uncompressed or with the deflate compression.