make zipfile.PyZipFile more usable

[ctismer]

BPO	19274
Nosy	@birkenfeld, @bitdancer, @serhiy-storchaka, @ctismer
Files	zipfile.diff: small addition to zipfile that makes it useful for the python library make_libzip.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-10-21.02:01:06.974>
created_at = <Date 2013-10-16.21:43:52.085>
labels = ['type-feature', 'library']
title = 'make zipfile.PyZipFile more usable'
updated_at = <Date 2014-03-06.15:09:34.175>
user = 'https://github.com/ctismer'

bugs.python.org fields:

activity = <Date 2014-03-06.15:09:34.175>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2013-10-21.02:01:06.974>
closer = 'Christian.Tismer'
components = ['Library (Lib)']
creation = <Date 2013-10-16.21:43:52.085>
creator = 'Christian.Tismer'
dependencies = []
files = ['32150', '32151']
hgrepos = []
issue_num = 19274
keywords = ['patch']
message_count = 17.0
messages = ['200088', '200089', '200112', '200269', '200275', '200279', '200282', '200290', '200684', '200701', '200702', '200703', '200750', '200856', '200859', '212762', '212810']
nosy_count = 5.0
nosy_names = ['georg.brandl', 'r.david.murray', 'python-dev', 'serhiy.storchaka', 'Christian.Tismer']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue19274'
versions = ['Python 3.4']

[ctismer]

zipfile.PyZipFile needs a filter function.

Reason:
When creating an archive of the python lib, we don't want the tests.
Especially the test file "badsyntax_future3.py" does not compile.

Use case:

With this little addition, it becomes very easy to create a zip file
of the whole python library. See the attached use case.

[ctismer]

Here is my use case as an example.
With this patch above, I can easily create a .zip file of the standard lib.

This was no longer possible at all, after revision 17558,
from 2001-04-18:

"""This is a test"""
from __future__ import nested_scopes
from __future__ import rested_snopes

def f(x):
    def g(y):
        return x + y
    return g

print f(2)(4)

[birkenfeld]

Looks fine but -- as a new feature -- is 3.4 only, and needs docs and tests.

[ctismer]

Hi Georg,
So do you think it is ok this way?
I was not sure if extending the function with an optional
arg is ok, or if a method to configure PyZipFile would be better.
At the moment I just needed the simple functionality.
Should it maybe get a regex like compileall.py ?

And a general question:
What is the right constraint for a filter function?
As I wrote it, returning nothing would simply be treated as "False".
Maybe it is better to enforce a return value which explicitly forbids
to be just None, which often just means "I forgot the return value" ?

About feature or fix: Well, I need this for python2.7, because without
it, the whole purpose of PyZipFile is pretty questionable. I might argue
it a fix, because it crashes on the standard library ;-)

Anyway, tell me and I'l add test, docs and put it into dev.

cheers - chris

[serhiy-storchaka]

I don't think this is needed. You can walk a tree and call writepy() for files and directories which you want.

[ctismer]

@serhiy.storchaka

I don't think this is needed. You can walk a tree and call writepy()
for files and directories which you want.

What exactly do mean by "this" and "needed"?
I cannot see the connection of my initial post and your reply.

Running PyZipFile on a package dir of the standard lib _does_ traverse
the tree, and there is no way to stop it from doing that.
That was the whole point of the issue.
Please correct me if I'm missing something.

[serhiy-storchaka]

"this" is a filter function. "Not needed" means that you can got what you want without adding a filter function to zipfile.PyZipFile. Just don't call writepy() on directories which contains files which shouldn't be zipped.

[ctismer]

Ah, I understand:

The case that does not compile comes from the toplevel "test" folder,
which I could have excluded explicitly.

But it is not a complete solution:
If I want to add every package from the standard lib, then I necessarily
encounter enclosed test folders, for instance:

[Lib/unittest](https://github.com/python/cpython/blob/main/Lib/unittest)

contains

[Lib/unittest/test](https://github.com/python/cpython/blob/main/Lib/unittest/test)

Your hint to just not call writepy() on directories which contain files
which shouldn't be zipped means that I cannot use writepy at all,
because many library packages contain tests.
But that is the only purpose of class PyZipFile, therefore the patch.

[python-dev]

New changeset 34fb83421119 by Christian Tismer in branch 'default':
add a filterfunc to zip file.PyZipFile.writepy, bpo-19274
http://hg.python.org/cpython/rev/34fb83421119

[birkenfeld]

Hi Chris, your commit is a bit hard to review due to all the unrelated spacing changes. I assume this is done automatically by your editor? It's probably best to switch off that feature for CPython development :)

[birkenfeld]

While reviewing: is it intended that the filter is only called for directories and not for individual files?

[python-dev]

New changeset 2d39b3555951 by Georg Brandl in branch 'default':
bpo-19274: use captured_stdout() in the test suite; add NEWS entry.
http://hg.python.org/cpython/rev/2d39b3555951

[ctismer]

ah, I just see that.
The problem was that the checkin drove me nuts. It forced me to
run reindent to normalize the code. I did that with my WindIde
editor, but this did not help.
The point was: Actually an end-of-line was missing at the end of
the files.
Sorry, I did not see that at all, because the indentation changes
are at the end. I usually avoid this strictly. It was just the check-in
rejection...

[ctismer]

@georg:

While reviewing: is it intended that the filter is only called for directories and not for individual files?

Not really. I will add this, later. Just wanted to see if this makes
sense and it's worth the effort to extend it.

[ctismer]

added that with tests.

[bitdancer]

For future reference, the update Christian refers to in the previous message is 4f1121ae1cb5.

[python-dev]

New changeset 064ee489982e by R David Murray in branch 'default':
whatsnew: improve PyZipFile filterfuc entry, and its docs (bpo-19274).
http://hg.python.org/cpython/rev/064ee489982e