PEP 441 - Improving Python Zip Application Support

[pfmoore]

BPO	23491
Nosy	@brettcannon, @pfmoore, @dholth, @ethanfurman, @berkerpeksag, @JimJJewett, @serhiy-storchaka, @zooba
Files	pep441.patch pep441.patch pep-441.patch pep-441.v4.patch pep-441.v5.patch pep-441.v6.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2015-03-13.14:45:26.083>
created_at = <Date 2015-02-20.12:52:51.249>
labels = ['type-feature', 'library']
title = 'PEP 441 - Improving Python Zip Application Support'
updated_at = <Date 2015-03-13.15:06:38.893>
user = 'https://github.com/pfmoore'

bugs.python.org fields:

activity = <Date 2015-03-13.15:06:38.893>
actor = 'paul.moore'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2015-03-13.14:45:26.083>
closer = 'brett.cannon'
components = ['Library (Lib)']
creation = <Date 2015-02-20.12:52:51.249>
creator = 'paul.moore'
dependencies = []
files = ['38187', '38192', '38250', '38258', '38389', '38419']
hgrepos = []
issue_num = 23491
keywords = ['patch']
message_count = 42.0
messages = ['236276', '236285', '236286', '236291', '236301', '236308', '236455', '236457', '236458', '236459', '236531', '236534', '236669', '236670', '236672', '236673', '236675', '236676', '236678', '236680', '236682', '236688', '236689', '236694', '236698', '236700', '236704', '236705', '236707', '236736', '236740', '236745', '237304', '237539', '237544', '237740', '237741', '237764', '237766', '238028', '238029', '238032']
nosy_count = 9.0
nosy_names = ['brett.cannon', 'paul.moore', 'dholth', 'ethan.furman', 'python-dev', 'berker.peksag', 'Jim.Jewett', 'serhiy.storchaka', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23491'
versions = ['Python 3.5']

[pfmoore]

This is the patch for PEP-441 (Zip Application Support).

Steve, could you check the installer changes, please? I haven't managed to get a setup where I can test the installer, and I'm not aware of any WiX coding tools, so I just edited the XML files by hand with grep and vim, and checked that Tools/buildmsi.bat doesn't error in anything I wrote... (There's a CRC check error in TCL, but that was there before I made changes)

light.exe : error LGHT0216: An unexpected Win32 exception with error code 0x17 occurred while accessing file 'C:\Work\Projects\cpython\externals\tcltk64\lib\tcl8.6\tzdata\America\Detroit': Data error (cyclic redundancy check) [C:\Work\Projects\cpython\Tools\msi\tcltk\tcltk.wixproj]

Also, I don't *think* I need to do anything for the new files in Doc\Lib, Lib and Lib\test to get picked up, but I'm not sure how to check that.

[zooba]

The installer changes look fine, though there shouldn't be PATH related strings in exe/, so there's probably a separate issue there. Wix doesn't really have any editors that aren't just XML editors.

We probably want to update the ContentType value from text/plain for all the shortcuts, though I'm not sure what to (it affects the implicit associations and extra commands on the files, inc. Open With).

The easiest way to make sure the files are in the installer is to install it :) But if they're just .py files under Lib/ then they'll be there.

[zooba]

I haven't looked closely at the rest, btw. Will try and find time today or tomorrow.

[pfmoore]

Thanks for checking, Steve. I don't get an installer because of the checksum error quoted, although I did get the component msi files.

As far as content type is concerned, I wasn't sure what effect it had so I just copied what was there. I guess application/zip might be better for .pyz files, based on what you said?

I'm not sure what you meant by the comment about exe/. All I did was mention .pyz as well as .py - although I don't see where the PathDescription string us used so I may be missing something important!

[zooba]

Either "application/zip" or "application/x-zip-compressed", I'm not sure exactly what the difference is, but the default .zip association has the latter.

The CRC error may be your machine or it may be because we're currently running on unstable WiX releases (hey, everything else is unstable right now :) ). If you can edit and re-save that file, I'd expect it to work, otherwise the 32-bit installer should be okay.

The exe/ comment is more a note for myself, sorry. Those strings are unused and should be removed (I'll do that now - got a few things to fix).

[pfmoore]

Thanks. Updated patch with the new mime-type.

Looks like there's disk errors on that file with the CRC check. Lovely :-)

[dholth]

It looks like your patch does not register PYZW in Windows

[zooba]

It registers the association, but doesn't add it to PATHEXT.

[dholth]

I don't see the string PYZW anywhere in the patch, I only see PYZ,
perhaps I don't know enough about this bug tracker or registering file
extensions?

On Mon, Feb 23, 2015, at 01:19 PM, Steve Dower wrote:

Steve Dower added the comment:

It registers the association, but doesn't add it to PATHEXT.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue23491\>

---

[zooba]

It's this line:

<Extension Id="$(var.ArchiveFileExtension)w" ContentType="application/x-zip-compressed">

$(var.ArchiveFileExtension) expands to "pyz" or "pxz" depending on the build options, and the literal w makes it pyzw or pxzw.

[dholth]

Spelling

raise PackError("Cannot spacify entry point if the source has __main__.py")

[pfmoore]

Thanks, I'll fix for the next iteration of the patch.

[dholth]

Create and open executable file respecting the Unix user's umask:

os.fdopen(os.open(filename, os.O_CREAT|os.O_RDWR), "rw")

On Tue, Feb 24, 2015, at 02:34 PM, Paul Moore wrote:

Paul Moore added the comment:

Thanks, I'll fix for the next iteration of the patch.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue23491\>

---

[pfmoore]

I don't follow (and I don't really want to do things this low level without a compelling reason...)

[pfmoore]

Following on from that, the code to make an archive executable is currently

os.chmod(new_archive, os.stat(new_archive).st_mode | stat.S_IEXEC)

Should I use "... | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH"? If so, do I need to protect that with an "if not Windows" test? (I've tested the existing code and it does nothing on Windows, so I omitted the test at the moment). Is there any *other* way I should be making a file executable on Unix?

(Side note: Maybe there should be an os.make_executable(pathname) or similar that does the right thing in a cross-platform way?)

[jimjjewett]

I think that instead of application/zip, you want application/zipped-python (or whatever the precise term was). This was one reason to register the MIME type. That said, application/zip is probably not harmful; the worst it should do it hand the archive to a zip program.

What is pxzw for, other than amusing mispronunciations?

[zooba]

The .px* extensions are for testing builds without messing up your actual file associations. I ported them forward from the old installer, but most people will never see them.

[pfmoore]

If I understand Steve's comments, the mime type is used by Windows, so application/zip lets Windows know that this filetype is fundamentally a zip file (and so it'll offer to open it with your zip program if you right click, stuff like that).

pxzw - the "w" on the end marks it as a GUI script, to be run by pythonw.exe. The x is a "test mode" thing for development builds of Python. You wouldn't see it in a production build (the production filetypes are pyz and pyzw).

[zooba]

There's also a PerceivedType property which we could set to make sure that archivers correctly light up. I think that'll get you the built-in Extract command. Most tools appear on every file and will try and read it to see what commands make sense, which will work fine for a zip file.

[dholth]

On Thu, Feb 26, 2015, at 09:41 AM, Paul Moore wrote:

Paul Moore added the comment:

Following on from that, the code to make an archive executable is
currently

os.chmod(new_archive, os.stat(new_archive).st_mode | stat.S_IEXEC)

Should I use "... | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH"? If so,
do I need to protect that with an "if not Windows" test? (I've tested the
existing code and it does nothing on Windows, so I omitted the test at
the moment). Is there any *other* way I should be making a file
executable on Unix?

(Side note: Maybe there should be an os.make_executable(pathname) or
similar that does the right thing in a cross-platform way?)

The chmod + umask analog that will work not just on a newly created file
is

umask = os.umask(0) # must change umask to get umask

os.umask(umask) # restore previous umask

os.chmod(new_archive, os.stat(new_archive).st_mode |
((stat.stat.S_IXUSR|stat.S_IXGRP|stat.S_IXOTH) & ~umask))

If I understand the man page correctly, "chmod +x filename" does exactly
the above. Depending on the umask the command may or may not create a
world / group / user executable file.

[pfmoore]

OK, thanks. I don't propose to go there with the initial implementation. If it's a problem in practice, someone can raise a bug and we'll fix it then. (I've never seen actual Python code in the wild that does all of that...)

[pfmoore]

OK, here is an updated patch based on the python-dev discussions

[pfmoore]

Sorry - typo. Try again...

[serhiy-storchaka]

Did you noticed my comments to previous patch Paul?

[pfmoore]

Serhiy - I just got a notification from the review tool - I'd not used it before and didn't know to check it. But I've read them now and corrected a number of places based on your comments and added a few replies to the review. Thanks for the review.

[serhiy-storchaka]

Emails from Rietveld often can be found in the spam folder.

[serhiy-storchaka]

Paul, don't click on Reply below the comment on the main page, it never works. Instead click the link above the comment, then click on the comment to unfold it, click on the Reply below the comment, enter your reply, press Send, and after replying all comments click "Publish+Mail Comments" and then press "Pubblish All My Drafts".

Your replies by email are not archived and are received only by these who already is added comments on Rietveld or added to the nosy list on Roundup.

[serhiy-storchaka]

As for zipimport, it doesn't support namespace packages when no 'directory' entry exists (bpo-14905). The zipfile module CLI now adds entries for directories (bpo-22219).

[pfmoore]

Serihy, thanks for the explanation. I was aware that my replies weren't getting archived, good to know how to avoid that in future.

I've worked out what you mean over the directory entries now. It was easy enough to fix - I just removed the is_file check before writing entries to the zipfile. Test added so it's checked in future as well.

[pfmoore]

Updated patch including (I hope!) all review comments so far.

[serhiy-storchaka]

There is not well known old way how to make executable Python files on Windows. Add line "@python -x %0 %*" at the start of your Python script and rename the script to *.bat. The "-x" option makes Python to skip first line that is not Python, and @ forbids echoing executed command in bat-file.

Could zipapp support this feature?

[pfmoore]

I'm -1 on this. The whole point of having a .pyz extension is so that you don't need to use an extension that's for files containing text to hold binary data.

If you want to do this, use zipapp and specify no interpreter line, then add the @python line yourself. Or just use a zip utility. Or build your own command line app. With the ability to supply an open file to create_archive, it's a 3-liner:

with open(dest_filename, 'wb') as f:
    f.write(b'@python -x %0 %*\n')
    zipapp.create_archive(source_dir, f)

[pfmoore]

There have been no further comments for a while now. Could this be committed by someone?

[berkerpeksag]

I left a couple of minor comments on Rietveld.

[pfmoore]

New patch incorporating Berker's comments

[pfmoore]

There's a problem with the latest patch, it's lost the MSI updates somehow. I'll add them back in this evening, please don't commit before a v6 patch is available,

[pfmoore]

Steve Dower - could you please review the updated patch here. The WiX changes specifically are intended to register the .pyz extension with the launcher, but I can see no code that does this for a per-user install. I've copied the same code for the .py extension so I don't think I've missed anything, but I can't see how this would work in practice.

[zooba]

The whole installer is run with a different global flag depending on the type of install. So it's automatic, though sometimes you can't do both per-user and all-user changes at the same time (not that you should anyway).

[pfmoore]

OK, so if the installer changes look good to you, then this PR is once again ready to go. Sorry to anyone watching for the glitch.

[python-dev]

New changeset d1e9f337fea1 by Brett Cannon in branch 'default':
Issue bpo-23491: Implement PEP-441: Improving Python Zip Application Support
https://hg.python.org/cpython/rev/d1e9f337fea1

[brettcannon]

Thanks for the patch, Paul! I touched up the formatting to be more PEP-8 compliant and fixed the tests as they initially failed for me (TESTFN didn't exist prior to you trying to make a directory, so it just failed; switched to tempfile.TemporaryDirectory instead).

[pfmoore]

Thanks Brett! Apologies about the TESTFN issue - I changed it (from TemporaryDirectory, in fact) based on one of the review comments - it looks like I didn't test the change properly. My mistake, thanks for catching it.