Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 441 - Improving Python Zip Application Support #67679

Closed
pfmoore opened this issue Feb 20, 2015 · 42 comments
Closed

PEP 441 - Improving Python Zip Application Support #67679

pfmoore opened this issue Feb 20, 2015 · 42 comments
Assignees
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@pfmoore
Copy link
Member

pfmoore commented Feb 20, 2015

BPO 23491
Nosy @brettcannon, @pfmoore, @dholth, @ethanfurman, @berkerpeksag, @JimJJewett, @serhiy-storchaka, @zooba
Files
  • pep441.patch
  • pep441.patch
  • pep-441.patch
  • pep-441.v4.patch
  • pep-441.v5.patch
  • pep-441.v6.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2015-03-13.14:45:26.083>
    created_at = <Date 2015-02-20.12:52:51.249>
    labels = ['type-feature', 'library']
    title = 'PEP 441 - Improving Python Zip Application Support'
    updated_at = <Date 2015-03-13.15:06:38.893>
    user = 'https://github.com/pfmoore'

    bugs.python.org fields:

    activity = <Date 2015-03-13.15:06:38.893>
    actor = 'paul.moore'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2015-03-13.14:45:26.083>
    closer = 'brett.cannon'
    components = ['Library (Lib)']
    creation = <Date 2015-02-20.12:52:51.249>
    creator = 'paul.moore'
    dependencies = []
    files = ['38187', '38192', '38250', '38258', '38389', '38419']
    hgrepos = []
    issue_num = 23491
    keywords = ['patch']
    message_count = 42.0
    messages = ['236276', '236285', '236286', '236291', '236301', '236308', '236455', '236457', '236458', '236459', '236531', '236534', '236669', '236670', '236672', '236673', '236675', '236676', '236678', '236680', '236682', '236688', '236689', '236694', '236698', '236700', '236704', '236705', '236707', '236736', '236740', '236745', '237304', '237539', '237544', '237740', '237741', '237764', '237766', '238028', '238029', '238032']
    nosy_count = 9.0
    nosy_names = ['brett.cannon', 'paul.moore', 'dholth', 'ethan.furman', 'python-dev', 'berker.peksag', 'Jim.Jewett', 'serhiy.storchaka', 'steve.dower']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue23491'
    versions = ['Python 3.5']

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 20, 2015

    This is the patch for PEP-441 (Zip Application Support).

    Steve, could you check the installer changes, please? I haven't managed to get a setup where I can test the installer, and I'm not aware of any WiX coding tools, so I just edited the XML files by hand with grep and vim, and checked that Tools/buildmsi.bat doesn't error in anything I wrote... (There's a CRC check error in TCL, but that was there before I made changes)

    light.exe : error LGHT0216: An unexpected Win32 exception with error code 0x17 occurred while accessing file 'C:\Work\Projects\cpython\externals\tcltk64\lib\tcl8.6\tzdata\America\Detroit': Data error (cyclic redundancy check) [C:\Work\Projects\cpython\Tools\msi\tcltk\tcltk.wixproj]

    Also, I don't *think* I need to do anything for the new files in Doc\Lib, Lib and Lib\test to get picked up, but I'm not sure how to check that.

    @pfmoore pfmoore added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Feb 20, 2015
    @zooba
    Copy link
    Member

    zooba commented Feb 20, 2015

    The installer changes look fine, though there shouldn't be PATH related strings in exe/, so there's probably a separate issue there. Wix doesn't really have any editors that aren't just XML editors.

    We probably want to update the ContentType value from text/plain for all the shortcuts, though I'm not sure what to (it affects the implicit associations and extra commands on the files, inc. Open With).

    The easiest way to make sure the files are in the installer is to install it :) But if they're just .py files under Lib/ then they'll be there.

    @zooba
    Copy link
    Member

    zooba commented Feb 20, 2015

    I haven't looked closely at the rest, btw. Will try and find time today or tomorrow.

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 20, 2015

    Thanks for checking, Steve. I don't get an installer because of the checksum error quoted, although I did get the component msi files.

    As far as content type is concerned, I wasn't sure what effect it had so I just copied what was there. I guess application/zip might be better for .pyz files, based on what you said?

    I'm not sure what you meant by the comment about exe/. All I did was mention .pyz as well as .py - although I don't see where the PathDescription string us used so I may be missing something important!

    @zooba
    Copy link
    Member

    zooba commented Feb 20, 2015

    Either "application/zip" or "application/x-zip-compressed", I'm not sure exactly what the difference is, but the default .zip association has the latter.

    The CRC error may be your machine or it may be because we're currently running on unstable WiX releases (hey, everything else is unstable right now :) ). If you can edit and re-save that file, I'd expect it to work, otherwise the 32-bit installer should be okay.

    The exe/ comment is more a note for myself, sorry. Those strings are unused and should be removed (I'll do that now - got a few things to fix).

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 20, 2015

    Thanks. Updated patch with the new mime-type.

    Looks like there's disk errors on that file with the CRC check. Lovely :-)

    @dholth
    Copy link
    Mannequin

    dholth mannequin commented Feb 23, 2015

    It looks like your patch does not register PYZW in Windows

    @zooba
    Copy link
    Member

    zooba commented Feb 23, 2015

    It registers the association, but doesn't add it to PATHEXT.

    @dholth
    Copy link
    Mannequin

    dholth mannequin commented Feb 23, 2015

    I don't see the string PYZW anywhere in the patch, I only see PYZ,
    perhaps I don't know enough about this bug tracker or registering file
    extensions?

    On Mon, Feb 23, 2015, at 01:19 PM, Steve Dower wrote:

    Steve Dower added the comment:

    It registers the association, but doesn't add it to PATHEXT.

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue23491\>


    @zooba
    Copy link
    Member

    zooba commented Feb 23, 2015

    It's this line:

    <Extension Id="$(var.ArchiveFileExtension)w" ContentType="application/x-zip-compressed">
    

    $(var.ArchiveFileExtension) expands to "pyz" or "pxz" depending on the build options, and the literal w makes it pyzw or pxzw.

    @dholth
    Copy link
    Mannequin

    dholth mannequin commented Feb 24, 2015

    Spelling

    raise PackError("Cannot spacify entry point if the source has __main__.py")

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 24, 2015

    Thanks, I'll fix for the next iteration of the patch.

    @dholth
    Copy link
    Mannequin

    dholth mannequin commented Feb 26, 2015

    Create and open executable file respecting the Unix user's umask:

    os.fdopen(os.open(filename, os.O_CREAT|os.O_RDWR), "rw")

    On Tue, Feb 24, 2015, at 02:34 PM, Paul Moore wrote:

    Paul Moore added the comment:

    Thanks, I'll fix for the next iteration of the patch.

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue23491\>


    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 26, 2015

    I don't follow (and I don't really want to do things this low level without a compelling reason...)

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 26, 2015

    Following on from that, the code to make an archive executable is currently

    os.chmod(new_archive, os.stat(new_archive).st_mode | stat.S_IEXEC)

    Should I use "... | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH"? If so, do I need to protect that with an "if not Windows" test? (I've tested the existing code and it does nothing on Windows, so I omitted the test at the moment). Is there any *other* way I should be making a file executable on Unix?

    (Side note: Maybe there should be an os.make_executable(pathname) or similar that does the right thing in a cross-platform way?)

    @jimjjewett
    Copy link
    Mannequin

    jimjjewett mannequin commented Feb 26, 2015

    I think that instead of application/zip, you want application/zipped-python (or whatever the precise term was). This was one reason to register the MIME type. That said, application/zip is probably not harmful; the worst it should do it hand the archive to a zip program.

    What is pxzw for, other than amusing mispronunciations?

    @zooba
    Copy link
    Member

    zooba commented Feb 26, 2015

    The .px* extensions are for testing builds without messing up your actual file associations. I ported them forward from the old installer, but most people will never see them.

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 26, 2015

    If I understand Steve's comments, the mime type is used by Windows, so application/zip lets Windows know that this filetype is fundamentally a zip file (and so it'll offer to open it with your zip program if you right click, stuff like that).

    pxzw - the "w" on the end marks it as a GUI script, to be run by pythonw.exe. The x is a "test mode" thing for development builds of Python. You wouldn't see it in a production build (the production filetypes are pyz and pyzw).

    @zooba
    Copy link
    Member

    zooba commented Feb 26, 2015

    There's also a PerceivedType property which we could set to make sure that archivers correctly light up. I think that'll get you the built-in Extract command. Most tools appear on every file and will try and read it to see what commands make sense, which will work fine for a zip file.

    @dholth
    Copy link
    Mannequin

    dholth mannequin commented Feb 26, 2015

    On Thu, Feb 26, 2015, at 09:41 AM, Paul Moore wrote:

    Paul Moore added the comment:

    Following on from that, the code to make an archive executable is
    currently

    os.chmod(new_archive, os.stat(new_archive).st_mode | stat.S_IEXEC)

    Should I use "... | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH"? If so,
    do I need to protect that with an "if not Windows" test? (I've tested the
    existing code and it does nothing on Windows, so I omitted the test at
    the moment). Is there any *other* way I should be making a file
    executable on Unix?

    (Side note: Maybe there should be an os.make_executable(pathname) or
    similar that does the right thing in a cross-platform way?)

    The chmod + umask analog that will work not just on a newly created file
    is

    umask = os.umask(0) # must change umask to get umask
    
    os.umask(umask) # restore previous umask

    os.chmod(new_archive, os.stat(new_archive).st_mode |
    ((stat.stat.S_IXUSR|stat.S_IXGRP|stat.S_IXOTH) & ~umask))

    If I understand the man page correctly, "chmod +x filename" does exactly
    the above. Depending on the umask the command may or may not create a
    world / group / user executable file.

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 26, 2015

    OK, thanks. I don't propose to go there with the initial implementation. If it's a problem in practice, someone can raise a bug and we'll fix it then. (I've never seen actual Python code in the wild that does all of that...)

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 26, 2015

    OK, here is an updated patch based on the python-dev discussions

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 26, 2015

    Sorry - typo. Try again...

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Feb 26, 2015

    Did you noticed my comments to previous patch Paul?

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 26, 2015

    Serhiy - I just got a notification from the review tool - I'd not used it before and didn't know to check it. But I've read them now and corrected a number of places based on your comments and added a few replies to the review. Thanks for the review.

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Feb 26, 2015

    Emails from Rietveld often can be found in the spam folder.

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Feb 26, 2015

    Paul, don't click on Reply below the comment on the main page, it never works. Instead click the link above the comment, then click on the comment to unfold it, click on the Reply below the comment, enter your reply, press Send, and after replying all comments click "Publish+Mail Comments" and then press "Pubblish All My Drafts".

    Your replies by email are not archived and are received only by these who already is added comments on Rietveld or added to the nosy list on Roundup.

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Feb 26, 2015

    As for zipimport, it doesn't support namespace packages when no 'directory' entry exists (bpo-14905). The zipfile module CLI now adds entries for directories (bpo-22219).

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 26, 2015

    Serihy, thanks for the explanation. I was aware that my replies weren't getting archived, good to know how to avoid that in future.

    I've worked out what you mean over the directory entries now. It was easy enough to fix - I just removed the is_file check before writing entries to the zipfile. Test added so it's checked in future as well.

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 27, 2015

    Updated patch including (I hope!) all review comments so far.

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Feb 27, 2015

    There is not well known old way how to make executable Python files on Windows. Add line "@python -x %0 %*" at the start of your Python script and rename the script to *.bat. The "-x" option makes Python to skip first line that is not Python, and @ forbids echoing executed command in bat-file.

    Could zipapp support this feature?

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Feb 27, 2015

    I'm -1 on this. The whole point of having a .pyz extension is so that you don't need to use an extension that's for files containing text to hold binary data.

    If you want to do this, use zipapp and specify no interpreter line, then add the @python line yourself. Or just use a zip utility. Or build your own command line app. With the ability to supply an open file to create_archive, it's a 3-liner:

    with open(dest_filename, 'wb') as f:
        f.write(b'@python -x %0 %*\n')
        zipapp.create_archive(source_dir, f)

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Mar 5, 2015

    There have been no further comments for a while now. Could this be committed by someone?

    @berkerpeksag
    Copy link
    Member

    berkerpeksag commented Mar 8, 2015

    I left a couple of minor comments on Rietveld.

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Mar 8, 2015

    New patch incorporating Berker's comments

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Mar 10, 2015

    There's a problem with the latest patch, it's lost the MSI updates somehow. I'll add them back in this evening, please don't commit before a v6 patch is available,

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Mar 10, 2015

    Steve Dower - could you please review the updated patch here. The WiX changes specifically are intended to register the .pyz extension with the launcher, but I can see no code that does this for a per-user install. I've copied the same code for the .py extension so I don't think I've missed anything, but I can't see how this would work in practice.

    @zooba
    Copy link
    Member

    zooba commented Mar 10, 2015

    The whole installer is run with a different global flag depending on the type of install. So it's automatic, though sometimes you can't do both per-user and all-user changes at the same time (not that you should anyway).

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Mar 10, 2015

    OK, so if the installer changes look good to you, then this PR is once again ready to go. Sorry to anyone watching for the glitch.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 13, 2015

    New changeset d1e9f337fea1 by Brett Cannon in branch 'default':
    Issue bpo-23491: Implement PEP-441: Improving Python Zip Application Support
    https://hg.python.org/cpython/rev/d1e9f337fea1

    @brettcannon
    Copy link
    Member

    brettcannon commented Mar 13, 2015

    Thanks for the patch, Paul! I touched up the formatting to be more PEP-8 compliant and fixed the tests as they initially failed for me (TESTFN didn't exist prior to you trying to make a directory, so it just failed; switched to tempfile.TemporaryDirectory instead).

    @pfmoore
    Copy link
    Member Author

    pfmoore commented Mar 13, 2015

    Thanks Brett! Apologies about the TESTFN issue - I changed it (from TemporaryDirectory, in fact) based on one of the review comments - it looks like I didn't test the change properly. My mistake, thanks for catching it.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants