Add an option to zipapp to produce a Windows executable

[pfmoore]

BPO	28247
Nosy	@brettcannon, @pfmoore, @tjguk, @zware, @serhiy-storchaka, @eryksun, @zooba, @csabella, @miss-islington
PRs	bpo-28247: Document Windows executable creation in zipapp #6158 [3.7] bpo-28247: Document Windows executable creation in zipapp (GH-6158) #6163 [3.6] bpo-28247: Document Windows executable creation in zipapp (GH-6158) #6164
Files	zipapp-doc.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pfmoore'
closed_at = <Date 2018-03-21.12:43:27.965>
created_at = <Date 2016-09-22.13:55:29.386>
labels = ['3.8', 'type-feature', '3.7', 'docs']
title = 'Add an option to zipapp to produce a Windows executable'
updated_at = <Date 2018-11-24.09:36:10.128>
user = 'https://github.com/pfmoore'

bugs.python.org fields:

activity = <Date 2018-11-24.09:36:10.128>
actor = 'paul.moore'
assignee = 'paul.moore'
closed = True
closed_date = <Date 2018-03-21.12:43:27.965>
closer = 'cheryl.sabella'
components = ['Documentation']
creation = <Date 2016-09-22.13:55:29.386>
creator = 'paul.moore'
dependencies = []
files = ['44863']
hgrepos = []
issue_num = 28247
keywords = ['patch']
message_count = 16.0
messages = ['277224', '277228', '277229', '277231', '277524', '277525', '277606', '277638', '314135', '314137', '314157', '314171', '314172', '314173', '330366', '330374']
nosy_count = 10.0
nosy_names = ['brett.cannon', 'paul.moore', 'tim.golden', 'donmez', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'cheryl.sabella', 'miss-islington']
pr_nums = ['6158', '6163', '6164']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28247'
versions = ['Python 3.7', 'Python 3.8']

[pfmoore]

The zipapp module allows users to bundle their application as a single file "executable". On Windows, the file is given a ".pyz" extension which is associated with the Python launcher. However, this approach is not always equivalent to a native executable (see http://paul-moores-notes.readthedocs.io/en/latest/wrappers.html for more details).

I suggest adding an option to zipapp that prepends a small executable to the zipapp that uses the Python C API to launch the application. A prototype implementation (zastub) is available at https://github.com/pfmoore/pylaunch.

If this seems reasonable, I'll work up a full patch.

[serhiy-storchaka]

Why not just change the extension to cmd and add the following line at the start?

@python -x "%0" %*

[pfmoore]

(1) It starts an extra process (unless you're running the application from cmd.exe) and (2) in some cases, the system won't recognise a cmd file as an executable. For a simple example,

t.cmd:

@echo Hello from t

example.py:

from subprocess import run
run(["t")]

If you run example.py you get "FileNotFoundError: [WinError 2] The system cannot find the file specified".

[eryksun]

Specifically, while CreateProcess does execute batch scripts via the %ComSpec% interpreter, the only extension it infers is ".exe". To run a ".cmd" or ".bat" file, you have to use the full name with the extension.

[pfmoore]

I'm still unsure whether this would be a good idea. On the plus side, it provides (in conjunction with the "embedded" distribution) a really good way of producing a standalone application on Windows. On the minus side there are some limitations which may trip up naive users:

• You can't put C extensions in a zipapp
• You have to take care with things like multiprocessing if you're using an embedded app.

And the wrapper's basically only 4 lines of code:

wchar_t **myargv = _alloca((__argc + 2) * sizeof(wchar_t*));
myargv[0] = __wargv[0];
memcpy(myargv + 1, __wargv, (__argc + 1) * sizeof(wchar_t *));
return Py_Main(__argc+1, myargv);

so it's not exactly rocket science. (By the way, the arguments to Py_Main are "exactly as those which are passed to a C program’s main" - IIRC, for a C program there's always a NULL pointer at the end of argv, does that rule apply to Py_Main, too? The code doesn't seem to rely on it, so I guess I could save a slot in the array above).

Maybe adding a section to the zipapp docs explaining how to make standalone applications would be a better way of handling this? That would have the advantage of being just as applicable to 3.6 (and 3.5, for that matter) at the cost of making users build their own wrapper (or find a published one).

[brettcannon]

I think documentation is definitely the cheapest option. It's also the most flexible since if we find out there's a lot of usage of the guide then we can add explicit support.

[pfmoore]

OK, here's a first draft documentation patch. As it's purely a documentation change, I guess it should go into the 3.5, 3.6 and trunk branches? For now it's against trunk (it's not like that file has changed recently anyway), and I'll sort out the merge dance once it looks OK.

[brettcannon]

Only backport if you want; it's not a bug fix so technically it doesn't need to be backported.

[csabella]

Hi Paul,

Were you interested in moving forward with this doc change?

[pfmoore]

Hi Cheryl,
Looks like I dropped the ball on this one :-( I think it just needs to be applied to the main branch (3.7/3.8) - I don't think it's worth backporting.

I'll try to get to it when I can, but it may not be for a few weeks, as I have other stuff on my plate at the moment. I've not done anything under the new github workflow, so I don't want to rush it and risk making mistakes. I'm happy if someone else wants to merge it in the meantime.

[csabella]

Hi Paul,

Thanks for your response. I've made the pull request from your patch, so it would great if you could review it. Thanks!

[pfmoore]

New changeset 4be79f2 by Paul Moore (Cheryl Sabella) in branch 'master':
bpo-28247: Document Windows executable creation in zipapp (GH-6158)
4be79f2

[miss-islington]

New changeset a70b8f5 by Miss Islington (bot) in branch '3.7':
bpo-28247: Document Windows executable creation in zipapp (GH-6158)
a70b8f5

[miss-islington]

New changeset 47a0e64 by Miss Islington (bot) in branch '3.6':
bpo-28247: Document Windows executable creation in zipapp (GH-6158)
47a0e64

[donmez]

The documentation helped a lot, so thanks for that! But it misses the final crucial step:

copy /b zastub.exe+app.pyz app.exe

The documentation talks about prepending the zastub.exe to the zip file but never mentions how, which is very confusing.

[pfmoore]

While I can see your point, I'm a little skeptical that anyone who's able to write C source code and compile it can't work out how to combine two binary files... :-)

[jcrben]

@pfmoore many people are pretty good at following instructions as long as they are complete - lots of just-in-time learners out there 😄

I'm wondering about this little snippet:

For a fully standalone distribution, you can distribute the launcher with your
application appended, bundled with the Python "embedded" distribution. This
will run on any PC with the appropriate architecture (32 bit or 64 bit).

I'm wondering how tricky it would be to embed that interpreter and if Python supports any official utilities to help with that?

Perusing projects like https://github.com/pantsbuild/pex and linkedin/shiv#32 make it seem like it can be tricky

See also: https://www.scylladb.com/2019/02/14/the-complex-path-for-a-simple-portable-python-interpreter-or-snakes-on-a-data-plane/

cc: @csabella

[pfmoore]

Actually, this reminded me that the instructions on how to build the executable are now out of date as distutils is no longer in the stdlib. And using the embedded interpreter is somewhat tricky. I’ve written various utilities over time to do this, but none really felt “production quality”. It’s much the same issue as the fact that code will mostly work if bundled in a zipapp - unless it won’t. There are a lot of details to get right and the stdlib documentation isn’t really the place to discuss them. On reflection, I think that the whole “Making a Windows Executable” section is probably better removed.

I think that building fully standalone executables is a topic better suited for a 3rd party application. Maybe it is something shiv or pep would want to support?

[pfmoore]

many people are pretty good at following instructions as long as they are complete

I should also have addressed this point. You are correct, but the problem here is less with the missing “how to append” step, and more with the fact that the rest of the instructions aren’t as complete as they look - they are suggestions that need to be applied with some care if they are to work. As such, they are probably more suitable as the subject of a blog post or article rather than being in the stdlib docs.

[zooba]

I agree that fully standalone executables are best documented elsewhere, because they will almost certainly rely on templates and tooling that we do not provide.

We could improve the documentation for working with the embeddable distro, and I've probably got 5-10 email threads worth of questions to help fill in the gaps. But even there there's enough variation in what people are doing to make it hard to cover all bases and also provide a workflow - a reference manual is hard to follow, but a tutorial largely won't be applicable. It's a really tough spot.

(That said, if we had a standard mechanism for doing this that didn't work well with all existing libraries, I'm sure those libraries would adapt to work with it. We have a bit of scope to lead here, not merely follow all the existing conventions/assumptions.)

[pfmoore]

Agreed. I think this is getting close to the question of whether (and how) the stdlib and core supports packaging. In this case, in the sense of "packaging up my Python code into a standalone tool/app/executable" rather than "packaging a Python library so that it can be imported".

I think it's entirely reasonable to assume that the stdlib and core will provide mechanisms for implementing tools to build standalone executables, but not provide those high-level tools directly. So we have zipapp, but not shiv, for example.

One part of this toolkit is zipapp, which covers bundling your Python code. A second part is the embeddable distribution, which covers shipping a dedicated interpreter with your app. A third part might be to enhance the py.exe launcher to allow it to be combined with a zipapp and the embeddable distribution to act as the driver executable. Of those three, the launcher is probably the furthest from fulfilling that role at the moment, which is why I added the documentation section we're discussing now. But I'd much rather drop that in favour of linking to the docs on "how to configure the launcher to use a specific interpreter and command line".

I'm imagining something along the lines of:

1. Create a directory, and put a copy of the launcher in there, renamed to foo.exe.
2. Unpack the embedded distribution into a runtime subdirectory.
3. Put your Python code into a zipapp, foo.pyz. Or just put it into a subdirectory, foo, if it has native binary dependencies.
4. Create a config file that says "Use the distribution in runtime and run python foo.pyz".

Ship the resulting directory.

The stdlib could even include a library that handled locating and copying the various "standard" items here (launcher, embeddable distribution) so that the user didn't have to manually locate and unpack them.

Unfortunately, my C skills aren't really good enough to make a change that substantial to the launcher. I have written something similar (probably not production quality) in Rust, but I don't imagine we're likely to be able to ship code written in Rust with CPython (yet!).

[zooba]

A third part might be to enhance the py.exe launcher to allow it to be combined with a zipapp and the embeddable distribution to act as the driver executable. Of those three, the launcher is probably the furthest from fulfilling that role at the moment

Most of the launcher is unnecessary if you've also got the embeddable distro somehow bundled. What we want here is a self-extracting app that can (safely!) generate the temp directories and then launch from in there (much like how many setup executables, including Python's, work).

But the launcher should be pretty close to concatenating a .pyz and being able to run from that. I believe the old implementation has that in there (under one of the preprocessor selectors). Really it's just looking at its own file for the ZIP header, inserting itself into the command line and doing a normal search for already-installed Python. But I don't think this is really what people want.

I think where it would make the most impact is integrated into a GUI framework (Electron, possibly?). Command-line users seem to be quite happy using other command-line tools to install things, but where the single-file executable really comes into its own is for users to download, double-click, and have just a normal app. I don't know of anyone seriously investing in this area though.

[jcrben]

Thanks for the comments. The PyOxidizer project by @indygreg looks like it's shaping up to be the most complete and polished third-party solution in this area. For my purposes, one thing that's nice about the builtin option is the trappings of being official which makes it an easier sell to the security and management folks. I just want to distribute Python apps with builtin interpreters in an enterprise environment and the fewer dependencies the better. Currently making do with shell scripts and some setup processes. I wouldn't mind cobbling together some additional code on my end to put the pieces together but sounds like it's more of a project.