Cover "Bundled Applications" as an additional topic

[ncoghlan]

We currently don't cover the question of "all inclusive" applications at all, where the whole of the application is bundled up into a single file. While that's beyond the scope of PyPA's responsibilities from a development perspective, we can still provide a useful index pointing folks towards the tools and communities that do address this problem space.

There are two main approaches to cover: those that assume a Python runtime will be available on the target system, and those that bundle the Python runtime as well.

Options that expect a runtime to be available:

• zipapp (standard library): https://docs.python.org/3/library/zipapp.html
• pex: https://pex.readthedocs.io/en/stable/

Options that bundle the runtime as well:

• PyInstaller (Windows, Mac OS X, Linux, *nix): http://www.pyinstaller.org/
• cx_Freeze (Windows, Mac OS X, *nix: https://cx-freeze.readthedocs.io/en/latest/
• py2exe (Windows): http://www.py2exe.org/
• py2app (Mac OS X): https://pythonhosted.org/py2app/
• briefcase (iOS, tvOS, macOS, Android): https://briefcase.readthedocs.io/en/latest/

[ncoghlan]

From http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html#comment-2902453788 there's also a relatively new option that uses conda as the internal mechanism for managing the bundled language runtime and all the components installed on top of that: https://www.continuum.io/blog/developer-blog/introducing-constructor

[ncoghlan]

Heh, I thought we had some info about this - it's just under "Application Deployment" at the moment. So it probably makes sense to split that existing topic into two topics, with "Application Deployment" focusing on server use cases (and mentioning things like Docker, Kubernetes, AWS Lambda, etc), and "Bundled Applications" focusing on client devices.

Existing page: https://github.com/pypa/python-packaging-user-guide/blob/master/source/deployment.rst
Live version: https://packaging.python.org/deployment/

I also just merged #248, which added a note about the Kivy app creation toolkit, Buildozer: https://buildozer.readthedocs.io/en/latest/

[ncoghlan]

(And to be absolutely clear: I'm not planning to do this topic split myself, but if someone does decide to pursue it, I'll definitely make time to do the PR review)

[cjrh]

@ncoghlan So I'm kind of in love with Tools/freeze/freeze.py in the CPython source, particularly how it bundles the Python stdlib code into the frozen executable as C files containing arrays of bytes that represent the bytecode of the original Python file. This tool also gets omitted from every list of bundling tools, and I'm wondering whether there's a reason for that?

It needs a patch to set NoSiteDir:

# HG changeset patch
# User Caleb Hattingh <caleb.hattingh@gmail.com>
# Date 1473556682 -36000
#      Sun Sep 11 11:18:02 2016 +1000
# Branch freeze-nosite
# Node ID 5f259d7d30b7f031cbb044d1211994f7c97cb0a2
# Parent  c5f9e8f2d9df7911362c129f43e738c2356bfdc2
Add Py_NoSiteFlag to makefreeze.py

diff --git a/Tools/freeze/makefreeze.py b/Tools/freeze/makefreeze.py
--- a/Tools/freeze/makefreeze.py
+++ b/Tools/freeze/makefreeze.py
@@ -23,6 +23,7 @@
 """ + ((not __debug__ and """
         Py_OptimizeFlag++;
 """) or "")  + """
+        Py_NoSiteFlag = 1;
         PyImport_FrozenModules = _PyImport_FrozenModules;
         return Py_FrozenMain(argc, argv);
 }

But besides that, it works really well for making standalone bundles (like go built binaries). I've gotten stuck at modifying Modules/Setup.dist and friends to static link the other deps like tk, bzip and openssl, but in principle it should be possible to build a single binary executable for deployment. Specifically, it should be entirely possible to make docker images hosting such executables under 20 MB in size, just like with golang.

Is there a reason you might know why the freeze tool is so little-known?

[ncoghlan]

@cjrh Aside from the general lack of documentation, freezing modules tends to run into the same kinds of problems as zipimport: you break the assumptions of quite a few other pieces of code.

A few key problems:

• Frozen submodules aren't supported at all: http://bugs.python.org/issue1644818
• Tracebacks can't provide the original text: http://bugs.python.org/issue25268 (although a post-processor could potentially reconstruct it)
• pkgutil doesn't handle them properly: http://bugs.python.org/issue16027

Those aren't insurmountable problems (small CLI applications in particular should find them relatively easy to workaround), but they're a barrier to elevation into the standard library proper. I do agree that with the increased interest in self-contained executables and static linking prompted by golang, the time is likely right for folks to start exploring that further, but it's also worth doing outside the constraints of the standard library first.

[cjrh]

Thanks for the links! I did not know about the pkgutil problem (haven't tried "-m"ing anything yet).

[ncoghlan]

While looking for something else, I stumbled across another entrant in this space: Cloudify's Wagon

The article by which I found it: https://www.infoq.com/articles/python-wagon-offline-packages

[ncoghlan]

Based on a discussion I had with @freakboy3742 at PyCon AU, I'm also thinking we should probably note somewhere that we're pretty much waiting for the dust to settle on Linux desktop application bundling at the Gnome & KDE level before weighing in explicitly on the Snappy vs Flatpak vs AppImage question (while I personally have a preference for Flatpak, I'm also aware that I'm far from being unbiased on the topic).

In the meantime, as @warsaw notes in https://www.wefearchange.org/2015/04/creating-python-snaps.html, targeting pex is a decent stepping stone to targeting the Linux-specific app bundling formats, as that way the pex level bundling handles the Python-specific parts, while the Snappy(/Flatpak/Appimage) glue can focus on handling pex files rather than arbitrary Python projects.

That said, @takluyver has also been conducting some experiments with targeting Flatpak directly fro PyGame applications: https://github.com/takluyver/pygame-flatpak-test

[bittner]

Just as a side comment (and question):

When bundling applications, in properly engineered projects there is a CI service setup that handles that. The fact that there is a dedicated chapter on AppVeyor also reflects the weight the Windows user base has.

But is there also a CI service that allows you to run builds in a macOS environment?

We should maybe provide an example setup that builds bundled applications cross-platform, for all major platforms (e.g. various GNU/Linux distros, Windows, macOS).

[di]

But is there also a CI service that allows you to run builds in a macOS environment?

Travis provides an "OS X Build Environment": https://docs.travis-ci.com/user/reference/osx/