Question about app size

[samschott]

Following the excellent tutorial at https://docs.beeware.org/en/latest/tutorial/, the default app results in a bundle size of ~ 190 MB on macOS. A lot of the bundled content may not be needed in many cases. This includes:

• Python tests (60MB)
• IDLE (5 MB)
• many other possibly unneeded standard lib packages...

For comparison, packing a full app with a Toga GUI and several large dependencies with PyInstaller results in an app bundle of about 18 MB. This is with their "folder" mode and not the "one-file" mode which compresses all dependencies into a self-extracting executable.

Is there an option to exclude certain dependencies explicitly? Or a way to automatically exclude unused modules?

[freakboy3742]

You're absolutely correct - there's a lot of unnecessary material in the macOS support package at present.

I'm actually looking at this right now; it turns out a good chunk of the excess size is a bug. The tooling to build support packages had an exclusion file that should have stripped out all the test files, but the pattern matching was incorrect and so it wasn't picking up everything that should have been stripped.

I've got the support package (zipped) down to about 7.5MB; unpacked, it results in an app that is about 30MB. I'm trying to get some CI infrastructure in place at the same time; but I hope to have updated builds available in a day or two.

Compressing the Python standard library into a zip file can reduce that to 18MB, which matches the size you're seeing from PyInstaller. It may be worth adding this post-processing as a packaging step in Briefcase.

However - even that has a lot of room for improvement. The support package also includes almost the full CPython standard library, and most apps won't need most of it. It also includes (statically linked) BZip, XZ, and OpenSSL; most applications won't need the first two at least. You can define a custom support package as part of your pyproject.toml; but it's up to you to build and manage that support package.

Another option that I think is worth exploring is adding a post-processing step to filter out parts of the standard library that aren't needed. It should be relatively straightforward to add an extra setting to pyproject.toml that matches the standard library paths that are (or aren't) needed, and purge them as part of the build process. That way the default build will be 18-30 MB (depending on whether we also compress the standard library), but with tuning you should be able to reduce that substantially.

The really long term option that I'd like to explore is something that I've discussed with the Python core team - the idea of a "Minimal Viable Python" that is just the interpreter, and allowing all the standard library to be specified as dependencies. That would guarantee the minimum possible build size - but it will also require the involvement of the Python core team to make this happen.

[samschott]

Thanks for detailed reply. I‘ll be happy to try an updated support package when it’s available.

I also really like the idea of a “minimal Python”. Packaging Python apps for standalone distribution seems to be one of the weak points of the ecosystem and this would certainly help.

As of now, it is of course possible to manually go through the bundle and delete unneeded modules. It is of course nicer to do this in an automated way, however, an automated detection of dependencies is always prone to miss some dependencies. As far I understand, PyInstaller tries to fix this with custom hooks for popular packages but maintaining those can be a pain (still there are many edge cases that don’t just work).

In any case, I like the way briefcase keeps the package and module structure when freezing and the resulting app bundles seem a lot cleaner than those produced by PyInstaller.

[freakboy3742]

Another option that might be worth exploring is tree shaking: building a mechanism to walk the user's code looking for import statements, and using that to build the set of modules that is preserved.

[samschott]

modulegraph or its successor modulegraph2 may be useful for this. I haven't used either directly but its used by py2app (by the same developer) and PyInstaller.

[freakboy3742]

Following up on this; new support packages have been published that are significantly smaller.

I'm leaving this ticket open as a placeholder for further size optimization approaches, including:

• Automated zipping of the standard library prior to publication
• Manual specification of a "purge list" of standard library components that aren't required.
• Automated tree shaking

[samschott]

The app size is indeed much smaller, down to 25 MB now! However, I have run into trouble when starting the app. Its complaining about a missing library:

dyld: Library not loaded: /usr/local/opt/gettext/lib/libintl.8.dylib

Running briefcase dev works without problems. I assume that the lib comes with homebrew or macports?

[freakboy3742]

Darn. Definitely looks like a Homebrew leak. Looks like we might need to add some tweaks to the build process to increase isolation from /usr/local. I'm guessing I'm not seeing that failure because I happen to have the magic library installed.

[kaelhem]

Hello,
I encountered the same issue in following the tutorial on step where you need to do a:
briefcase run.
In fact the primary error (which is quite disappointing) was:
LSOpenURLsWithRole() failed with error -10810...

After launching the app directly following the comment of @freakboy3742 in this issue : beeware/beeware#53 (comment)

the error comes a bit clearer:
dyld: Library not loaded: /usr/local/opt/gettext/lib/libintl.8.dylib

I fixed it just by running brew install gettext. I hope this can help!

(I'm runnning on OSX Catalina with python 3.7.7 BTW)

---

Edit:
hum, the briefcase run command still output the LSOpenURLsWithRole error, but I'm able to launch the app directly, and there is no issues when I package it.

[freakboy3742]

@kaelhem Thanks for the report; definitely sounds like we need to do some work on the support package build process.

[samschott]

Something else which I have noticed about the bundled files: both .py and .pyc files are included in the bundle while technically only .pyc files would be needed to run the app. Omitting .py files could potentially halve the size of the app_packages and Python lib directories. Or are those files needed in some way?

[freakboy3742]

Closing this as the original question was answered, but there's nothing immediately actionable coming from the discussion.

[tricks-ter]

The app size is indeed much smaller, down to 25 MB now! However, I have run into trouble when starting the app. Its complaining about a missing library:

dyld: Library not loaded: /usr/local/opt/gettext/lib/libintl.8.dylib

Running briefcase dev works without problems. I assume that the lib comes with homebrew or macports?

try updating pyproject.toml requiments field and use this commandbriefcase update -r