PyInstaller compatibility

[Korijn]

To support freezing apps with wgpu-py, we need to take a couple of things into account:

• PyInstaller can't see dynamic imports, so static imports of specific backends are required; otherwise nested dependencies are not included in the frozen app. See Simplify initialization #18 (comment) for the original discussion.
• Our package data (wgpu/resources folder) won't be included automatically either, unless we submit a hook to the pyinstaller repo or users write their own, containing a call to collect_data_files('wgpu.resources'). See https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/hook-certifi.py for an example

Progress:

• Implement the new hook system in wgpu-py
• Test it in CI using the PR branch of PyInstaller
• Make a release of wgpu-py so PyInstaller can use it in their tests.
• PyInstaller branch merged (Moving hooks to the packages they support pyinstaller/pyinstaller#4582)
• PyInstaller 4 release
• Update wgpu-py dev requirments to use pyinstaller>=4 instead of the branch

[almarklein]

Submitting a hook that simply ads the whole resourced dir should be easy enough. Let's get to that once we have a release that includes binaries.

[Korijn]

Created the hook here: https://github.com/Korijn/pyinstaller/commit/c9f333ec7c15890ae019559e22697db027635502

Verified that it works:

korijn@swatcat:/media/ssd2/dev/wgpu-py (git:master) ✔ 
$ pipenv run pyinstaller --clean -y examples/triangle.py
69 INFO: PyInstaller: 4.0.dev0+gc4f43a07.mod
69 INFO: Python: 3.6.8
70 INFO: Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid
70 INFO: wrote /media/ssd2/dev/wgpu-py/triangle.spec
71 INFO: UPX is not available.
71 INFO: Removing temporary files and cleaning cache in /home/korijn/.cache/pyinstaller
73 INFO: Extending PYTHONPATH with paths
['/media/ssd2/dev/wgpu-py/examples', '/media/ssd2/dev/wgpu-py']
73 INFO: checking Analysis
73 INFO: Building Analysis because Analysis-00.toc is non existent
73 INFO: Initializing module dependency graph...
74 INFO: Caching module graph hooks...
77 INFO: Analyzing base_library.zip ...
2130 INFO: Caching module dependency graph...
2186 INFO: running Analysis Analysis-00.toc
2201 INFO: Analyzing /media/ssd2/dev/wgpu-py/examples/triangle.py
2334 INFO: Processing pre-find module path hook   distutils
2335 INFO: distutils: retargeting to non-venv dir '/home/korijn/.pyenv/versions/3.6.8/lib/python3.6'
2357 INFO: Processing module hooks...
2357 INFO: Loading module hook "hook-wgpu.py"...
2358 INFO: Loading module hook "hook-sysconfig.py"...
2364 INFO: Loading module hook "hook-distutils.py"...
2365 INFO: Loading module hook "hook-pkg_resources.py"...
2539 INFO: Processing pre-safe import module hook   win32com
2717 INFO: Excluding import '__main__'
2717 INFO:   Removing import of __main__ from module pkg_resources
2718 INFO: Loading module hook "hook-encodings.py"...
2757 INFO: Loading module hook "hook-xml.py"...
2929 INFO: Looking for ctypes DLLs
2960 INFO: Analyzing run-time hooks ...
2963 INFO: Including run-time hook 'pyi_rth_pkgres.py'
2967 INFO: Looking for dynamic libraries
3197 INFO: Looking for eggs
3198 INFO: Python library not in binary dependencies. Doing additional searching...
3211 INFO: Using Python library /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
3214 INFO: Warnings written to /media/ssd2/dev/wgpu-py/build/triangle/warn-triangle.txt
3233 INFO: Graph cross-reference written to /media/ssd2/dev/wgpu-py/build/triangle/xref-triangle.html
3238 INFO: checking PYZ
3238 INFO: Building PYZ because PYZ-00.toc is non existent
3238 INFO: Building PYZ (ZlibArchive) /media/ssd2/dev/wgpu-py/build/triangle/PYZ-00.pyz
3534 INFO: Building PYZ (ZlibArchive) /media/ssd2/dev/wgpu-py/build/triangle/PYZ-00.pyz completed successfully.
3537 INFO: checking PKG
3537 INFO: Building PKG because PKG-00.toc is non existent
3537 INFO: Building PKG (CArchive) PKG-00.pkg
3556 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
3557 INFO: Bootloader /media/ssd2/dev/pyinstaller/PyInstaller/bootloader/Linux-64bit/run
3557 INFO: checking EXE
3557 INFO: Building EXE because EXE-00.toc is non existent
3557 INFO: Building EXE from EXE-00.toc
3557 INFO: Appending archive to ELF section in EXE /media/ssd2/dev/wgpu-py/build/triangle/triangle
3561 INFO: Building EXE from EXE-00.toc completed successfully.
3562 INFO: checking COLLECT
3562 INFO: Building COLLECT because COLLECT-00.toc is non existent
On your own risk, you can use the option `--noconfirm` to get rid of this question.
3562 INFO: Removing dir /media/ssd2/dev/wgpu-py/dist/triangle
3566 INFO: Building COLLECT COLLECT-00.toc
3585 INFO: Building COLLECT COLLECT-00.toc completed successfully.

korijn@swatcat:/media/ssd2/dev/wgpu-py (git:master) ✔ 
$ ls dist/triangle/wgpu/resources/
codegen_report.md  libwgpu_native.so  webgpu.idl  wgpu.h  wgpu_native-version

[almarklein]

Updated first comment to keep track of progress.

[Legorooj]

@Korijn @almarklein the branch has been merged in. The new hook-api is "live" in development. 4.0 should be released before lockdown is over (So should 4.1 for that matter).

[Korijn]

@Korijn @almarklein the branch has been merged in. The new hook-api is "live" in development. 4.0 should be released before lockdown is over (So should 4.1 for that matter).

Thanks for the heads up! I've updated the first comment accordingly.

[Korijn]

I just realized this has probably gone to production a while ago. We can test and close this :)