Script for generating Linux AppImage package

[morevnaproject]

This is a script to generate Linux AppImage package. Just execute build/linux-appimage/build.sh script and it will generate package for you.

Here is a sample package generated by this script - Papagayo_NG-d91a1b7-x86_64.AppImage

At the moment the package doesn't includes Allosaurus, as it makes very large package - 1,2 Gb (comparing to ~300 Mb without Allosaurus). You can enable Allosaurus by replacing following line

./squashfs-root/AppRun -m pip install $(grep -ivE "allosaurus" "${SOURCES_DIR}requirements.txt")

to this one

./squashfs-root/AppRun -m pip install -r "${SOURCES_DIR}requirements.txt"

Have fun! ^__^

[steveway]

That's really nice! I'm currently on vacation. I'll try to test this as soon as possible. I'll be home during the weekend so I might look into that if I find the time, otherwise sometime the week after that or so. Can you distribute binaries frozen with pyinstaller using appimage? Even with allosaurus the windows binaries are "only" around 300MB, so I would think that the result for Linux should be similar.

[morevnaproject]

Can you distribute binaries frozen with pyinstaller using appimage?

I haven't worked with pyinstaller, but I quickly looking at its docs I've got an impresson that it is kind of replacing concept for AppImage. It would be nice to investigate it as alternative method of distribution for Linux, but I have not time for that right now, unfortunately.

Other possibility is to look for redundant files in the build and cleanup them before packing into AppImage, as it is done here - https://github.com/AppImage/AppImageKit/wiki/Bundling-Python-apps
(see code after "# Remove unneeded parts." line)

[morevnaproject]

Quick investigation gives me that the package size is so big because of two files:

opt/python3.7/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so (1,2 Gb)
opt/python3.7/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so (372 Mb)

Discussion on this topic - https://discuss.pytorch.org/t/libtorch-cuda-so-is-too-large-2gb/103155/25

[morevnaproject]

I have forced installation of Torch package without CUDA and enabled Allosaurus. This results in package at 538 Mb which is much better.

I believe there are more a room for optimization by removing redundant files from PySide and maybe other things like torch/tests.

[steveway]

Hi, about the size.
I just tested building this on a Ubuntu VM and the script seems to work nicely.
One of the bigger files inside is libQT5WebEngineCore.so.5 which I believe we don't need if I'm not mistaken.
I'm not exactly sure how the Appimage file gets created exactly with the script and where we can manually remove files using the script inside it.
Oh, I think I see. Yes that works, I pushed a small update which removes that, this shrinks the Appimage a little bit.
I also fixed a warning from QT.
steveway@0825db3
Why is it set to Python 3.7? I saw that there are also prebuild images up to 3.10, for Windows I'm testing this with 3.9 should we use that too for Linux?

[morevnaproject]

Why is it set to Python 3.7? I saw that there are also prebuild images up to 3.10, for Windows I'm testing this with 3.9 should we use that too for Linux?

I remember my first attempt was to make it with Python 3.10 and the resulting AppImage was throwing an error (probably issued by PySide, but not remember exactly). Then I have googled for that error and found some page which was referencing this error as coming starting from Python 3.8. So I have downgraded to Python 3.7 and it worked. Sorry I don't remember exact details.
In any case it is simple to change for other Python version and test. ^__^

[steveway]

Ok, I tested this with the 3.9 manylinux2014 package and that seems to work correctly too.
I am now removing the tests from Torch too to save some space.
And I created a small script which copies the needed project files for Papagayo-NG to a separate folder.
steveway@4efe61e
With these changes I was able to get the size of the AppImage down to 352.1MiB

[morevnaproject]

Awesome! I suggest to use this script to build version 1.4.2 - https://github.com/morevnaproject-org/papagayo-ng/releases/tag/v1.4.2
I can look at OSX build then.