Flatpak recipes

[m-kuhn]

Flatpak is a cross-distribution way of shipping applications to users including dependencies (bye-bye Qt 5.6)

There is a readme included in this pr, that explains how to build it. It's potentially the simplest way to build QGIS yourself ;)

My long-term goals are

• Building packages directly on the CI (be it Flatpak or AppImage or whatever)
• Shipping packages that have been tested - including the dependencies
• Automatically ship master builds from CI (No need to wait for nightlies)
• Potentially even ship builds from pull requests (Get feedback of non-dev people)

Installing

Flatpak is available on various distributions, although in general only in their
very latest release. Get flatpak!

Building

Since we don't have an online repository (yet), you need to build a package locally.

In the QGIS sourcecode folder

cd flatpak
./create-flatpak.sh
# Wait for everything to be built, this takes some time when done the first time

flatpak --user remote-add --no-gpg-verify qgis-repo-local repo
flatpak --user install qgis-repo-local org.qgis.qgis

[probonopd]

You might be interested in linuxdeployqt. It allows to bundle a Qt application as a standalone bundle that runs on various Linux distributions. As a side effect, it can also generate an AppImage.

[m-kuhn]

@probonopd that sounds interesting.

Am I correct, that

• if we build on precise it will be compatible with almost everything available today?
• it will just package "whatever is around" (libs from the repository, downloaded, or locally compiled)
• there is a possibility to update deltas

[probonopd]

@m-kuhn yes, you are correct

[rduivenvoorde]

@m-kuhn very interesting!

Tried your branch (Debian testing here...)

--2016-11-20 11:41:05--  http://distribute.kde.org/kdeflatpak.gpg
Resolving distribute.kde.org (distribute.kde.org)... 2a01:4f8:162:31a2::4, 5.9.99.188
Connecting to distribute.kde.org (distribute.kde.org)|2a01:4f8:162:31a2::4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1219 (1.2K)
Saving to: ‘kdeflatpak.gpg’

kdeflatpak.gpg       100%[=====================>]   1.19K  --.-KB/s    in 0s      

2016-11-20 11:41:05 (57.9 MB/s) - ‘kdeflatpak.gpg’ saved [1219/1219]

Installing: org.kde.Platform/x86_64/master from kde

1360 metadata, 24729 content objects fetched; 275225 KiB transferred in 331 seconds
Installing: org.kde.Platform.Locale/x86_64/master from kde

1 metadata, 0 content objects fetched; 313 B transferred in 0 seconds              
Installing: org.kde.Sdk/x86_64/master from kde

3484 metadata, 38646 content objects fetched; 188240 KiB transferred in 342 seconds                                                                                      
Installing: org.kde.Sdk.Locale/x86_64/master from kde

1 metadata, 0 content objects fetched; 313 B transferred in 0 seconds                                                                                                    
./create-flatpak.sh: line 9: flatpak-builder: command not found
error: Build directory qgis-app not initialized, use flatpak build-init
richard@kwik~/dev/qgis/git/qgis/flatpak (pr/3784)$ 

ah... flatpak-builder is a separate package/binary. apt-get install flatpak-builder

Still building :-)...

Seeing this in QGIS make step:

```-- Found SIP version: 4.18
Traceback (most recent call last):
  File "/run/build/qgis/cmake/FindQsci.py", line 45, in <module>
    from PyQt5.Qsci import QSCINTILLA_VERSION_STR
ImportError: No module named 'PyQt5.Qsci'
FindGSL.cmake: gsl-config not found. Please set it manually. GSL_CONFIG=GSL_CONFIG-NOTFOUND
-- txt2tags not found - disabled
-- Qsci sip file not found - disabling bindings for derived classes

That takes a long time!
But ending with:

Committing stage build-qgis to cache
Cleaning up
Rewriting contents of org.qgis.qgis.desktop
Committing stage cleanup to cache
Finishing app
Exporting share/applications/org.qgis.qgis.desktop
Please review the exported files and the metadata
Committing stage finish to cache
Pruning cache
WARNING: Binary not found for Exec line in /home/richard/dev/qgis/git/qgis/flatpak/qgis-app/export/share/applications/org.qgis.qgis.desktop: /usr/bin/qgis %F
WARNING: Icon not matching app id in /home/richard/dev/qgis/git/qgis/flatpak/qgis-app/export/share/applications/org.qgis.qgis.desktop: qgis
WARNING: Icon referenced in desktop file but not exported: qgis

Ok. I presume?

Run it with:

flatpak run org.qgis.qgis

Starting with:

Traceback (most recent call last):
  File "", line 1, in 
  File "/app/share/qgis/python/qgis/utils.py", line 20, in 
    from future import standard_library
ImportError: No module named 'future'

So no Python yet? I'm also wondering which dir is used for configs..

But nice:

QGIS version
2.99.0-Master
QGIS code branch
Release 2.99
Compiled against Qt
5.7.0
Running against Qt
5.7.0
Compiled against GDAL/OGR
2.1.2
Running against GDAL/OGR
2.1.2
Compiled against GEOS
3.6.0-CAPI-1.10.0
Running against GEOS
3.6.0-CAPI-1.10.0 r0
PostgreSQL Client Version
9.6.0
SpatiaLite Version
4.3.0a
QWT Version
6.1.2
PROJ.4 Version
493
QScintilla2 Version
2.9.3

Thanks for this journey!!

[m-kuhn]

Thanks for testing Richard!

Building only takes so long the first time, because all dependencies need to be built as well (and the ccache for building qgis is still cold). Do it again and it will be fast as a lightning.

If we go down this road, we could also send all the dependencies into a separate runtime/sdk so they would live in a separate package and could be downloaded prebuilt.

I think configuration is sent to

.var/app/org.qgis.qgis/config/QGIS/QGIS3.conf

Python support is compiled (sip/pyqt/qgis bindings are available) but there are a number of packages still missing, python3-future probably being to most critical one.

And there's certainly a lot to be cleaned up and improved

[m-kuhn]

@probonopd I gave it a short try and it was very easy to create a package. Running it gave me an error

$ ./output/bin.AppImage 
./output/bin.AppImage: error while loading shared libraries: libqgis_app.so.2.99.0: cannot open shared object file: No such file or directory

Can I manually specify additional libraries to bundle?

Maybe -libpath? It could be but I'm not convinced...

-libpath= : Add the given path to the library search path

[probonopd]

@m-kuhn if your application links libqgis_app.so.2.99.0 and you use the -bundle-non-qt-libs option then libqgis_app.so.2.99.0 should be detected and included by linuxdeployqt automatically. There is a bug that sometimes requires linuxdeployqt to be run twice.

Did you use the -bundle-non-qt-libs option?

[m-kuhn]

@probonopd Yes, -bundle-non-qt-libs was passed.

I was trying to package directly from the build folder (i.e. without make install step).

Something else, there are other resources (additional libs which are loded as plugins, sqlite databases etc.) which need to be packaged as well.

I think it would be easiest if this could be installed to a custom prefix and packaged from there, is this possible?

[probonopd]

Yes, that is the "FHS-like mode" that I have started to work on (in a separate feature branch) but it still has some issues and needs some love. If you can invest some time in this, your help and PRs would greatly be appreciated.

[m-kuhn]

@probonopd I might have some time to look into this again soon. I noticed that your feature branch has disappeared, does that mean this is merged and functional?
The goal for this would be that

• we can have builds of master available somewhere (after_success script on travis)
• on other independent projects using travis ci, these builds/bundles can be downloaded
• unit tests can be executed within this environment (more specifically with python modules shipped within the bundle)

Questions therefore:

• is this possible?
• how will this be dealing with python? Is it the system python version or a bundled one that will/should be used?

Thanks for any input!

[probonopd]

Yes, this is now in the appimagetool/master branch. You should bundle Python and its dependencies inside the AppImage. The AppRun wrapper script/executable sets $PYTHONPATH to the usr/ inside the AppImage. Let me know if you are running into issues.

[Lvaskz]

https://flathub.org/
https://flathub.org/apps.html
https://github.com/flathub/flathub
https://flathub.org/builds/#/
https://github.com/flatpak/flatpak-builder

Hello, now released flatpak builder and leave flatpak links in the hub. I hope to see qgis 2.18. and 3.0 soon in these easy to install formats. I am not a programmer so I can not help much. Thank you

[Lvaskz]

Hello, now released flatpak builder and leave flatpak links in the hub. I hope to see qgis 2.18. and 3.0 soon in these easy to install formats. I am not a programmer so I can not help much. Thank you

[alexlarsson]

You should probably use the stable kde runtime version 5.9 which is available on flathub, rather than the unstable master version.

[alexlarsson]

We would love to have this on flathub (although it needs to be "last stable relases" rather than a snapshot, thats what flathub targets). If you're interested in this, here are some docs on how to submit it:
https://github.com/flathub/flathub/wiki/Submission-Guidelines

[aleixpol]

Would it make sense to look into this again? Last QGIS is based on Qt 5.

[Lvaskz]

Has any progress in appimages or flatpak Qgis?

[m-kuhn]

This is still a very interesting topic I'd like to see land but due to time constraints (and a project which I thought I could use as umbrella for this took a different direction) I cannot make it at the moment.

Is there anyone else available to take a lead on this?

[palmerj]

I still have a question about creating a QGIS snap vs flatpak. Its obvious the community doesn't have the resources to support both. @m-kuhn why did you prefer to do this initial work in flatpak?

[m-kuhn]

Because I assumed snap is another candidate for a Canonical NIH-syndrome-project like upstart and mir. Not sure about the status nowadays.

[luipir]

my little experience with snap it creates more common user problems due to sandbox mechanism... other that building in dev mode the package. It's simpler to use docker with the flexibility to mount volumes. No idea if flatpak is affected to the same security limitations.

[nyalldawson]

@luipir I don't think the two are mutually exclusive -- docker is great, but it's not a good solution for end users. Flatpak/snappy are designed to be gui-friendly for easy install with no technical experience.

My understanding is that currently flatpak is giving the better desktop app experience than snaps (but I've only direct experience with flatpak, so I'm going off 2nd hand information)

[alexbruy]

There is also AppImage which seems more simpler to use for end-users (there is no need to install additional packages, just download file and run it) and does not require root rights.

[luipir]

@nyalldawson exactly from the user experience point of view I'm talking about... I'm currently using CloudCompare that, on linux, is distributed via snap and it's hard to use you local data (that are really BIG data) nor linking... the only (simple) way is to duplicate them, that is time/space consuming! At the end I had to compile from source to be usable.
BTW probably 99% of users do not case of these limitations and they use not so big data!
so my concern if distributing with snap will introduce a new class of user issues like... where are my data!

[nyalldawson]

@luipir I think that's more than likely just an issue with that particular snap not requesting the correct permissions to access local files.

[m-kuhn]

There is also AppImage which seems more simpler to use for end-users (there is no need to install additional packages, just download file and run it) and does not require root rights.

That's probably the way I'd go now. Leave repository approaches to package maintainers / distributions while offering a downloadable executable which works out of the box.

[palmerj]

I'm wondering about Flatpak as well. From what I have read it can't run CLI tool (e.g newer GDAL versions shipped with QGIS) or server processes (.eg QGIS server). See https://askubuntu.com/questions/866511/what-are-the-differences-between-snaps-appimage-flatpak-and-others.

If we used AppImage then we could could distribute it via qgis.org and provide the option to auto-update.

[alexlarsson]

Flatpak can run both server and CLI tools, there is nothing magic about such programs after all. However, the focus of flatpak is on GUI tools, so that is where all the features for integration are. This also means that it doesn't have the features you expect in a tool focused on server use, like docker or kubernetes. For instance there is nothing handling firewall/network management. load balancing or migration.

[m-kuhn]

For me the main usage for this would be shipping development (or recently new) releases to users as quickly and easily as possible. What's your main goal with integrating these package formats?

For server components the same thing can be done conveniently using technologies like docker images, so that would make it a second priority.

[palmerj]

Flatpak can run both server and CLI tools, there is nothing magic about such programs after all.

Ok sorry. I was only going on the askubuntu.com post.

For me the main usage for this would be shipping development (or recently new) releases to users as quickly and easily as possible.

I would say a real benefits are user who are on older distros that don't have the newer dependences, or want to run side by side versions of QGIS. We have both of these situations in our department.

For server components the same thing can be done conveniently using technologies like docker images, so that would make it a second priority.

Yes that is true. Ignore that feature then.

[aleixpol]

Here's some flatpak packaging, in case someone wants to review:
flathub/flathub#529

[Lvaskz]

hi, guys thanks for qgis 3.2 flatpak. Is there some plan to make 2.18 version flatpak or appimage?

[aleixpol]

For completion, 3.2.3 can be obtained here: https://flathub.org/apps/details/org.qgis.qgis
The packaging here: https://github.com/flathub/org.qgis.qgis

If any member of QGIS upstream want to review and help, that's more than welcome.

[nyalldawson]

@aleixpol I did have a brief look, but don't feel qualified to give a proper review. At least, nothing stood out to me. But, I just want to say that your efforts here are VERY much appreciated and that having flatpaks available is just fantastic.

[aleixpol]

❤️

FWIW this issue can probably be closed.

[nyalldawson]

@aleixpol it already is 😛

[m-kuhn]

Thanks @aleixpol
Much appreciated !

[rduivenvoorde]

@aleixpol @nyalldawson @m-kuhn is this package stable/mature enough to be put on the 'download' pages in qgis.org? Or should we try to encourage people to use it a little further first?

[m-kuhn]

Putting it there sounds good. Maybe put an "experimental" or "beta" label next to it (and remove it when 3.4 hits the osgeo4w ltr branch - unless a good reason to keep the beta label has surfaced by then).

[Lvaskz]

hi there, with this version qgis 3.2 the repo plugins is not loaded. I opened a issue with this.flathub/org.qgis.qgis#5

[rduivenvoorde]

Ok, I will test an install, and add some instructions to the docs

@Lvaskz as mentioned in the issue reply, adding --share=network just works :-)

@aleixpol Hi, trying to install here on a Debian Gnome box.
Just a question: I'm asked to install org.kde.Platform runtimes. Is that because QGIS is Qt (so KDE contains all Qt stuff) based? OR is it the way it is packaged (and more Gnome-based packaging would be an alternative (I understand Qt is essential off course...))

[rduivenvoorde]

@aleixpol commenting, here unless you want an issue for it:

QGIS 3.2.3 is running nicely here next to my other QGiS's (self compiled, installed in my homedir) !
Thanks for all this work !

One thing I noted: the flatpak QGIS is not using the 'normal' QGIS profile folder paths, so instead of:
~/.local/share/QGIS/QGIS3/profiles/default
it uses:
~/.var/app/org.qgis.qgis/data/QGIS/QGIS3/profiles/

Is this a flatpak convention? Or is it to be safe for now?
If I'm correct the profile folder location chosen by QGIS is some general profile-convention (for every OS). And I think(!) that, in later versions, if all is safe, we should use the same profile location. Whatever way you are running QGIS. Do others confirm?

[palmerj]

I noticed 3d support is not working for me. I've raised an issue for that

[aleixpol]

@aleixpol commenting, here unless you want an issue for it:

QGIS 3.2.3 is running nicely here next to my other QGiS's (self compiled, installed in my homedir) !
Thanks for all this work !

yay!

One thing I noted: the flatpak QGIS is not using the 'normal' QGIS profile folder paths, so instead of:
~/.local/share/QGIS/QGIS3/profiles/default
it uses:
~/.var/app/org.qgis.qgis/data/QGIS/QGIS3/profiles/

Is this a flatpak convention? Or is it to be safe for now?
If I'm correct the profile folder location chosen by QGIS is some general profile-convention (for every OS). And I think(!) that, in later versions, if all is safe, we should use the same profile location. Whatever way you are running QGIS. Do others confirm?

Flatpak does it this way to separate every application in a sandbox effectively. This way every application doesn't get to read and mess with each other's configuration.

[Lvaskz]

Grass gis 7.4 come not with qgis flatpak., I reported this like issue
flathub/org.qgis.qgis#7

[palmerj]

Is anyone else get a startup message of:

Warning: libpng warning: iCCP: known incorrect sRGB profile

Maybe some PNG file needs to have it's iCCP chunk removed

[nyalldawson]

The libpng warning is caused by PNGs inside qtwebkit -- nothing we can fix unfortunately

[palmerj]

Ok thanks.

[palmerj]

The flatpack desktop icon doesn't seem to have a transparent background. See comparison with Debian package install:

I've raised an issue for this flathub/org.qgis.qgis#9

[reginaldc]

Hi,

When I try to install qgis with flatpack I get:
Failed to install org.qgis.qgis/x86_64/stable: GPG signatures found, but none are in trusted keyring
I did the setup in:
https://flatpak.org/setup/Ubuntu/
after I rebooted I did without sudo:
flatpak install flathub org.qgis.qgis
Any idea?

Thanks,

[rduivenvoorde]

@reginaldc please create an issue at https://github.com/flathub/org.qgis.qgis/issues as that is where the flatpak pro's are, but my guess is that your flatpak install is not ok, or you miss keys?
Are you able to install other flatpak's?

[alexlarsson]

This is a temporary issue with the flathub CDN, and it will hopefully go away soon. Tracked here: flatpak/flatpak#1450

[reginaldc]

@richard
I just installed the spotify client and it works alright.
Im gonna check out the comment of alex and see if there is a solution.

[Lvaskz]

Hi guys, Can some one of you check if this issues flathub/org.qgis.qgis#10
flathub/org.qgis.qgis#7 are solved? I can not get any update or any improvements. I updated millions of times and I didn't get any update.