"flatpak install" doesn't use static deltas when downloading repository data #3412
Description
Linux distribution and version
Debian 10 (Buster) and Fedora 31 (tested both)
Flatpak version
1.2.5 (Debian) and 1.4.3 (Fedora)
Description of the problem
Flatpak documentation advertises that you can generate a repository with static deltas to make downloads much faster.
From: https://docs.flatpak.org/en/latest/hosting-a-repository.html
Flatpak supports something called static deltas. These are single files that contain all the data needed to go between two revisions (or from nothing to a revision). Creating such deltas will take up more space on the server, but will make downloads much faster. This can be done with the flatpak build-update-repo --generate-static-deltas option.
From: https://blogs.gnome.org/alexl/2017/02/10/maintaining-a-flatpak-repository/
Additionally, as objects are whole files, we still have to download the entire file even if just one byte in the file changed.
In order to solve this flatpak supports something called static deltas. These are pre-generated delta files (using bsdiff) that contain all the data needed to go from one version to another (or from nothing to a version). If these files are available they are used instead of the individual objects, which allows pulls to be much more efficient.
You can generate static deltas for the latest versions of all apps by passing --generate-static-deltas to flatpak build-update-repo, and I recommend everyone do this for production repositories.
The problem is that the command flatpak install seems unable to use static deltas in practice when downloading the repository data, and it downloads all the objects one by one, making the download to be very slow for a repository that contains lot of small objects (lot of small files), even when the repository has enabled static deltas.
Downloading the very same repository with ostree pull instead of with flatpak install works as expected and static deltas are used, therefore making the download much faster.
Steps to reproduce
1. test: flatpak install (static deltas not working)
Try to download the following test repository with flatpak install and see how the download speed its slow (less than 2Mbytes/sec).
rm -fr /tmp/test-fp
mkdir /tmp/test-fp
export FLATPAK_USER_DIR=/tmp/test-fp
cd /tmp/test-fp
flatpak remote-add --user webkit-sdk https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo
flatpak install --user webkit-sdk org.webkit.Sdk 0.1 --assumeyes --no-related
On top of the obvious slow download, you can verify that flatpak its downloading the (many small) individual objects instead of the deltas by checking the many HTTP petitions its doing, either with wireshark/tcpdump or by passing the environment variable OSTREE_DEBUG_HTTP=1 and counting the GET petitions it reports the debug log.
rm -fr /tmp/test-fp
mkdir /tmp/test-fp
export FLATPAK_USER_DIR=/tmp/test-fp
cd /tmp/test-fp
flatpak remote-add --user webkit-sdk https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo
OSTREE_DEBUG_HTTP=1 flatpak install --user webkit-sdk org.webkit.Sdk 0.1 --assumeyes --no-related 2>&1 | grep GET | wc -l
And the last command (wc -l) reports (after a long while): 85751 (that its, it did more than 85k GET petitions!, that its clearly not using deltas)
2. test: ostree pull (static deltas working)
Now let's do the same but instead of flatpak install let's use ostree pull to pull the very same repository.
rm -fr /tmp/test-fp
mkdir /tmp/test-fp
export FLATPAK_USER_DIR=/tmp/test-fp
cd /tmp/test-fp
flatpak remote-add --user webkit-sdk https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo
ostree --repo=repo pull webkit-sdk runtime/org.webkit.Sdk/x86_64/0.1
Now you can see the download speed much faster. Then you can also see how OSTree prints its downloading a few deltas only. On top of that you can verify its the case by debugging it like previously:
rm -fr /tmp/test-fp
mkdir /tmp/test-fp
export FLATPAK_USER_DIR=/tmp/test-fp
cd /tmp/test-fp
flatpak remote-add --user webkit-sdk https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo
OSTREE_DEBUG_HTTP=1 ostree --repo=repo pull webkit-sdk runtime/org.webkit.Sdk/x86_64/0.1 2>&1 | grep GET | wc -l
In this case (with ostree pull), the last command completes much faster and (wc -l) only reports: 295 GET petitions (because it used deltas for downloading)
This very same issue its reproducible with other flatpak repositories, like for example, the LibreOffice one on FlatHub, but there the issue its less obvious because the LibreOffice flatpak repository has much less objects than the one from this example.
The one from this example has 85744 objects with an average size of 10kb per object, meanwhile the LibreOffice one (app/org.libreoffice.LibreOffice/x86_64/stable) has "only" 5669 objects with an average size of 46Kb per object.
Downloading the objects individually instead of using the static deltas its a problem for this kind of flatpak repositories with so many small objects. I would like "flatpak install" to support downloading it using static deltas like "ostree pull" does.