Skip to content
A tool for checking if the external data used in Flatpak manifests is still up to date
Python Other
  1. Python 98.3%
  2. Other 1.7%
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/workflows
src
tests
wrappers
.flake8
.gitignore
Dockerfile
Jenkinsfile
LICENSE
README.md
canonicalize-manifest
flatpak-external-data-checker
run-in-container.sh

README.md

Flatpak External Data Checker

This is a tool for checking for outdated or broken links of external data in Flatpak manifests.

Motivation

Flatpak apps are built using external data (git checkouts, tarballs, simple files, etc.). A very specific case of this is the use of extra data, which works as a way to download third party binaries at installation time.

Of course, the links pointing to external data can become obsolete, so it is very important to account for and correct such issues. This is especially critical for the extra data, in which a broken link impedes the installation of the application.

This project offers ways to easily check or monitor the state of such links, as well as the suggestion of new versions.

It works by extracting all the external data of a Flatpak manifest and giving it to a collection of checkers, which will set up the right state and, possibly, new versions of each external data.

Use

The simplest use of this tool is by calling:

flatpak-external-data-checker MANIFEST_FILE

it should display messages about any broken or outdated external data.

Running in a container with Podman

run-in-container.sh builds a container using podman and the Dockerfile in this repository, then runs it with appropriate UID and filesystem mappings. This is a convenient way to get all necessary dependencies regardless of your host system's distribution.

~/src/endlessm/flatpak-external-data-checker/run-in-container.sh \
    ~/src/endlessm/flatpak-external-data-checker/flatpak-external-data-checker \
    [ARGS …] \
    ~/src/flathub/com.example.App/com.example.App.json

Automatically submitting PRs

When run with the --update flag, this tool can commit any necessary changes to Git and open a GitHub pull request. In order to do this, it needs to have two credentials:

  • A GitHub access token, specified in the GITHUB_TOKEN environment variable
  • git push access for the repository containing the manifest passed on the command line; ie an appropriate ssh key

Changes to Flatpak manifests

For simple checks to see if a URL is broken, no changes are needed. However, you can add additional metadata to the manifest to allow the checker to discover new versions.

URL checker

If the upstream vendor has an URL that redirects to the latest version of the application, you can add something like the following to check and update the URL for the latest version:

"x-checker-data": {
    "type": "rotating-url",
    "url": "http://example.com/last-version",
    "pattern": "http://example.com/foo-v([0-9.]+).tar.gz"
}

The version number for the latest version can be detected in two ways:

  • If the filename ends with .AppImage, the version number is extracted from the AppImage. (It is run in a bwrap sandbox.)
  • Otherwise, if "pattern" is specified in "x-checker-data", the given regular expression is matched against the full URL for the latest version, and the first match group is taken to be the version. (This follows the convention used by debian/watch files.)

HTML checker

Both the version number and the download URL will be gathered from a static HTML page which contains this information:

"x-checker-data": {
    "type": "html",
    "url": "https://www.example.com/download.html",
    "version-pattern": "The latest version is ([\d\.-]*)",
    "url-pattern": "https://www.example.com/pub/foo/v(\d+)/foo.tar.gz"
}•

Debian repo checker

For the DebianRepoChecker, which deals only with deb packages, it can read the following metadata (add it to manifest element it refers to, e.g. where "type": "extra-data" is declared):

"x-checker-data": {
    "type": "debian-repo",
    "package-name": "YOUR_PACKAGE_NAME",
    "root": "ROOT_URL_TO_THE_DEBIAN_REPO",
    "dist": "DEBIAN_DIST",
    "component": "DEBIAN_COMPONENT"
}

Firefox checker

For the FirefoxChecker, which deals only with new versions of Firefox and its translations, it can read the following metadata (opposed to other checkers, this metadata should be added to the module element instead of the source element in the manifest):

"x-checker-data": {
    "type": "firefox",
}

The checker will try to match the module sources to be checked as follows:

  • The module source with filename "firefox.tar.bz2" will be considered to refer to the firefox tarball (at least one source should refer to the firefox tarball)
  • All sources whose filename has ".xpi" as extension will be considered translations

Also note that translations will be automatically added and removed depending on what is available upstream.

Flash player checker

For the FlashChecker, which deals only with new version of Adobe Flash player, it can read the following metadata (add it to the manifest element it refers to, e.g. where "type": "extra-data" is declared):

"x-checker-data": {
    "type": "flash",
    "browser": "chrome|firefox"
}

The value for "browser" is used to determine whether to download the Firefox-compatible Flash binaries (npapi) or the Chrome-compatible ones (ppapi).

In addition, if you add that metadata, you must set only-arches on the extra-data source itself to i386 or x86_64:

  "type": "extra-data",
  "only-arches": ["i386"],
  // ...

FlashChecker will use this to determine which architecture to check the binaries for.

Contributing

# Run all the tests (some of which need an internet connection):
./run-in-container.sh python3 -m unittest discover

# Run one suite of tests
./run-in-container.sh python3 -m unittest tests.test_appdata

# More information
./run-in-container.sh python3 -m unittest --help

You can of course install the necessary modules on your host system and omit ./run-in-container.sh if you prefer.

License and Copyright

License: GPLv2

Copyright © 2018–2019 Endless Mobile, Inc.

You can’t perform that action at this time.