Combined PR

doc/flatpak-support.rst

@@ -0,0 +1,95 @@
+Flatpak Support in IoT Reference OS Kit
+#######################################
+
+IoT Reference OS Kit supports installing and running applications packaged
+as `flatpaks <http://flatpak.org>`_.
+
+Flatpak in a Nutshell
+----------------------
+
+Flatpak is a framework for building, distributing, and managing applications.
+It conceptually splits the software stack into
+
+* a runtime: the core/common bits,
+* an SDK: bits necessary for building software for a runtime,
+* the applications themselves
+
+Flatpak decouples the life-cycle of the application from that of the
+underlying distro. By doing so it enables updating the distro and the
+(flatpak) applications independently from each other.
+
+Additionally flatpak provides application sandboxing out of the box and
+relies largely on `ostree <http://ostree.readthedocs.io>`_ for providing
+an infrastructure for application (binary) version control, distribution
+and deployment.
+
+For more information and details see the corresponding documentation of
+flatpak and ostree.
+
+Basic Flatpak Support
+---------------------
+
+Basic flatpak support includes recipes for flatpak and its runtime and
+buildtime dependencies as well as Yocto helper classes for building
+flatpak-enabled versions of images. It can be enabled by the *flatpak*
+distro feature. Since it has additional prerequisites, the easiest way
+to enable it is to include :file:`meta-flatpak/conf/distro/include/flatpak.inc`
+in your build configuration.
+
+Flatpak Image Variants
+----------------------
+
+When the *flatpak* distro feature is enabled ``meta-flatpak`` defines
+two flatpak-specific image variants:
+
+* *flatpak-runtime*: a flatpak-enabled image to be used on a target device
+* *flatpak-sdk*: a *flatpak SDK runtime* to be used on a (development) host
+
+The *flatpak-runtime* variant adds flatpak, ostree, etc., all the necessary
+runtime bits to run flatpak, to the image. With such an image on a target
+device you should be able to invoke flatpak to carry out the normal flatpak
+operations, including defining flatpak remotes, installing, uninstalling and
+running flatpak applications.
+
+The *flatpak-sdk* variant adds the compiler toolchain, developement packages
+debug symbols, version control software, etc., in short everything you might
+need to compile and turn your applications into flatpaks intended to be
+installed on the *flatpak-runtime* image variant.
+
+You can refer to these image variants by appending their name to that
+of the base image. For instance you can build both of these variants for
+*refkit-image-gateway* by running:
+
+```
+bitbake -c refkit-image-gateway-flatpak-runtime refkit-image-gateway-flatpak-sdk
+```
+
+Extra Flatpak-based Functionality - Flatpak Session
+---------------------------------------------------
+
+In addition to stock flatpak support, *meta-flatpak* provides support
+for running a set of flatpaks from a common remote using a dedicated
+user, monitoring the remote for updates and/or new applications and
+automatically installing and activating those.
+
+Support for this extra set of functionality is controlled by the
+*flatpak-session* distro feature. The easiest way to enable it is to
+include :file:`meta-flatpak/conf/distro/include/flatpak.inc` in your
+build configuration.
+
+See the comment section in :file:`meta-flatpak/classes/flatpak-config.bbclass`
+for more details about the necessary steps or configuration you need to go
+through to have a *flatpak session* set up for a remote on your device.
+
+Caveats
+-------
+
+In addition to the basic flatpak dependencies a *flatpak-runtime* variant
+also pulls into the image a systemd service which is used to activate a
+'fake' flatpak runtime. This is basically a few files and bind mounts that
+are used to emulate an flatpak-installed *flatpak runtime* for the image
+itself. If you don't want it in your image, you can exclude it by putting
+an appropriately crafted _remove for your image in your build configuration.
+Note however, that either this fake runtime or the real flatpak runtime for
+the image needs to be installed, otherwise you cannot install and run
+flatpaks compiled for your image.

doc/howtos/yocto-compatible.rst

@@ -0,0 +1,190 @@
+Yocto Compatible 2.0
+====================
+
+The main goal of the next revision of the Yocto compliance program,
+called "Yocto Compatible 2.0", is to increase inter-operability of
+different layers. One key aspect is that a layer must not change a
+build configuration merely because it gets added to a build. Instead,
+only some explicit build configuration change (like setting distro
+flags or including a .inc file provided by the layer) may cause the
+build to change.
+
+All meta-refkit layers are "Yocto Compatible 2.0". In addition to
+that, it is guaranteed that including :file:`refkit-config.inc` will
+also not change the build. It merely provides support for choosing
+some of the distro features implemented in intel-iot-refkit, without
+actually turning them on. That choice is still left to the developer
+or distro.
+
+The "Layer Overview and Reuse" section in :file:`introduction.rst`
+explains how to use this. This document is about the implementation.
+
+
+Implementation
+==============
+
+Optional ``.bbappends``
+-----------------------
+
+:file:`refkit-config.inc` contains global distro settings, including
+"normal" re-configuration of recipes via ``PACKAGECONFIG``
+changes. Everything that goes beyond that needs to be in a
+``.bbappend``. Often these are workarounds or new recipe features that
+long-term might be more suitable for inclusion into the upstream
+recipe.
+
+Such ``.bbappends`` go into ``meta-refkit-core`` (when they are common
+to multiple profiles) or individual profile layers. Normally a layer
+can only provide ``.bbappends`` for recipes that are known to be
+available. A fatal error aborts parsing when no ``.bb`` file is found
+for a ``.bbappend``, to catch cases where the original ``.bb`` was
+renamed and the ``.bbappend`` must be updated.
+
+``meta-refkit-core/bbappends`` contains a directory hierarchy that
+matches the structure of upstream layers that are used in
+``intel-iot-refkit``. ``BBFILES_DYNAMIC`` instead of the normal
+``BBFILES`` is used to enable ``.bbappend`` files only when the
+corresponding layer is present in a build, so one still gets the error
+when the ``.bb`` file changes, but not when the entire layer is
+missing.
+
+Avoiding Signature Changes
+--------------------------
+
+:file:`refkit-config.inc` and ``.bbappend`` files must not change the
+build when not explicitly enabled via distro features. This is checked by
+comparing task signatures.
+
+There are several different techniques for avoiding task signature
+changes, listed here in order of increasing complexity. Use the simplest
+approach that works.
+
+#. conditional ``_append/remove``
+
+   :file:`refkit-config.inc` defines certain distro features (for
+   example, ``refkit-config``) which get turned into overrides with
+   ``df-`` as prefix. By adding those overrides to ``_append`` or
+   ``_remove``, one can limit that change to configurations where
+   the feature is active::
+
+     do_install_append_df-refkit-config () {
+         ...
+     }
+
+   Combining multiple overrides applies when all overrides are set
+   ("and" semantic)::
+
+     do_install_append_df-refkit-firewall_df-refkit-gateway () {
+        ...
+     }
+
+   Repeating the variable with different conditions applies the change
+   when one or more conditions are met ("or" semantic). But beware
+   that values will be appended more than once when more than one
+   condition is true. This may be acceptable in some cases, like this
+   one::
+
+     EXTRA_OECONF_append_df-refkit-gateway = " --enable-foo"
+     EXTRA_OECONF_append_df-refkit-industrial = " --enable-foo"
+
+#. conditional setting of variables
+
+   Setting a variable with an override suffix chooses that value when
+   the overrides are active::
+
+      PACKAGECONFIG_pn-foo_df-refkit-config = "bar"
+
+   As before, multiple overrides can be combined again ("and"
+   semantic). Assignments with more or more important overrides are
+   chosen over less specific assignments (see the bitbake manual for
+   details).
+
+   When repeating the same assignment for different cases ("or"
+   semantic), a helper variable may be useful to avoid repeating code.
+   Use ``refkit_`` or ``REFKIT_`` as prefix for those::
+
+     REFKIT_PACKAGECONFIG_FOO = "bar"
+     PACKAGECONFIG_pn-foo_df-refkit-gateway = "${REFKIT_PACKAGECONFIG_FOO}"
+     PACKAGECONFIG_pn-foo_df-refkit-industrial = "${REFKIT_PACKAGECONFIG_FOO}"
+
+#. conditional includes
+
+   Manipulating varflags or some operations (like ``addtask``) do not support
+   overrides. Such changes can be placed in a .inc file alongside a ``.bbappend``
+   and then get included conditionally::
+
+     require ${@oe.utils.all_distro_features(d, 'refkit-gateway refkit-firewall', 'gateway-and-firewall.inc') }
+     require ${@oe.utils.any_distro_features(d, 'refkit-gateway refkit-industrial', 'gateway-or-industrial.inc') }
+     require ${@oe.utils.all_distro_features(d, 'refkit-config', 'config1.inc config2.inc') }
+
+   As shown here, one can check for one or more distro features and
+   include one or more files at once. The semantic depends on the
+   function ("or" for ``any_distro_features()``, "and" for
+   ``all_distro_features()``). When only one feature is listed,
+   both functions behave the same. When the condition is not satisfied,
+   these functions return the empty string and nothing gets included.
+
+   They can also be used in a boolean context::
+
+     if oe.utils.all_distro_features(d, 'refkit-config'):
+        bb.note('refkit-config is in DISTRO_FEATURES')
+
+   More complex ``${@ }`` expressions are also possible. The two functions
+   above are merely helper functions that cover the common cases.
+
+   Beware that includes are executed while parsing. Checking for
+   distro features in a ``.bbappend`` is safe because distro features
+   are finalized before parsing recipes. For changes affecting the
+   base configuration, conditional variable changes with overrides
+   have to be used because these conditions are then checked each time the
+   variables are used.
+
+   Checking recipe variables is not safe because those might still be
+   changed later (for example, in another ``.bbappend``).
+
+#. anonymous Python methods
+
+   Anonymous python methods embedded into a ``.bbappend`` can make
+   arbitrary changes to the recipe after checking for some condition.
+   In contrast to conditional includes, anonymous Python methods
+   are executed at the end of parsing and thus can typically check
+   recipe variables. The only caveat is that another method might
+   still change those variables.
+
+   This can be used to replace the ``addtask`` directive without
+   having to create a separate ``.inc`` file::
+
+     python () {
+        if bb.utils.contains('IMAGE_FEATURES', 'ostree', True, False, d) and \
+           oe.utils.all_distro_features(d, 'ostree'):
+           bb.build.addtask('do_ostree_prepare_rootfs', 'do_image_wic', 'do_image', d)
+     }
+
+
+Testing
+=======
+
+``oe-selftest -r refkit_poky.TestRefkitPokySignatures`` applies the
+``yocto-compatible-layer.py`` test script to all ``meta-refkit``
+layers against a base configuration which mirrors the ``Poky``
+distribution. The output in case of a failure contains differences
+between task signatures, which usually highlights were some undesired
+change is happening.
+
+``oe-selftest`` can be run in a local build configuration, without
+affecting the build itself. See ``oe-selftest --list-tests`` for a
+full list of tests that can be run this way.
+
+The compatibility tests do not guarantee that actual builds will work,
+because they only check syntax, dependencies and task
+signatures. Actual building is covered by ``oe-selftest -r
+refkit_poky.TestRefkitPokyBuilds`` for two build configurations:
+
+test_common_poky_config
+  Building ``refkit-image-common`` using just the plain ``Poky``
+  distribution settings (for example, without ``systemd``).
+
+test_common_refkit_config
+  Building ``refkit-image-common`` using the ``Poky``
+  distribution settings plus :file:``enable-refkit-config.inc``,
+  i.e. all refkit distro features set.

meta-flatpak/classes/flatpak-config.bbclass

@@ -63,3 +63,31 @@ FLATPAK_BUILD  ?= "${DISTRO}/${FLATPAK_PN}/build/${BUILD_ID}"
 # This is the GPG key id of our repository signing key. If you set this to
 # empty, signing is disabled altogether.
 FLATPAK_GPGID ?= "refkit-signing@key"
+
+# We can pre-populate the image during build with a set of pre-declared
+# flatpak repositories and associated dedicated flatpak-session users.
+# These repositories will be monitored for flatpaks and any new or updated
+# flatpaks which are tagged for automatic installation will be pulled in
+# and started within the appropriate flatpak session.
+#
+# To declare such a remote/repository, you need to
+#   1) give the remote a unique name
+#   2) provide a remote URL and a GPG public key for signature checking
+#   3) associate a user with the remote
+#
+# The variable FLATPAK_APP_REPOS lists the names of the remotes to pre-
+# declare and put into the image. For every remote <r> you have to provide
+#
+#   1) the remote HTTP URL in ${TOPDIR}/conf/<r>.url
+#   2) the remote GPG signing public key in ${TOPDIR}/conf/<r>.key
+#   3) provide passwd- and group entries for the associated user in
+#        ${TOPDIR}/conf/flatpak-passwd and
+#        ${TOPDIR}/conf/flatpak-group
+#      The GECOS for the associated entry must be of the form
+#      'flatpak user for <r>', where <r> is the remote name. By
+#      convention the user's login ID is also <r>, but this is
+#      not strictly necessary.
+#
+# By default we do not have any pre-declared application repositories,
+# so FLATPAK_APP_REPOS defaults to empty.
+FLATPAK_APP_REPOS ?= ""

meta-flatpak/classes/flatpak-image-variants.bbclass

@@ -8,6 +8,8 @@ IMAGE_FEATURES[validitems] += " \
 
 FEATURE_PACKAGES_flatpak = " \
     packagegroup-flatpak \
+    ${@bb.utils.contains('DISTRO_FEATURES', 'flatpak-session', \
+           'packagegroup-flatpak-session', '', d)} \
 "
 
 #

meta-flatpak/conf/distro/include/flatpak-session.inc

@@ -0,0 +1,3 @@
+REFKIT_DEFAULT_DISTRO_FEATURES += " \
+    flatpak-session \
+"

meta-flatpak/conf/layer.conf

@@ -14,3 +14,12 @@ LAYERDEPENDS_flatpak-layer = "core openembedded-layer filesystems-layer"
 
 # Set a variable for easy access to the top directory of the flatpak layer.
 FLATPAK_LAYERDIR = '${@os.path.normpath("${LAYERDIR}")}'
+
+# Pick up our passwd and group fragments for users associated with
+# pre-declared remotes/repositories, if we have any.
+USERADD_UID_TABLES_append_refkit-config = " \
+    ${@ d.getVar('TOPDIR') + '/conf/flatpak-passwd' \
+            if d.getVar('FLATPAK_APP_REPOS') else ''}"
+USERADD_GID_TABLES_append_refkit-config = " \
+    ${@ d.getVar('TOPDIR') + '/conf/flatpak-group' \
+            if d.getVar('FLATPAK_APP_REPOS') else ''}"

meta-flatpak/recipes-core/packagegroups/packagegroup-flatpak-session.bb

@@ -0,0 +1,10 @@
+SUMMARY = "IoT Reference OS Kit Basic Flatpak Session/Application Support"
+LICENSE = "MIT"
+
+inherit packagegroup
+
+RDEPENDS_${PN} = "\
+    packagegroup-flatpak \
+    flatpak-utils \
+    flatpak-predefined-repos \
+"