Merge pull request #548 from ubports/personal/peat-psuwit/focal-porting

Add guides on how porters can update their ports for 20.04

index.rst

@@ -69,6 +69,7 @@ You may view this documentation in the following languages:
    porting/build_and_boot/index.rst
    porting/configure_test_fix/index.rst
    porting/finalize/index.rst
+   porting/UpdatePortsFor2004.rst
 
 .. toctree::
    :maxdepth: 1

porting/UpdatePortsFor2004.rst

@@ -0,0 +1,64 @@
+.. _UpdatePortsFor2004:
+
+Update ports for Ubuntu Touch 20.04
+===================================
+
+Ubuntu Touch 20.04 introduces a lot of changes underneath, which requires changes from the porters. This document is intended to serves as an overview on how to make your port runs on Ubuntu Touch 20.04. However, every port is different, and may requires more than what is outlined in this document. We'll be available in the UBports porting Telegram group to answer any question that might occur during process.
+
+For GitLab CI-based ports: switch to shared building script & GitLab CI configuration
+-------------------------------------------------------------------------------------
+
+Because of changes in how Ubuntu Touch 20.04's rootfs is made, we need to changes the way device tarballs are built. Instead of cherry-picking script changes from other port's repository, we recommend transitioning to the shared building script. By using the shared script, your port will receive the latest changes in device tarball building as rootfs continue to evolves (e.g. for 22.04 or newer). To transition to the shared script, remove the whole ``build/`` directory, and replace the content of ``build.sh`` with:
+
+.. code-block:: bash
+
+    #!/bin/bash
+    set -xe
+
+    [ -d build ] || git clone https://gitlab.com/ubports/community-ports/halium-generic-adaptation-build-tools build
+    ./build/build.sh "$@"
+
+You may want to add ``build/`` into your ``.gitignore``.
+
+And for the same reason, we recommend using the shared GitLab CI configuration so that when the shared script has its requirement changes, your port will not be broken. Using the shared config can be done by replacing the content of ``.gitlab-ci.yml`` with:
+
+.. code-block:: yaml
+
+    include:
+        - https://gitlab.com/ubports/porting/community-ports/halium-generic-adaptation-build-tools/-/raw/main/gsi-port-ci.yml
+
+    variables:
+        BUILD_DEVEL_FLASHABLE_FOCAL: "1"
+
+The ``variables`` section enables building the flashable partition image from the Focal rootfs, which you might want to leave out for intial porting.
+
+.. note::
+    The script currently doesn't handle the single repository which builds for multiple devices. If your port needs this feature, please upvote `this issue <https://gitlab.com/ubports/porting/community-ports/halium-generic-adaptation-build-tools/-/issues/5>`_.
+
+If you have custom changes in the build script or in the GitLab CI configuration, you may need to carefully inspect the changes to make sure that your port will still work. Note that you can add additional steps to the GitLab CI configuration and order it after the device tarball building steps, or even overriding some steps (not recommended).
+
+For GitLab CI-based ports: inspect your overlay in Android partitions
+---------------------------------------------------------------------
+
+Many ports overlay the files in Android partitions to makes the port work. This is often achived by overlaying the ``mount-android.conf`` Upstart job to either contain the code itself, or to call another script. Since Ubuntu Touch 20.04 no longers use Upstart (sees below), the code will no longer run. If your port ships your overlay files under ``/opt/halium-overlay`` or ``/usr/share/halium-overlay``, the new overlay system will take care of it automatically, except when the new file is added in which case you may need ``.halium-overlay-dir`` (see :ref:`Overlay`).
+
+Alternatively, you may want to consider moving your port completely to use the `overlaystore` system. However the migration is not straight forward and it will make your port incompatible with 16.04 (unless you have another branch). Contact @peat-psuwit in the UBports porting group for more info.
+
+For Halium 7 ports or older: update bluetooth-touch script
+----------------------------------------------------------
+
+The way bluetooth-touch script used to work is inherently tied to Upstart. In Ubuntu Touch 20.04, bluetooth-touch is migrated to use Systemd. So, instead of overlaying ``bluetooth-touch-android.conf`` Upstart job, you'll have to overlay ``/usr/share/bluetooth-touch/android.sh``. Note that the script will be run by ``/bin/sh``, so avoid Bash-ism here.
+
+Halium 9 ports doesn't require this, as Bluebinder replaces ``bluetooth-touch`` by directly talking to HIDL service to provide Bluetooth host interface.
+
+For everyone: configure usb-moded
+---------------------------------
+
+usb-moded replaces ``setupusb`` script in setting and managing USB modes. We've written a specific page for that. See :ref:`USBModed`.
+
+For everyone: check custom Upstart jobs
+---------------------------------------
+
+Some ports has custom Upstart jobs to do certain things on boot. Those jobs has to be converted to Systemd units in order to run at all in Ubuntu Touch 20.04. One may find `Ubuntu's guide on this topic <https://wiki.ubuntu.com/SystemdForUpstartUsers#Job_vs._unit_keywords>` useful.
+
+Alternatively, instead of writing custom Systemd units, use ``devicehack`` script which will run on every boot after the Android container runs. The path is ``/usr/libexec/lxc-android-config/device-hacks``, and the script will be run with ``/bin/sh``.

porting/configure_test_fix/Overlay.rst

@@ -146,14 +146,20 @@ After a successful build, the resulting ``system.img`` must reflashed together w
 Overlay files in Gitlab CI script-based builds
 ----------------------------------------------
 
-When using Gitlab CI script-based builds (see :ref:`Gitlab-CI`) overlay files and directories have to be placed in in the repository below ``overlay/system`` in a directory tree mirroring the structure of the root filesystem. Note that any overlay file placed in that directory tree will overwrite an existing file on the root filesystem. While Ubuntu Touch 16.04 only allows overlaying files by overwriting them, in case of Ubuntu Touch 20.04 or later versions the mount-based overlay files should be used instead. Thus overlay files and directories should be placed in a directory tree below ``overlay/system/opt/halium-overlay`` in the repository rather than directly below ``overlay/system``. 
+When using Gitlab CI script-based builds (see :ref:`Gitlab-CI`) overlay files and directories have to be placed in in the repository below ``overlay/system`` in a directory tree mirroring the structure of the root filesystem.
+
+By default, the build system will cause the file to be overwriten directly in the filesystem. This allows arbitrary files to be added to the rootfs, but can cause problem when the overlaid file gets updated in the base system as part of the delta upgrade. Ubuntu Touch 16.04 supports only this mode.
+
+While Ubuntu Touch 16.04 only allows overlaying files by overwriting them, in case of Ubuntu Touch 20.04 or later versions the mount-based overlay files should be used instead. This can be done by specifying ``deviceinfo_use_overlaystore="true"`` in the port's ``deviceinfo``. If the port is not using the shared building script, it'll have to transition to the shared build script too (see :ref:`UpdatePortsFor2004`). When the option is being set, the build script ensures that the file ends up inside ``/opt/halium-overlay`` in the system, and gets mounted in the same way as ``system.img`` builds. Thus, the same config and limitations apply.
 
 Example
 ^^^^^^^
 
-Following the above example on Ubuntu Touch 20.04, adding the overlay files ``/lib/udev/rules.d/70-android.rules`` and ``/etc/ubuntu-touch-session.d/android.conf`` requires them to end up below ``/opt/halium-overlay`` on the resulting filesystem, that is as ``/opt/halium-overlay/lib/udev/rules.d/70-android.rules`` and ``/opt/halium-overlay/etc/ubuntu-touch-session.d/android.conf``. This is achieved by placing them at ``overlay/system/opt/halium-overlay/lib/udev/rules.d/70-android.rules`` and ``overlay/system/opt/halium-overlay/etc/ubuntu-touch-session.d/android.conf`` in the repository.
+Following the above example, overlaying files ``/lib/udev/rules.d/70-android.rules`` and ``/etc/ubuntu-touch-session.d/android.conf`` requires placing the file at ``overlay/system/lib/udev/rules.d/70-android.rules`` and ``overlay/system/etc/ubuntu-touch-session.d/android.conf`` in the repository.
+
+For ports with ``deviceinfo_use_overlaystore="true"`` running Ubuntu Touch 20.04, the files will end up in ``/opt/halium-overlay/``, and will be bind-mounted into their respective places.
 
-In case of Ubuntu Touch 16.04 the files on the root filesystem need to be overwritten due to the lack of mount-based overlay files and must thus be placed at ``overlay/system/lib/udev/rules.d/70-android.rules`` and ``overlay/system/etc/ubuntu-touch-session.d/android.conf`` in the repository.
+In case of Ubuntu Touch 16.04 or port without ``deviceinfo_use_overlaystore="true"``, the files on the root filesystem will be overwritten due to the lack of mount-based overlay files.
 
 Building on GitLab CI
 ^^^^^^^^^^^^^^^^^^^^^

porting/configure_test_fix/USBModed.rst

@@ -0,0 +1,90 @@
+.. _USBModed:
+
+Configuring usb-moded
+=====================
+
+.. note::
+    Information on this page is applicable for ports supporting Ubuntu Touch 20.04. To make your port support Ubuntu Touch 20.04, see :ref:`UpdatePortsFor2004`.
+
+usb-moded is a new daemon in Ubuntu Touch 20.04 which handles transitions between USB modes e.g. MTP and developer mode. It replaces a number of ad-hoc Upstart jobs on 16.04, and ``setupusb`` script used by recent ports.
+
+The system can detects most of the important settings. But for usb-moded to work best, we encouraged porters to give some configurations to ensure best experience for porters.
+
+History of the USB mode handling
+--------------------------------
+
+In Ubuntu Touch 16.04, we used Android properties to store and set the phone's USB configuration. This worked well in the past, but today it doesn't work anymore:
+
+- From Halium 7.1, the component in the Android side which sends signal to Upstart to drive state transition is not going in. Besides, the signal concept doesn't seem to exist in Systemd.
+- From Android 8, USB mode configuration code moved into a vendor-controlled HAL process which talks over HIDL.
+
+This has led to porters resorting to an ad-hoc script like ``setupusb`` which runs at boot to configure the USB mode once. And when a basic things like this gets pushed to porters, inevitably it leads to fragmentation.
+
+On Ubuntu Touch 20.04, we transition to usb-moded which is a daemon originating from SailfishOS. The daemon doesn't rely on Android properties, but requires configuration. We've created a "configurator" which auto-detect many of those settings, but it cannot detect everything. So, some configuration from the porters is still required.
+
+The basic configurations
+------------------------
+
+The main configuration is at ``/etc/default/usb-moded.d/device-specific-config.conf`` which porters can use the overlay system to place the file. The file is in a simple ``<key>=<value>`` format. An example of it is as below:
+
+.. code-block:: text
+
+    IDVENDOR=0E8D
+    IDPRODUCT_MTP=2008
+    IDPRODUCT_MTP_ADB=201D
+    IDPRODUCT_RNDIS=2004
+    IDPRODUCT_RNDIS_ADB=2005
+    USB_MODED_ARGS=
+
+The following can be configure:
+
+USB vendor & product IDs
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following configurations are highly-recommended to be set. Without it, the system will fallback to the test code from `pid.codes project <https://pid.codes/pids/>`_.
+
+- IDVENDOR
+- IDPRODUCT_MTP
+- IDPRODUCT_MTP_ADB
+- IDPRODUCT_RNDIS
+- IDPRODUCT_RNDIS_ADB
+
+It accepts the IDs in hexadecimal without ``0x`` prefix. The IDs could be found in:
+
+- Your device's ``init.<device>.rc`` or ``init.<chipset>.rc``.
+- Your device's USB HAL source code, if it happens to ship one.
+- Your device's ``setupusb`` script, if available.
+
+Failing that, you might have to probe how your device shows up when booted on Android.
+
+.. note::
+    We're aware that some device use different vendor IDs for different modes. If your device are one of those, please upvote `this issue on GitLab <https://gitlab.com/ubports/development/core/packaging/usb-moded/-/issues/5>`_.
+
+Devices' pretty name
+^^^^^^^^^^^^^^^^^^^^
+
+This is the name that will show up when plugged into a computer. Most of the time, we can detect this from Android properties, but if the detection is not correct or you would like the device to display differently, it can be configured here.
+
+- MANUFACTURER
+- PRODUCT
+
+usb-moded's arguments
+^^^^^^^^^^^^^^^^^^^^^
+
+This config controls the launching flag of the usb-moded daemon itself. Most of the time, usb-moded should work without any flag. However, the rootfs ships a default with ``-r`` which enables the rescue mode upon boot (see below). The behavior of the rescue mode can be confusing for end users, so it's recommended that porters config this value to the blank value to disable the rescue mode.
+
+- USB_MODED_ARGS
+
+About usb-moded's rescue mode
+-----------------------------
+
+usb-moded's rescue mode is similar to hybris-usb's usb-tethering mode. It enables the phone's USB to function as a network interface, and starts a DHCP server. The phone will be available at 10.15.19.82, and an emergency SSH server will be available on port 8022. This is intended for porters to debug issues during the development, and thus it will stay in that mode until the cable is re-plugged.
+
+This behavior is helpful for the porters. However, this can be confusing to end-users when the port is shipped. So I recommended disabling this mode before making the port available to users by overriding ``USB_MODED_ARGS`` as discussed above.
+
+Another rescue feature available is the ability to force usb-moded to go to "rescue mode" or to enable ADB on every boot. This can be done by placing an empty file at ``/userdata/.force-ssh`` or ``/userdata/.force-adb`` (which translate to ``/data/.force-ssh`` or ``/data/.force-adb`` in the recovery), and the system will make sure the correct mode is enabled.
+
+Configuring usb-moded directly
+------------------------------
+
+Besides tweaking options for the configurator script, porters can also set usb-moded options directly via ``/etc/usb-moded/90-device-specific-config.ini``. However, this is intended as an escape hatch in case the automatic detection fails, and if you found yourself need to use this file, consider filing issue over usb-moded's packaging repository. See `usb-moded's docs <https://github.com/sailfishos/usb-moded/blob/master/docs/usb_moded-doc.txt>` for the syntax.

porting/configure_test_fix/index.rst

@@ -22,4 +22,5 @@ As you go along, it can be helpful to have a list of device functions to go by w
    Apparmor
    Wifi
    Sound
-   Bluetooth
+   Bluetooth
+   USBModed