First class support for systemd portable services via ignition

[keithy]

This has been discussed in #37 as a means to provide system containers.

Systemd provides the container isolation features for Portable Containers, but lacks the transport layer. Therefore it would seem useful for ignition to add the missing piece.

One option would be for portable service metadata to be defined outside of the portable data package, as a yaml file which can be included in the ignition file (as I did with goss.yaml in this walkthrough video https://youtu.be/cvWN8dXHaVo ), thus health-check.ps.yaml is processed on boot by two simple scripts

• ignition.ps.sh - (ps = post script)
• services.upgrade.sh - (updates to portable services or normal services)

This then means that there is an opportunity for ignition itself (or installer-coreos) to take on the role of that first script. Attaching a portable service is a very simple operation and could easily be performed in initramfs right up front by ignition.

example of yaml that could be processed by ignition or service/script ignition.ps

composable:
  default_repository: https://composable.io
  packages:
    - name: goss
      repository: ftp://localhost:25
      version: v0.0.1 | latest
      track_stream: stable|next|testing
      enable:
      - goss-serve.service
      start:
      - goss.socket
      - goss-serve.service
      profile: default|nonetwork|strict|trusted
      install_method: copy|symlink|auto
     

Package naming:-
<protocol>:<repository>/<arch>/<name>_><version>
file:///media/usb1/fcos/x86_64/goss_.30.20191014.1>
file:///media/usb1/fcos/x86_64/goss_v0.0.1>
file:///media/usb1/fcos/x86_64/goss_latest>
(match the current/previous FCOS version)

In the meantime I propose a systemd unit that performs the same tasks, on first boot.

[keithy]

redbaron wrote:

If you manage to integrate systemd portable services + Ignition + ostree , that would be a killer feature.

This suggests adding ostree overlay functionality to ignition would be useful and need not be "dangerous" if the overlayed item is a portable service, but as an ordinary I dont want to be making custom ostrees.

So for now I will settle for using ignition to pull an archive into: /opt, and a systemd Unit to decompress archives into /var/lib/portables

[keithy]

found: https://www.freedesktop.org/software/systemd/man/machinectl.html#pull-tar export-tar
further fact finding selinux issues mentioned: systemd/systemd#8620 (comment)
More detailed explanations on Portable Containers are found in this earlier presentation (as compared to the later presentations by Lennart) https://www.youtube.com/watch?v=sqhojVPr7xM

[keithy]

https://trello.com/b/mo7EHqbw/fcos-safe-layer

Intro

This is the simplest approach I can think of which combines portable services and user packages. The aim here is to get as much bang as possible out of a small buck (two scripts and two services) added to ignition, thus bringing us up to the next level.

1. Script coreos-install-pkg.sh
   • unpacks tarballs ( /opt/inbox -> /opt/pkgs )
   • attaches, enables, starts portable services (ignition sets the security - profile)
   • installs scripts/executables (ignition sets the security - user)
2. Service - First Boot
   • runs coreos-install-pkg.sh for each tarball
3. 2nd Service and Script pair - TBD
   • updates TBD

The next level

Is the ability to install code in a second layer, and run code in a third layer on top of the OS completely safely using built in chroot features without using containers. Each layer is provided as a chroot-ing shell, that can be directly invoked, attached to a user login, or run as portable service.

1. User "core" - layer 0 - Fedora Core OS (sudo is the genuine root)
2. User "arne" - layer 1 - Code Layer (arne is psuedo-root a chroot and can install packages)
3. User "otto" - layer 2 - Runtime Layer (otto is a normal user a chroot and can run code)

Thus a package created by "arne" can be run as an executable by "otto", or as a portable service (restricted by a suitable profile)

The tools for managing layers will be collated in a package fcos-tools, the shell is 'layer-shell.sh' invokable directly or via the tools e.g. 'core shell --help'.

Design

A single tarball/package may provide: a code package, a portable service, either/and/or provide utility scripts (install.sh) that provide a user package and functions.

In both cases the writer of the ignition file chooses the level of protection afforded the package installation, whether by portable service profile, or by restricted user.

• The archive expands to a filesystem, that is used as the chroot.
• If /usr/local/lib/systemd/system/*.{service,socket,timer...} is present then the systemd portablectl attach is used to bring these services online.
• Conventionally within the chroot aka porous-container
  • the host filesystem will be mounted at /host (mounted ro)
  • the package files are provided in /pkg (mounted/used ro)
  • the runtime layer is /this (mounted/used rw)
  • Units are part of the package tree
    • /pkg/units -> /usr/local/lib/systemd/system (symlink)
  • Scripts being part of the package
    • /pkg/scripts/attach.sh - an example of manually attaching this container (for testing)
    • /pkg/scripts/detach.sh - an example of manually detaching this container (for testing)
    • /pkg/scripts/enable.sh - for manually enabling services
    • /pkg/scripts/start.sh - for manually starting services
    • /pkg/scripts/install.sh - an install script for other arbitrary installations
• The design may be used to install:
  • Arbitrary code executables/scripts via the /pkg/scripts/install.sh
    • note: the install script is run (restricted) by the user specified in the ignition file.
  • systemd units that are restricted by a portablectl "profile"
    • note: the profile is selected by the writer of the ignition file.
  • More complex layered packages chroot(ro) <- core_os(ro) <- pkg(ro) <- this(rw)
  • Executables that run from the commandline via systemd-run
    (that use the same chrooted/layered environments)

Issues

Enabling/Starting Services is now supported.

Implementation

The Ignition File:
a) download pkg/portable archive, to a path that indicates the mapping of "profile", "user" and "name_version"

storage: 
  files:
   - path: /opt/inbox/trusted/core/goss_v0.1.tar.gz
      mode: 0644
      contents:
        source: https://github.com/keithy/portable_goss/archive/goss_v0.1--fedora31-x86_64.tar.gz

On each archive provided (via storage:files:) the service unit performs the script which

• a) extracts the archive
• b) attach portables according to the "profile"
• c) runs the install script under the selected "user"

[Unit]
 Description=Attach Portable Services & Install Packages
 ConditionFirstBoot=yes
 After=multi-user.target
 Before=boot-complete.target
[Service]
 Type=oneshot
 ExecStartPre=setenforce Permissive
 ExecStart=-find /opt/inbox -mindepth 3 -maxdepth 3 -name "*.tar.[xg]z" -exec sh /root/coreos-install-pkg.sh {} \;
ExecStartPost=systemctl daemon-reload
[Install]
 WantedBy=multi-user.target
 RequiredBy=boot-complete.target       

The script:

    - path: /root/coreos-install-pkg.sh
      mode: 0755
      user:
        id: 0
      group:
        id: 0
      contents:
        inline: |
          set -a
          PACKAGES="/opt/pkgs"
          TAR_PATH="$1"
          IFS=/ read -r a b c PROFILE USER ARCHIVE <<< "$TAR_PATH"
          PKG="${ARCHIVE%.tar.[xg]z}"
          PKG_NAME=${PKG%_*}
          mkdir -p "$PACKAGES/$PKG"
          tar xvzf "$TAR_PATH" --strip-components 1 -C $PACKAGES/$PKG && \
            ln -s "$PACKAGES/$PKG" "$PACKAGES/$PKG_NAME"
          INSTALL_SH="$PACKAGES/$PKG/pkg/scripts/install.sh"
          [[ -e "$INSTALL_SH" ]] && su -m "$USER" "$INSTALL_SH" "$PACKAGES/$PKG_NAME" || true
          METADATA="$PACKAGES/$PKG/pkg/pkg-release"
          portablectl attach --no-reload --copy=symlink "--profile=$PROFILE" "$PACKAGES/$PKG" && \
            systemctl enable $(grep "^UNITS_ENABLE=" "$METADATA" | cut -d '=' -f2) && \
            systemctl start $(grep "^UNITS_START=" "$METADATA" | cut -d '=' -f2) || true
          echo "Finished installing $PACKAGES/$PKG"        
  

Nice to have

• Executables, that look and act entirely native, but actually run in the safe layer.
  • affords opportunities to remove more things from the core os.
• Packages, that are built expecting to share libraries with the core os
  • lightweight
  • package self-updates (via pull) when the core os updates (via push)