Running LMS as a secure, rootless podman container, managed by systemd

[ipilcher]

I've been working on running LMS as a rootless podman container, managed by systemd, with as many hardening options as possible turned on. I thought that I'd share what I've come up with.

Obviously, some of this is specific to my system, but it should be readily adaptable. Hopefully this is of some use to someone.

[Unit]
Description=Lyrion Music Server - podman container
Wants=network-online.target
BindsTo=user@992.service
After=network-online.target network-online.target user@992.service
PropagatesStopTo=user@992.service

[Service]
User=podman-lms
Group=podman-lms
Environment=PODMAN_SYSTEMD_UNIT=%n
Environment=XDG_RUNTIME_DIR=/run/user/992
Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/992/bus
KillMode=mixed
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStartPre=/usr/bin/podman rm -f %N
ExecStartPre=/usr/bin/podman system migrate
ExecStart=/usr/bin/podman run \
        --name %N \
        --replace \
        --detach \
        --rm \
        --network host \
        --sdnotify conmon \
        --uidmap 0:@500100 \
        --uidmap 42:@500101 \
        --uidmap 499:@500102 \
        --uidmap 65534:@500103 \
        --gidmap 0:@500100 \
        --gidmap 4:@500101 \
        --gidmap 5:@500102 \
        --gidmap 8:@500103 \
        --gidmap 42:@500104 \
        --gidmap 43:@500105 \
        --gidmap 50:@500106 \
        --gidmap 499:@500107 \
        --gidmap 65534:@500108 \
        --volume /opt/mediaserver/lms/config:/config:rw,Z \
        --volume /opt/mediaserver/lms/playlist:/playlist:rw,Z \
        --volume /srv/media/mp3:/music:ro \
        --tmpfs /tmp:rw,size=64k,mode=1777 \
        --label io.containers.autoupdate=registry \
        --env TZ=America/Chicago \
        --env PUID=499 \
        --env PGID=499 \
        --health-interval 5m \
        --security-opt no-new-privileges \
        docker.io/lmscommunity/lyrionmusicserver:stable
ExecStopPost=/usr/bin/runcon -t container_runtime_t \
        /usr/bin/rm /dev/shm/libpod_rootless_lock_992
#
#   Hardening
#
RemoveIPC=yes
RestrictRealtime=yes
LockPersonality=yes
ProtectClock=yes
ProtectKernelModules=yes
PrivateDevices=yes
BindPaths=/dev/fuse
DeviceAllow=/dev/fuse rw
PrivateTmp=yes
MemoryDenyWriteExecute=yes
SystemCallFilter=@system-service @mount @resources sethostname
CapabilityBoundingSet=CAP_SETUID CAP_SETGID
AmbientCapabilities=CAP_SETUID CAP_SETGID
ProtectSystem=strict
ProtectHome=tmpfs
BindPaths=/run/user/992
BindPaths=/opt/mediaserver/lms/.config
BindPaths=/opt/mediaserver/lms/.local
BindPaths=/opt/mediaserver/lms/config
BindPaths=/opt/mediaserver/lms/playlist
BindReadOnlyPaths=/srv/media/mp3
SystemCallArchitectures=native
UMask=0077
ProtectProc=invisible
RestrictNamespaces=user mnt pid uts ipc cgroup time net
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictSUIDSGID=yes

[Install]
WantedBy=multi-user.target

Notes

• The cool way to run a systemd-managed podman container is to use a Quadlet. However, Quadlets do not support setting User= or Group= in the [Service] section. Instead, one is expected to run the Quadlet as a user service, managed by a systemd user instance. This works OK for normal users, but it would be an administrative nightmare when the user is a service account with their shell set to /bin/nologin.

  Thus, I just use a "normal" service that runs podman.

• Despite this, rootless podman requires that the user session exist. Thus, the references to user@992.service in the [Unit] section. (992 is the UID of the podman-lms user.)

• ExecStartPre=/usr/bin/podman system migrate causes podman to re-read /etc/subuid and /etc/subgid. I use very specific mapped UIDs and GIDs (rather than just blindly assigning 65537 of them to each service account), so if/when I need to map an additional UID/GID, I need podman to pick up the expanded ranges from those files.

• The --volume options should be relatively self-explanatory. Note that I am using priviate SELinux contexts (Z) for the /config and /playlist directories, and I've told podman to not touch the the SELinux context of the /music directory. On my host, /srv/media/mp3 is labeled as public_content_t, and I've set the container_read_public_content SELinux boolean. (See Allow containers to access shared public content containers/container-selinux#422.)

• --tmpfs /tmp:rw,size=64k,mode=1777 is required, because I use fuse-overlayfs. (More on that below.) /lms/gdresize.pl contains code to remove this socket (if it didn't get cleaned up properly) before it creates it, but it seems that fuse-overlayfs doesn't handle this correctly, so gdresize.pl fails with Unable to open socket: Address already in use at /lms/gdresized.pl line 109.

  Putting /tmp on tmpfs solves the issue.

• The ExecStopPost=... line is needed, because systemd (running as init_t doesn't have permission to remove container_runtime_tmpfs_t files, so it will fail to clean up podman's lock file.

• RestrictSUIDSGID=yes was a real challenge to get working. It prevents the systemd service from creating files with the SUID or SGID bits set. The Debian base image, upon which the LMS image is based, contains a number of files with these bits set. With the systemd option set, podman can't even pull the LMS image. (Even if the LMS image were to remove these bits, the layered nature of container images means that podman still couldn't pull it.)

  I solve this by using the force_mask and mount_program options in my storage configuration (~/.config/containers/storage.conf) to use fuse-overlayfs rather than the kernel's built-in overlay support.

  [storage]
  driver = "overlay"
  
  [storage.options.overlay]
  force_mask = "shared"
  mount_program = "/usr/bin/fuse-overlayfs"

  When this option is set, the metadata of files within image layers are not set directly in the host filesystem. Instead, default ownership and permissions are used, and the metadata from the image layer is stored in the extended attributes of the file. For example, the /usr/bin/chage file within the LMS container looks like this.

  # ls -l chage
  -rwxr-sr-x. 1 root shadow 80376 Dec 14 08:00 chage
  

  But from the host, it looks like this.

  # ls -l chage
  -rwxr-xr-x. 1 podman-lms podman-lms 80376 Dec 14 08:00 chage
  
  # getfattr -d chage
  # file: chage
  user.containers.override_stat="1:5:2755:file"
  

  The kernel overlay file system does not have the capability of using extended attributes to store this metadata. Thus the user-space (FUSE) overlay filesystem is a requirement for the force_mask storage option.