Feature: declarative host-service lifecycle for system-container workloads

[kingsleydon]

Summary

We run Incus/LXC system containers inside a DStack CVM. For our storage,
cgroup, and reboot semantics, the Incus daemon runs as a PID 1-owned host
systemd service rather than as an ordinary application container.

The integration works, but the application currently owns an imperative
host-service installer and its complete reboot, readiness, update, and rollback
protocol. We would like DStack to provide a constrained, declarative lifecycle
contract for this class of host service.

Existing DStack mechanisms

DStack already provides host execution mechanisms; this request does not claim
otherwise.

At current next commit
312dc605ebaaf93372534ffb135ea011533d28e9:

• app-compose.service
  is a host systemd oneshot that runs app-compose.sh and remains active after
  the script exits.
• app-compose.sh
  supports a pre-launch script, Docker or nerdctl Compose, and a bash runner.
• The vsock HostApi
  exposes Info, Notify, and GetSealingKey.

These are useful imperative building blocks, but the current public contract
does not describe a typed host-service declaration with persistent/boot-scoped
resources, service ownership, readiness, or rollback semantics.

Why an ordinary Compose service is insufficient for this workload

For our required deployment model, the Incus daemon must:

• own a delegated cgroup subtree for nested system containers;
• use host-native ZFS state under /dstack/persistent;
• remain a PID 1-owned service independently of the finite installer; and
• receive explicit persistent and boot-scoped binds into a content-addressed
  userspace root.

Our current integration therefore uses a finite privileged Compose job with
host PID/root access. One centralized nsenter --target 1 boundary verifies a
pinned payload, materializes a boot-scoped root, installs runtime systemd
units, binds persistent state and boot-scoped resources, starts the daemon, and
checks readiness. A mismatch or partial generation fails closed.

This is workable, but every consumer of this pattern must implement and secure
its own installer, convergence, reboot recovery, update, and rollback logic.

Requested capability

Please consider a constrained host-service declaration that can express:

• a content-addressed and authenticated payload;
• explicit persistent state and boot-scoped runtime mounts;
• bounded systemd unit properties, capabilities, cgroup delegation, and
  ordering;
• readiness and health conditions; and
• atomic update, reboot recovery, and rollback behavior.

The declaration should be authenticated and capability-scoped. We are not
requesting unrestricted host command execution, and the implementation need
not extend HostApi; a parallel mechanism would be equally suitable.

Namespace-safe host resources

One concrete edge case is resolver projection. On the released image,
/etc/resolv.conf was an absolute symlink chain ending under
/run/systemd/resolve. Reading it through a container's /host root bind
resolves the absolute target in the container mount namespace and misses the
host file. Current mkosi source similarly creates
/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf.

Our installer handles this inside the PID 1 mount namespace. A declarative
contract should instead document or provide a namespace-safe projection for
host resources consumed by the declared service. We are not requiring
resolv.conf to be a regular file.

Suggested acceptance criteria

• A declared service is restored to ready state after an unchanged CVM reboot.
• Persistent state survives reboot and application update, while boot-scoped
  resources are reconstructed.
• A failed update leaves or restores the last ready generation.
• The application does not need a host-root bind or nsenter to install and
  reconcile the declared service.

Scope

This is a feature request, not a report that DStack Compose or HostApi is
broken. Incus networking/kernel compatibility is tracked separately.