Spec 002: systemd agent

AGENTS.md

@@ -1,8 +1,10 @@
 # core-ops Development Guidelines
 
-Auto-generated from all feature plans. Last updated: 2026-03-18
+Auto-generated from all feature plans. Last updated: 2026-03-19
 
 ## Active Technologies
+- Rust (stable toolchain) + Git (CLI), systemd (systemctl), Quadlet generator, clap, thiserror, miette, journald logger (002-systemd-agent)
+- Files on disk (Quadlet unit files + optional reconciliation state) (002-systemd-agent)
 
 - Rust (stable toolchain) + Git (CLI), systemd (systemctl), Podman/Quadlet generator (001-gitops-quadlet-controller)
 
@@ -22,6 +24,7 @@ cargo test [ONLY COMMANDS FOR ACTIVE TECHNOLOGIES][ONLY COMMANDS FOR ACTIVE TECH
 Rust (stable toolchain): Follow standard conventions
 
 ## Recent Changes
+- 002-systemd-agent: Added Rust (stable toolchain) + Git (CLI), systemd (systemctl), Quadlet generator, clap, thiserror, miette, journald logger
 
 - 001-gitops-quadlet-controller: Added Rust (stable toolchain) + Git (CLI), systemd (systemctl), Podman/Quadlet generator
 

docs/development.md

@@ -15,3 +15,35 @@ Do not assume Rust tooling is installed globally.
 - Test: `cargo test` (or `make test`)
 
 Assume the nix shell is already active, and do not run commands via `direnv exec`.
+
+## Systemd Agent Configuration
+
+The host agent is designed to run as a oneshot service triggered by a timer.
+Use a systemd drop-in to configure the repo source and revision without editing
+unit files in place. The contract units are named `core-ops.service` and
+`core-ops.timer`.
+
+```
+systemctl edit core-ops.service
+```
+
+Suggested drop-in content:
+
+```
+[Service]
+Environment=CORE_OPS_REPO=ssh://git@github.com/your-org/quadlets.git
+Environment=CORE_OPS_REV=main
+Environment=CORE_OPS_QUADLET_DIR=/etc/containers/systemd
+Environment=CORE_OPS_SYSTEMD_UNIT_DIR=/etc/systemd/system
+```
+
+Apply changes with:
+
+- `systemctl daemon-reload`
+- `systemctl restart core-ops.service`
+
+Timer enablement example:
+
+```
+systemctl enable --now core-ops.timer
+```

specs/002-systemd-agent/checklists/requirements.md

@@ -0,0 +1,34 @@
+# Specification Quality Checklist: Systemd-Managed Host Agent
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-03-19
+**Feature**: specs/002-systemd-agent/spec.md
+
+## Content Quality
+
+- [X] No implementation details (languages, frameworks, APIs)
+- [X] Focused on user value and business needs
+- [X] Written for non-technical stakeholders
+- [X] All mandatory sections completed
+
+## Requirement Completeness
+
+- [X] No [NEEDS CLARIFICATION] markers remain
+- [X] Requirements are testable and unambiguous
+- [X] Success criteria are measurable
+- [X] Success criteria are technology-agnostic (no implementation details)
+- [X] All acceptance scenarios are defined
+- [X] Edge cases are identified
+- [X] Scope is clearly bounded
+- [X] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [X] All functional requirements have clear acceptance criteria
+- [X] User scenarios cover primary flows
+- [X] Feature meets measurable outcomes defined in Success Criteria
+- [X] No implementation details leak into specification
+
+## Notes
+
+- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`

specs/002-systemd-agent/contracts/systemd-units.md

@@ -0,0 +1,19 @@
+# Contract: Systemd Service + Timer
+
+## Service Unit (oneshot)
+
+- Purpose: Execute a single reconciliation run.
+- Inputs: repo source, revision, quadlet directory, optional audit export path.
+- Output: journald audit events plus operator-facing report.
+- Exit codes: non-zero on failure; failure class logged to journald.
+
+## Timer Unit
+
+- Purpose: Trigger the oneshot service on a schedule.
+- Behavior: Must not allow overlapping runs; timer re-triggers only after the
+  previous run finishes.
+
+## CLI Invocation Contract
+
+- Service unit MUST call the CLI with explicit repo and revision.
+- Timer unit MUST reference the service unit, not the binary directly.

specs/002-systemd-agent/contracts/systemd/core-ops.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=Core Ops Quadlet GitOps Agent
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=oneshot
+# Set the repo + revision to reconcile against.
+Environment=CORE_OPS_REPO=https://example.com/org/quadlets.git
+Environment=CORE_OPS_REV=main
+Environment=CORE_OPS_QUADLET_DIR=/etc/containers/systemd
+ExecStart=/usr/bin/core-ops apply --repo ${CORE_OPS_REPO} --rev ${CORE_OPS_REV} --quadlet-dir ${CORE_OPS_QUADLET_DIR}
+
+[Install]
+WantedBy=multi-user.target

specs/002-systemd-agent/contracts/systemd/core-ops.timer

@@ -0,0 +1,11 @@
+[Unit]
+Description=Run Core Ops Quadlet GitOps Agent on a schedule
+
+[Timer]
+OnBootSec=5min
+OnUnitActiveSec=5min
+Persistent=true
+Unit=core-ops.service
+
+[Install]
+WantedBy=timers.target

specs/002-systemd-agent/data-model.md

@@ -0,0 +1,37 @@
+# Data Model: Systemd-Managed Host Agent
+
+## Entities
+
+### HostAgentRun
+- **Fields**: run_id, mode, status, started_at, finished_at, failure_class
+- **Relationships**: references a ReconciliationPlan and AuditEvent
+- **Validation**: must always record outcome (success/failure)
+
+### QuadletArtifact
+- **Fields**: name, artifact_type (container | socket | volume), unit_file_name,
+  contents, desired_state
+- **Relationships**: referenced by ReconciliationPlan actions
+- **Validation**: artifact_type must be supported; name must be unique per type
+
+### ReconciliationPlan
+- **Fields**: plan_id, desired_revision_id, observed_revision_id, ordered_actions
+- **Validation**: actions ordered Volume → Container → Socket
+
+### VerificationResult
+- **Fields**: artifact_name, artifact_type, unit_state, passed, message
+- **Validation**: unit_state derived from systemd query
+
+### AuditEvent
+- **Fields**: run_id, plan_id, action_count, summary, timestamp
+- **Validation**: emitted for every run under systemd
+
+### RunLock
+- **Fields**: lock_id, acquired_at, owner
+- **Validation**: only one active lock per host
+
+## Relationships
+
+- HostAgentRun → ReconciliationPlan (1:1)
+- HostAgentRun → AuditEvent (1:1)
+- ReconciliationPlan → QuadletArtifact (1:N)
+- QuadletArtifact → VerificationResult (1:1 per run)

specs/002-systemd-agent/plan.md

@@ -0,0 +1,115 @@
+# Implementation Plan: Systemd-Managed Host Agent
+
+**Branch**: `[002-systemd-agent]` | **Date**: 2026-03-19 | **Spec**: specs/002-systemd-agent/spec.md
+**Input**: Feature specification from `/specs/002-systemd-agent/spec.md`
+
+**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/plan-template.md` for the execution workflow.
+
+## Summary
+
+Deliver a systemd-managed host agent that runs unattended via a oneshot service
+triggered by a timer, emits journald audit events by default, and reconciles
+container/volume Quadlet artifacts alongside native systemd socket units with
+explicit ordering and verification. Generated units are not enabled/disabled by
+the controller; Quadlet [Install] semantics govern enablement. The agent
+preserves functional-core/imperative-shell boundaries, explicit failure
+reporting, idempotence, and observability while staying within native system
+primitives and avoiding generic host configuration management.
+
+## Technical Context
+
+**Language/Version**: Rust (stable toolchain)  
+**Primary Dependencies**: Git (CLI), systemd (systemctl), Quadlet generator, clap, thiserror, miette, journald logger  
+**Storage**: Files on disk (Quadlet unit files + systemd socket units + optional reconciliation state)  
+**Testing**: cargo test (unit + integration)  
+**Target Platform**: Fedora CoreOS (single host)  
+**Project Type**: CLI + systemd service/timer agent  
+**Performance Goals**: 95% of scheduled runs finish within 2 minutes for up to 50 artifacts  
+**Constraints**: No host configuration management beyond Quadlet/systemd/container scope; explicit failures; journald observability  
+**Scale/Scope**: Single host, tens of artifacts, no fleet orchestration
+
+## Declarative State Model
+
+Desired state is sourced from the Git repository’s Quadlet definitions, observed
+state is derived from the host’s systemd-managed artifacts, and reconciliation
+plans represent ordered actions required to converge. Runs persist their plan,
+actions, and outcomes as explicit data structures surfaced through audit output.
+
+## Idempotence Strategy
+
+Reconciliation is designed to be safe to repeat: applying the same desired state
+does not introduce additional changes, and repeated runs converge to the same
+observed outcome. The plan phase determines no-op results when desired and
+observed state already align, and apply actions are constructed to be stable on
+subsequent executions.
+
+## Phases
+
+1. **Setup**: Ship systemd unit templates and deployment guidance.
+2. **Foundational**: Add socket/volume types, ordering, verification model, and run lock.
+3. **User Story 1 (MVP)**: Unattended agent entrypoint, CLI wiring, journald audit, lock usage.
+4. **User Story 2**: Reconcile socket + volume artifacts end-to-end and report artifact types.
+5. **User Story 3**: Verification checks wired through reconcile and audit outputs.
+6. **Polish**: Documentation updates, performance/idempotence checks, targeted refactors.
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+- Functional core and imperative shell boundaries are explicit; side effects are isolated.
+- Desired/observed state, reconciliation plans, and outcomes are represented as data.
+- Abstractions are minimal and justified; complexity tracking added if needed.
+- Effects, assumptions, and failure modes are explicit in interfaces and returns.
+- Idempotence and convergence strategy are defined, including retry behavior.
+- Open standards and native interfaces are preferred; deviations justified.
+- Observability plan covers diffs, plans, actions, failures, and dry-run/audit needs.
+- Safe defaults are documented; destructive actions require explicit intent.
+- Compatibility impact is assessed; breaking changes are documented with migration.
+- Test strategy covers invariants, external behavior, convergence, and failures.
+- Modules are structured to be regenerable from specs and tests.
+
+Status: PASS (pre-design). Post-design re-check: PASS.
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/002-systemd-agent/
+├── plan.md              # This file (/speckit.plan command output)
+├── research.md          # Phase 0 output (/speckit.plan command)
+├── data-model.md        # Phase 1 output (/speckit.plan command)
+├── quickstart.md        # Phase 1 output (/speckit.plan command)
+├── contracts/           # Phase 1 output (/speckit.plan command)
+└── tasks.md             # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan)
+```
+
+### Source Code (repository root)
+
+```text
+src/
+├── cli/
+├── core/
+└── io/
+
+tests/
+├── integration/
+└── unit/
+```
+
+**Structure Decision**: Single project with `core` (pure planning/verification),
+`io` (Git/systemd/Quadlet side effects), and `cli` (entrypoints and reporting).
+Systemd service/timer definitions live in `specs/002-systemd-agent/contracts/`.
+
+## Compatibility Impact
+
+No breaking changes expected for existing container-only workflows; new artifact
+types and agent automation are additive.
+
+## Complexity Tracking
+
+> **Fill ONLY if Constitution Check has violations that must be justified**
+
+| Violation | Why Needed | Simpler Alternative Rejected Because |
+|-----------|------------|-------------------------------------|
+| | | |

specs/002-systemd-agent/quickstart.md

@@ -0,0 +1,60 @@
+# Quickstart: Systemd-Managed Host Agent
+
+**Goal**: Run the GitOps Quadlet controller unattended via systemd service +
+timer on a Fedora CoreOS host.
+
+## Prerequisites
+
+- Fedora CoreOS host with systemd and Quadlet support
+- Git repository containing Quadlet files (container, socket, volume)
+- Operator access to install systemd unit files
+
+## Steps
+
+1. Install the provided oneshot service and timer unit files on the host.
+   - Copy `specs/002-systemd-agent/contracts/systemd/core-ops.service` to
+     `/etc/systemd/system/core-ops.service`
+   - Copy `specs/002-systemd-agent/contracts/systemd/core-ops.timer` to
+     `/etc/systemd/system/core-ops.timer`
+2. Configure the repo and revision the agent should reconcile.
+   - Use `systemctl edit core-ops.service` and add:
+     ```
+     [Service]
+     Environment=CORE_OPS_REPO=ssh://git@github.com/your-org/quadlets.git
+     Environment=CORE_OPS_REV=main
+     Environment=CORE_OPS_QUADLET_DIR=/etc/containers/systemd
+     Environment=CORE_OPS_SYSTEMD_UNIT_DIR=/etc/systemd/system
+     ```
+3. Enable and start the timer:
+   - `systemctl daemon-reload`
+   - `systemctl enable --now core-ops.timer`
+4. Confirm journald output includes plan/action summaries per run:
+   - `journalctl -u core-ops.service -f`
+5. Update the Git repository and verify the agent converges to the new state.
+
+## What to Expect
+
+- Runs are scheduled by systemd timer and execute the oneshot service.
+- Journald contains structured audit events for each run.
+- Artifacts are reconciled in Volume → Container → Socket ordering, including
+  container, socket, and volume Quadlets.
+- Verification uses systemd unit state checks.
+
+## Environment Overrides
+
+You can override the agent configuration with environment variables on the
+service unit:
+
+- `CORE_OPS_REPO` (required)
+- `CORE_OPS_REV` (required)
+- `CORE_OPS_QUADLET_DIR` (default `/etc/containers/systemd`)
+- `CORE_OPS_SYSTEMD_UNIT_DIR` (default `/etc/systemd/system`)
+- `CORE_OPS_AUDIT_DIR` (optional)
+- `CORE_OPS_LOCK_PATH` (optional)
+
+## Non-Goals
+
+- Fleet orchestration or multi-host coordination
+- Secret distribution
+- Generic host configuration management
+- Arbitrary environment file management as first-class objects

specs/002-systemd-agent/research.md

@@ -0,0 +1,24 @@
+# Research: Systemd-Managed Host Agent
+
+## Decision: Systemd automation mode
+
+**Decision**: Ship both a oneshot service and a timer that triggers it.
+**Rationale**: Aligns with Fedora CoreOS operational norms and keeps unattended
+execution explicit and inspectable.
+**Alternatives considered**: Long-running daemon only; timer-only without
+standalone service.
+
+## Decision: Artifact ordering
+
+**Decision**: Volume → Container → Socket ordering for reconciliation.
+**Rationale**: Volumes must exist before containers reference them; sockets
+typically depend on services or containers.
+**Alternatives considered**: Socket-first ordering; container-first ordering.
+
+## Decision: Verification behavior
+
+**Decision**: Verify via systemd unit state checks (active/enabled where
+applicable) for each artifact type.
+**Rationale**: Uses native systemd primitives without custom probes and provides
+consistent, inspectable outcomes.
+**Alternatives considered**: File existence-only verification; runtime probes.