Scenario
st2 already gives each managed Linux task its own transient systemd scope. That is a strong lifecycle boundary: an agent survives a supervisor or transport cgroup cascade. It is not currently a resource boundary; task scopes have unlimited CPU and memory and share the host with system services, builds, and CI.
On a shared agent host, many individually valid agents can therefore exhaust memory or CPU in aggregate. The desired invariant is host survivability: preserve an explicit reserve for non-agent work and slow or withhold agent work before the host enters swap or load collapse.
The general use case is broader than one host:
- host policy owns a fixed aggregate envelope for all managed agents;
- an Agent Spec can state numeric memory request and limit values for the whole agent, including its primary task and sidecars;
- admission guarantees the request while allowing bursts toward the limit inside the aggregate envelope;
- a declared supervisor can request a bounded temporary increase for a child, with target, reason, and expiry;
- st2 grants, partially grants, queues, or refuses the request against host-owned capacity;
- a successful temporary adjustment can optionally be promoted into the durable baseline.
CPU reservations, preemption, and automatic tuning can remain out of the initial design. Memory alone would address the sharpest safety failure while keeping the first contract small.
Concrete cgroup evidence
Controlled systemd 260 experiments established the relevant mechanics:
- a parent slice enforced a 128 MiB aggregate envelope across child scopes;
- MemoryHigh and MemoryMax on a live child scope could be raised without restarting it;
- a child exceeding a 64 MiB hard limit was OOM-killed while a sibling remained healthy;
- lowering a live hard limit from 96 MiB to 32 MiB while about 80 MiB was resident immediately OOM-killed the task;
- lowering only MemoryHigh kept the task alive and generated pressure events, but its allocator retained about 58 MiB even after releasing the application allocation.
A temporary lease therefore cannot always be revoked safely by simply restoring MemoryMax. A plausible initial expiry state is reclaiming: restore the baseline MemoryHigh, keep charging the granted capacity, retain the temporary hard ceiling while usage remains above baseline, and ask the supervisor to renew, persist, stop, or restart the agent.
Minimal first end-to-end slice
Before settling temporary lease ownership, we can establish aggregate host containment through a smaller orthogonal change:
- Host policy places the st2 supervisor service in a dedicated bounded slice.
- Every Linux task keeps its own transient sibling scope but inherits the slice containing the invoking st2 supervisor.
- Host policy sets the aggregate MemoryHigh, MemoryMax, CPUWeight, and TasksMax values; Agent Spec does not gain host-specific slice names or cgroup properties.
- Existing transport-cascade survival remains mandatory: moving task scopes into the caller slice must not nest them inside the supervisor service cgroup.
The smallest candidate mechanism is adding systemd-run --slice-inherit when st2 creates a task scope. This introduces no Agent Spec, catalog, environment, or st2 CLI setting. A manually invoked st2 keeps its caller slice; a service managed in a dedicated slice carries that host policy to its task scopes. The option requires systemd 246 or newer, so the supported-version/fallback contract needs to be explicit.
This first slice establishes only an aggregate host safety boundary. It deliberately does not claim per-agent requests, admission, supervisor leases, expiry, or persistence. Those remain the broader design below.
The proof should use real systemd units to show that a task scope is a sibling of the supervisor service within the same configured slice, is charged to that slice aggregate, and still survives a cascade kill of the supervisor service cgroup.
Open ownership question
@myobie, which model fits the intended Agent Spec/catalog architecture?
A. Separate durable st2 runtime lease state
Agent Spec contains the durable baseline. st2 stores the granted lease, requester, reason, expiry, target generation, and reclaiming state in runtime/control state and re-adopts it after restart.
This keeps transient grants out of declarative intent, but creates another durable state model that must compose with catalog snapshots, ownership receipts, and reconciliation.
B. Agent Spec-backed temporary adjustment
The safe Agent Spec authoring path writes the effective temporary request/limit plus explicit lease metadata, then restores the prior baseline on expiry. Promotion simply removes the temporary marker and keeps the values.
This is attractive because the effective desired state stays visible through one existing source of truth, catalog locking and validation already apply, and restart recovery can re-read the lease. The concerns are declaration churn, concurrent writers, digest/restart semantics, clock/expiry fencing, rollback after a crash, source-controlled catalogs, and declarations whose authoring authority is external or read-only.
Johannes currently leans toward B if temporary intent can be represented without pretending it is permanent configuration, but we are not sure whether that conflicts with the declaration-versus-runtime boundary st2 is converging on.
Questions to settle
- Is a resource lease desired state, observed runtime state, or a separate granted-policy record?
- If it belongs in Agent Spec, should the file store the baseline plus an overlay, rather than overwriting and later reconstructing the baseline?
- How should externally authored or read-only declarations receive temporary grants?
- Which lease fields affect the launch fingerprint? A limit adjustment should not restart a healthy task merely because the declaration digest changed.
- What generation or Agent Run evidence fences a late expiry or supervisor request from affecting a replacement task?
- Should numeric resource fields be part of canonical Agent Spec, while host envelopes and grant policy remain st2/provider policy?
Related: #166, #173, #177, and #150.
Posted on behalf of @schickling
| field |
value |
agent_name |
🔄 co2-tide |
agent_session_id |
b11aaa26-d085-4ef3-adca-7a64735035c9 |
agent_tool |
Codex CLI |
agent_tool_version |
0.144.1 |
agent_runtime |
Codex CLI 0.144.1 |
agent_model |
unknown |
runtime_profile |
/nix/store/9pwivx3z8bh69fjrsziw62ggmipmi35p-coding-agent-runtime-profile/share/coding-agents/profile.json |
skills_manifest |
/nix/store/y9zyjnc2d5mkgm8p7pv5xd10prcha6z0-agent-skills-corpus/share/agent-skills/manifest.json |
worktree |
schickling |
machine |
mbp2025 |
tooling_profile |
dotfiles@unknown-dirty |
Scenario
st2 already gives each managed Linux task its own transient systemd scope. That is a strong lifecycle boundary: an agent survives a supervisor or transport cgroup cascade. It is not currently a resource boundary; task scopes have unlimited CPU and memory and share the host with system services, builds, and CI.
On a shared agent host, many individually valid agents can therefore exhaust memory or CPU in aggregate. The desired invariant is host survivability: preserve an explicit reserve for non-agent work and slow or withhold agent work before the host enters swap or load collapse.
The general use case is broader than one host:
CPU reservations, preemption, and automatic tuning can remain out of the initial design. Memory alone would address the sharpest safety failure while keeping the first contract small.
Concrete cgroup evidence
Controlled systemd 260 experiments established the relevant mechanics:
A temporary lease therefore cannot always be revoked safely by simply restoring MemoryMax. A plausible initial expiry state is reclaiming: restore the baseline MemoryHigh, keep charging the granted capacity, retain the temporary hard ceiling while usage remains above baseline, and ask the supervisor to renew, persist, stop, or restart the agent.
Minimal first end-to-end slice
Before settling temporary lease ownership, we can establish aggregate host containment through a smaller orthogonal change:
The smallest candidate mechanism is adding systemd-run --slice-inherit when st2 creates a task scope. This introduces no Agent Spec, catalog, environment, or st2 CLI setting. A manually invoked st2 keeps its caller slice; a service managed in a dedicated slice carries that host policy to its task scopes. The option requires systemd 246 or newer, so the supported-version/fallback contract needs to be explicit.
This first slice establishes only an aggregate host safety boundary. It deliberately does not claim per-agent requests, admission, supervisor leases, expiry, or persistence. Those remain the broader design below.
The proof should use real systemd units to show that a task scope is a sibling of the supervisor service within the same configured slice, is charged to that slice aggregate, and still survives a cascade kill of the supervisor service cgroup.
Open ownership question
@myobie, which model fits the intended Agent Spec/catalog architecture?
A. Separate durable st2 runtime lease state
Agent Spec contains the durable baseline. st2 stores the granted lease, requester, reason, expiry, target generation, and reclaiming state in runtime/control state and re-adopts it after restart.
This keeps transient grants out of declarative intent, but creates another durable state model that must compose with catalog snapshots, ownership receipts, and reconciliation.
B. Agent Spec-backed temporary adjustment
The safe Agent Spec authoring path writes the effective temporary request/limit plus explicit lease metadata, then restores the prior baseline on expiry. Promotion simply removes the temporary marker and keeps the values.
This is attractive because the effective desired state stays visible through one existing source of truth, catalog locking and validation already apply, and restart recovery can re-read the lease. The concerns are declaration churn, concurrent writers, digest/restart semantics, clock/expiry fencing, rollback after a crash, source-controlled catalogs, and declarations whose authoring authority is external or read-only.
Johannes currently leans toward B if temporary intent can be represented without pretending it is permanent configuration, but we are not sure whether that conflicts with the declaration-versus-runtime boundary st2 is converging on.
Questions to settle
Related: #166, #173, #177, and #150.
Posted on behalf of @schickling
agent_nameagent_session_idagent_toolagent_tool_versionagent_runtimeagent_modelruntime_profileskills_manifestworktreemachinetooling_profile