Skip to content

Operating

Jump to bottom
Tej Pochiraju edited this page Jun 14, 2026 · 17 revisions

Operating

Day-2 concerns: the state store, managing the template catalog, rotating keys, shipping the audit log, scraping metrics, and health checks.

The state store (always on)

The SQLite state store is the always-on backbone: it holds the template catalog and every instance's desired-state (parameters and, when a key is set, encrypted secrets). It is not optional and there is no on/off switch — -state-db is the store's location (default /var/lib/podman-api/state.db), and the store is opened on every boot. Its directory must exist and be writable by the daemon user (see Provisioning-a-Podman-Host § 8).

Features layered on the store — migrate, evacuate, ingress (-ingress-enabled), and scheduled prune (-prune-enabled) — are still gated by their own flags, but the store underneath them is always present.

-spec-key-file is optional and gates secrets only. Set it to a 32-byte key file to persist secrets at rest (so migrate/evacuate can re-supply them). Without it the store still opens and runs key-less: the template catalog and no-secret deploys work normally, but a secret-bearing deploy is rejected with secrets require an encryption key (-spec-key-file). The key is loaded once at startup — there is no hot-reload (rotating to a different key would strand existing rows undecryptable); re-encrypting rotation is a separate, future capability.

Managing the template catalog

Templates live in the store, not on disk — there is no -templates-dir. They are managed over the /templates API; reads need scope templates:read, writes need templates:write (both distinct from instances:*). The bundled postgres and basic-web templates are seed data: loaded only when the catalog is empty on first boot, then owned by you (edits and deletes survive restarts; a populated store is never re-seeded). basic-web is a deliberate starter — clone it and point it at any container image.

Method & path Scope What it does
GET /templates templates:read list all templates (structured JSON, below)
GET /templates/{id} templates:read one template
GET /templates/{id}/render?<params> templates:read preview the rendered pod (omitted params take their defaults)
POST /templates templates:write create (id in body); 409 if the id exists
PUT /templates/{id} templates:write replace an existing template; 404 if unknown
POST /templates/{id}/clone templates:write clone to a new id (body {"new_id":"..."})
DELETE /templates/{id} templates:write delete; 404 unknown id, 409 if in use (override with ?force=true) 
TOK=...   # key with templates:*

# clone the starter, then edit the copy for your own image
curl -s -X POST -H "Authorization: Bearer $TOK" \
  https://api.example/templates/basic-web/clone -d '{"new_id":"my-app"}'
curl -s -X PUT  -H "Authorization: Bearer $TOK" \
  https://api.example/templates/my-app -d @my-app.json

# delete — blocked while instances reference it unless you force it
curl -s -X DELETE -H "Authorization: Bearer $TOK" \
  "https://api.example/templates/my-app?force=true"

Deleting a template that any instance still references returns 409 (template_in_use) unless you pass ?force=true. An unknown id is 404.

Structured templates and one-click deploy

Template parameters are typed. Each is a ParamDef — name, type (string | int | bool | select), required, label, description, default, placeholder, options (for select), and secret. A deploy (POST /instances, scope instances:write) fills any omitted parameter from its declared default — so a fully-defaulted template is one-click — and the persisted spec records the effective parameters that were applied. GET /templates/{id}/render previews with the same default-filling so a preview matches what a deploy would render.

GET /templates returns one object per template:

{
  "id": "basic-web",
  "display": {"name": "Basic web app", "description": "...", "category": "Generic"},
  "parameters": [
    {"name": "slug",  "type": "string", "required": true, "label": "Instance name"},
    {"name": "image", "type": "string", "required": true, "label": "Image",
     "description": "Container image (e.g. docker.io/library/nginx:1)"}
  ],
  "secrets": {"per_instance": [], "per_host_referenced": []},
  "volumes": [],
  "ingress": {"container": "app", "port": 8080},
  "body": "apiVersion: v1\nkind: Pod\n...",
  "origin": "seed",
  "created": "2026-06-04T...",
  "updated": "2026-06-04T..."
}

origin is seed for a bundled template or user for one you created or cloned; an edit preserves the stored origin (it can't silently flip seed to user).

Managing per-instance secrets

A template's per-instance secrets (secrets.per_instance in its meta) are supplied per deployed instance and stored encrypted in the state store (needs -spec-key-file; see § The state store). The Admin UI exposes a Manage secrets control on an instance's detail page — shown only when the instance's template declares any per-instance secrets — that lists each declared secret and lets you rotate its value.

The model is write-only, by design:

  • Current values are never shown. The form lists each secret with a set / not set badge (presence only) and a blank password field — the stored value is never sent to the browser.
  • Blank keeps the current value. Only the fields you fill are rotated; leaving a field empty leaves that secret untouched. Filling a field replaces (rotates) it, or sets a secret that was previously unset.
  • There is no "clear". Because blank means "keep", the UI cannot remove a per-instance secret. Deleting a secret means redeploying the instance without it (and only if the template no longer declares it as required). To retire a secret across a template, edit the template's per_instance list, then rotate/redeploy affected instances.
  • Rotating re-applies the instance, which restarts the pod (the new secret is materialised on play). Plan rotation like an upgrade — expect a brief restart.

Secrets only ever travel in the request body, never the URL or query string, so they don't reach access logs or browser history; rotation pages are also served Cache-Control: no-store.

Template gained a required secret after deploy? Rotation (and the image-only Upgrade) tolerate a stored spec that is missing a per-instance secret the template now requires — so you can still rotate password or bump the image on an older instance without being forced to re-supply every newly-declared secret in the same step. Use Manage secrets to set the new one when convenient.

Per-host-referenced secrets (secrets.per_host_referenced) are a separate concern — they are host-wide and managed over the secrets API, not per instance.

Rotate a bearer key without dropping log streams

Keys reload on SIGHUP, applied on the next inbound request:

$EDITOR /etc/podman-api/keys.yaml
systemctl reload podman-api        # or: kill -HUP $(pidof podman-api)

In-flight log streams are not interrupted. A bad reload (parse error or zero keys) is logged and the previous key list stays live — a fat-fingered edit can't lock you out.

Audit log

Every state-changing request (POST/PUT/DELETE) emits one JSON line with method, path, host, template, slug, status, duration, key_id, and any error. By default it goes to stdout; with -audit-log-file=/var/log/podman-api/audit.log it goes to that file (the fd is held open across rotations).

A) systemd / journald (default)

stdout is captured by journald. Cap on-disk size and extract just the audit lines:

# /etc/systemd/journald.conf.d/podman-api.conf
[Journal]
SystemMaxUse=2G
MaxFileSec=1day
journalctl -u podman-api -o cat | jq -c 'select(.method)'

B) On-disk file with logrotate

# /etc/logrotate.d/podman-api
/var/log/podman-api/audit.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    copytruncate    # binary keeps the fd open; copytruncate avoids a restart
}

Use copytruncate because the process holds the file open. If you prefer create, add a postrotate hook that restarts the service (you lose in-flight log streams).

C) External collector (Vector / Promtail / Fluent Bit)

Keep audit on stdout and let the collector tail journald. Vector sketch:

[sources.podman_api]
type = "journald"
include_units = ["podman-api.service"]

[transforms.parse]
type = "remap"
inputs = ["podman_api"]
source = '. = parse_json!(.message)'

[sinks.loki]
type = "loki"
inputs = ["parse"]
endpoint = "http://loki.internal:3100"
labels = {job = "podman-api", host = "{{ host }}", template = "{{ template }}"}

Metrics

Only exposed when -metrics-addr is set, on its own listener:

  • podman_api_requests_total{host,template,method,status} — counter
  • podman_api_request_duration_seconds{host,template,method} — histogram
  • plus standard Go/process metrics.

Labels include host/template/path, so do not expose this publicly — scrape over an SSH tunnel or bind to a VPC-internal address.

Health checks

  • GET /healthz — no auth, liveness of the daemon itself.
  • GET /hosts/{host}/healthz — hosts:read, pings the target's podman over the SSH tunnel.
  • GET /hosts/{host}/ports-in-use — hosts:read, what host ports are bound (useful before choosing a hostPort).

Draining a host

A host can be marked draining (it still allows lifecycle ops and deletes, just signals intent via /hosts). Combine with instance_count to decide when a host is safe to take out of rotation. Draining also blocks a host from being a migrate/evacuate destination (you don't want to fill a host you're emptying).

Migrating & evacuating instances

These ride on the always-on state store, so they are available by default. The daemon is a stateful controller — it holds each instance's parameters and (when -spec-key-file is set) encrypted secrets, so it can re-create an instance on another host by name, without the client re-supplying secrets. Moves are cold: stop → copy volumes → re-apply on the destination → verify → reap the source. Placement is the client's choice; the daemon executes it (there is no scheduler).

Instances that carry secrets can only be migrated/evacuated when the store was opened with -spec-key-file — without a key the daemon never persisted the secret material, so it has nothing to re-supply on the destination. Run with a key if you rely on migrate/evacuate for secret-bearing instances.

Move one instance — POST /migrate 

curl -sS -X POST https://api.example/migrate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","to_host":"node-b","template":"postgres","slug":"acme"}'
# -> 202 {"job_id":"..."}

Synchronous validation (unknown host, same source/destination, no stored spec) fails fast as 4xx. Everything else — host-port conflict, missing per-host secret, draining destination, an instance already present, a destination that won't come up — surfaces as a job failure, not the POST status. Before it reaps the source the job (a) content-verifies every copied volume against the source and (b) waits for the destination to be ready; if either fails, the move rolls back (restart the source, reap the partial destination) and the source is left intact.

Readiness gate. "Ready" means the pod and every container are Running and every container that declares a healthcheck reports healthy — so a move won't commit while an app is still warming up (DB WAL replay, cache priming). Containers without a healthcheck are gated on liveness alone, exactly as before. The wait is bounded by -migrate-verify-timeout (default 60s); if readiness isn't reached in time the move rolls back. If you already run templates with healthchecks, review this timeout before rolling out — an app whose healthcheck start_period / warm-up exceeds 60s will now time out and roll back where a liveness-only check previously committed. Raise the flag to fit your slowest instance.

Volume integrity. Each cold-copied volume is verified by re-exporting source and destination and comparing a content manifest (every file's path + size + SHA-256); a mismatch fails the move, naming the offending volume and path. This is a content check — it deliberately ignores mode/uid/gid/mtime (a cross-host export need not preserve them), so it is not a permissions/ownership guarantee. Disable it with -migrate-verify-volumes=false to skip the extra source+dest re-export on very large volumes (trading the integrity guarantee for speed).

Per-host secrets on the destination

Templates can reference per-host secrets (shared, host-scoped credentials — e.g. a registry pull token) by name. A migrate/evacuate destination must have them or the move fails (host_secret_missing). The daemon can auto-provision them so you don't seed every host by hand:

  • PUT /hosts/{host}/secrets/{name} pushes the value to that host and, when the store was opened with -spec-key-file, persists it by default (sealed at rest, keyed by host). Send "persist": false to push to the host only without storing. (Key-less, persistence has nothing to seal it with — see The state store above.)
  • On a move, a per-host secret absent on the destination but persisted for the source host is created on the destination from the stored value before the instance is applied — no manual seeding. A secret that is absent and not persisted still blocks the move (host_secret_missing); the preview lists the ones that will be provisioned under provisions.
  • Provisioning records the destination too, so multi-hop evacuations chain (A→B→C works without re-PUT-ing the secret on B).
  • On rollback, provisioned secrets are left in place — they are shared and additive, and other instances on the destination may rely on them.

Rotation caveat. Rotating with "persist": false updates the host but not the store, so a later move would provision the previously-stored (stale) value onto a destination. Keep persistence on (the default) for secrets you rely on migrate/evacuate to replicate.

Out-of-band secrets. A secret seeded directly on a host (not via PUT …/secrets) has no stored value — podman has no read-API to recover one — so it can't be auto-provisioned and will block a move until you PUT it.

Clear a whole host — POST /evacuate 

curl -sS -X POST https://api.example/evacuate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","map":{"acme":"node-b","globex":"node-c"}}'
# -> 202 {"job_id":"..."}   (the PARENT job)

map is slug → destination host and must cover exactly the instances on from_host (every instance mapped, no stray keys) — otherwise 400. The same 400 covers any bad map content: an unmapped/extra/ambiguous slug, a same-host destination, or an unknown destination host. (404 is reserved for an unknown from_host — the host being operated on.) The parent job fans out one child migrate per instance at bounded concurrency; a sibling failure does not abort the others. The parent succeeds only if every child succeeds, else it fails with per-child detail.

Tuning concurrency. migrate is heavy (stop → volume cold-copy → apply → verify), so the fan-out runs a bounded number of children at once — 2 by default. Raise the daemon-wide default with -evacuate-concurrency <n> (see Deploying), or override per call by adding "concurrency": <n> to the POST /evacuate body. Either value is clamped to [1,32]. Higher concurrency finishes a large host faster but multiplies disk/network load on the destinations.

A parent evacuate occupies one job-runner worker for its entire fan-out (its children run inside the parent handler, not as separate pool jobs), so several concurrent evacuates can tie up workers and delay plain migrate/other jobs. The pool defaults to 8 workers; raise it with -job-workers <n> (see Deploying) if you routinely run many evacuates at once. (This is headroom, not a hard guarantee — it raises the threshold rather than isolating orchestration onto a separate pool.)

Typical workflow to retire a host:

  1. PATCH/configure the host as draining so nothing new lands on it, then reload (SIGHUP). Read /hosts to pick destinations with capacity (instance_count, ports-in-use).
  2. POST /evacuate with a map placing each instance on a chosen destination.
  3. Poll the parent job; on partial failure, re-issue evacuate with a map covering only what remains on the host (already-moved instances are gone from the source, so re-including them fails the bijection check).

Watching jobs

curl -sS https://api.example/jobs/$JOB_ID -H "Authorization: Bearer $TOKEN"
curl -sS "https://api.example/jobs?parent_id=$PARENT" -H "Authorization: Bearer $TOKEN"

GET /jobs/{id} returns the job's state, progress steps, and error. GET /jobs?parent_id=<id> lists an evacuate's child migrations (also filter by state / kind). Scope jobs:read.

The list is paginated, newest-first: ?limit= defaults to 100 (max 1000) and ?before=<job_id> is a cursor — pass the previous page's last id, and stop when fewer than limit rows come back. This means an evacuate fanning out more than 100 child migrations is only fully visible by paging: GET /jobs?parent_id=<id> shows the 100 newest children; page with before to see the rest.

# walk every child of a large evacuate
before=""
while :; do
  page=$(curl -sS "https://api.example/jobs?parent_id=$PARENT&limit=100${before:+&before=$before}" \
    -H "Authorization: Bearer $TOKEN")
  [ "$(echo "$page" | jq 'length')" -eq 0 ] && break
  echo "$page" | jq -r '.[].id'
  before=$(echo "$page" | jq -r '.[-1].id')
done

Previewing an evacuate

curl -sS -X POST https://api.example/evacuate/plan \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","map":{"acme":"node-b","globex":"node-c"}}'
# -> 200 {plan}

POST /evacuate/plan (scope instances:read) is a dry-run: it resolves the map and runs the same live destination checks the real move runs, but enqueues nothing and changes nothing. It answers "would this evacuate succeed?" before you commit.

The static validation is identical to POST /evacuate, so a request that would 4xx there 4xxes here the same way (unmapped/extra/ambiguous slug, unknown destination → 400; unknown from_host → 404). Once the map resolves you get 200 with one entry per instance:

{
  "from_host": "node-a",
  "moves": [
    {"slug":"acme","template":"postgres","to_host":"node-b","ok":true,
     "issues":[],"provisions":["registry-creds"]},
    {"slug":"globex","template":"redis","to_host":"node-c","ok":false,
     "issues":[{"code":"port_conflict","message":"required host port already in use: 6379"}],
     "provisions":[]}
  ]
}

A move is ok only when issues is empty. Issue codes:

  • destination_draining — the destination host is draining; evacuate refuses it.
  • instance_exists — an instance with this name is already on the destination.
  • host_secret_missing — a required per-host secret is neither present on the destination nor persisted in the store, so it can't be auto-provisioned. A secret that is persisted is non-blocking and appears under provisions.
  • port_conflict — a host port the pod binds is already taken on the destination.
  • invalid_parameters — the stored spec's parameters/secrets no longer satisfy the template's contract (e.g. the template added a required parameter after the instance was created); the real evacuate would fail when re-applying the spec.
  • check_error — the check was inconclusive (e.g. the destination was unreachable); the preview cannot vouch for the move, treat it as not-ready.

Each move also carries a provisions array (always present, [] when empty): per-host secrets absent on the destination that the evacuate will auto-provision there from the source host's persisted value (see Per-host secrets on the destination above). Provisioning is non-blocking — a move with provisions and no issues is still ok — but it tells you, before you commit, that secret material will be written onto the new host.

A blocked plan does not stop you fixing the named problems (seed the secret, free the port, fix the spec, pick another destination) and re-previewing until every move is ok.

Scope note. Because scopes are not hierarchical, an instances:write-only key can run the destructive POST /evacuate but not this read-only preview — grant instances:read (or instances:*) to operators who should be able to dry-run.

Cancelling a job

curl -sS -X POST https://api.example/jobs/$JOB_ID/cancel \
  -H "Authorization: Bearer $TOKEN"
# -> 202 {job}   (cancellation accepted)

POST /jobs/{id}/cancel (scope instances:write, like migrate/evacuate) stops a job. A queued job is cancelled immediately and moves to the terminal state canceled. A running job is signalled to stop: cancellation is asynchronous — the handler unwinds (a migrate rolls back, leaving the source intact, exactly as in the verify-fail path) and the canceled state is recorded shortly after, so the 202 body may still show running. Poll the job to confirm.

  • Cancel the parent to stop an evacuate. A child migrate runs inside the parent handler, so it is not independently cancellable — POST /jobs/{childID}/cancel returns 409 while it runs. Cancel the parent evacuate: its in-flight children roll back and are recorded canceled too.
  • Responses. 202 accepted; 404 unknown job; 409 job_terminal when the job already finished; 409 job_not_cancelable (transient — the job was just claimed but isn't registered yet) — retry shortly.
  • Cancel a reconciling job. A migrate left reconciling by a daemon restart (see below) is cancellable too: cancel records canceled and signals any in-flight reconciliation pass to stop. This is the escape hatch when a reconcile is stuck retrying against a permanently-unreachable host — a pass that has already begun applying changes may still complete.

Auto-reconciliation after restart

If the daemon restarts while a move is in flight, boot recovery no longer just fails everything:

  • Queued jobs survive untouched and run after boot.
  • A migrate that was running (including each child of an evacuate) moves to a new non-terminal state reconciling. A background loop then inspects the real state of both hosts and drives the move to a consistent end:
    • destination healthy and its spec persisted → roll forward: the source is reaped and the job is recorded succeeded;
    • destination absent/unhealthy (or its spec not yet persisted) while the source is restorable → roll back: the source is restarted, the partial destination reaped, job failed;
    • source already gone and the destination not committable → the destination is left in place (it is the only copy) and the job is failed with a "manual cleanup required" message.
  • A host or template removed from config makes its reconcile terminal (failed, "no longer configured") rather than looping forever.
  • If a host is unreachable, the job stays reconciling and is retried every 30s until it can be resolved — nothing is force-failed for a transient outage.
  • The parent evacuate itself is recorded failed (its children reconcile individually); re-issue the evacuate to move whatever instances remain on the source.

A reconciling job is non-terminal, so retention never prunes it. Use cancel (above) to abandon one that cannot resolve.

Pruning old jobs

The jobs table is unbounded by default — terminal (succeeded/failed/canceled) rows accumulate (one parent + N children per evacuate). Set -jobs-retention <dur> (e.g. 168h) to enable a background sweep that, hourly, deletes terminal jobs older than the duration. Parent/child families stay intact: a parent is removed only once it has no surviving child, so an in-flight or recent move is never orphaned. The default 0 disables pruning (the table grows until you clean it up manually). Retention also self-heals job rows written by a pre-pagination build (their timestamps render near 1970 until the first sweep removes them).

Restart handling. A daemon restart mid-move no longer just fails in-flight jobs — interrupted migrates are driven to a consistent state automatically; see Auto-reconciliation after restart above. (Continuing a migrate from its exact interrupted phase, and re-spawning an evacuate's un-started children, remain out of scope — re-issue the evacuate to cover whatever instances are still on the host.)

Scheduled prune (host-health automation)

An opt-in, headless scheduler that keeps hosts from filling their container-storage partition by running podman prune on a safe policy. Each run is an auditable prune job (see Watching jobs), so you can see exactly what was reclaimed and when. The scheduler is off by default — nothing is auto-deleted until you turn it on with -prune-enabled. It records its runs as jobs in the always-on store.

How it triggers

A scheduler evaluates every prune-enabled host roughly once a minute and enqueues a prune when either:

  • the interval has elapsed since that host's last successful prune, or
  • the host's disk crosses a high-water threshold (whichever comes first).

The disk-usage probe behind the threshold is throttled to about once every five minutes per host, so the per-minute tick stays cheap. A host that already has a prune in flight is skipped, and a host whose last prune failed is held off for an hour before retrying rather than hammering every tick.

Flags (global defaults)

Each host may override these via a prune: block in hosts/*.yaml (below).

Flag Default Meaning
-prune-enabled false Turn the feature on.
-prune-interval <dur> 24h Routine sweep interval per host. 0 disables the interval trigger (threshold-only).
-prune-disk-threshold <pct> 85 Disk used-% that triggers an early prune before the interval is due. 0 disables the threshold trigger.
-prune-scope <list> dangling Comma-separated scopes (see below). Only dangling runs unless you opt into more.
-prune-dry-run false Remove nothing; report what would be reclaimed.

Scopes: dangling (dangling image layers — the only default), all-images (also unused tagged images — costs a re-pull on next deploy), containers (exited containers), build-cache, volumes (unused/unattached volumes). Everything except dangling is strictly opt-in.

Per-host override

id: web1
addr: user@web1
socket: /run/user/1000/podman/podman.sock
prune:
  enabled: true
  interval: 12h
  disk_threshold_pct: 70
  scope: [dangling, build-cache]
  dry_run: false

Safety

Prune leans on podman's safe-by-default semantics — only dangling/unused objects are removed, never anything in use, and nothing is force-removed. Two extra guards protect stateful workloads:

  • The scheduler will not start a prune for a host that already has a running migrate or evacuate job, and while any such move is in flight the volumes scope is dropped from that run (re-checked at run time, just before the volume prune) — so a migration's transiently detached volume can't be reaped.
  • The volumes scope skips any volume carrying the protect label podman-api.protect=true. Volumes are opt-in regardless.

Operating it

  • Start cautiously. Enable with -prune-dry-run first, or set a high -prune-disk-threshold and a long -prune-interval, and watch a few runs before letting it remove anything. A dry run reports volume-reclaimable bytes (from system df) when the volumes scope is on; image/build-cache sizes aren't available without actually pruning.
  • Inspect runs via the jobs API: GET /jobs?kind=prune (add &state=succeeded). Each job records the scopes run and bytes reclaimed.
  • Metrics: the counters podman_api_prune_runs_total{host,result} and podman_api_prune_reclaimed_bytes_total{host,scope} are exported on /metrics (see Metrics).

SIGHUP caveat. The podman client is fixed at daemon startup, so a host added via a config reload (SIGHUP) is not pruned until the daemon restarts — the scheduler logs and skips such hosts rather than enqueuing prunes that can only fail. The same limitation affects migrate/evacuate for reload-added hosts.

Registry-down behaviour on apply

Apply pulls every image before playing the pod. A pull failure aborts before any secret is written, returns 502 with the registry's message, and leaves no orphan state on the target. Use ?skip_pull=true only when the image is known to be local (CI).

Ingress + auto-TLS

Optional: with -ingress-enabled, each host runs a managed Caddy pod that terminates TLS via Let's Encrypt (HTTP-01) and reverse-proxies every instance's declared domains to its pod over a shared podman network. Apps join that network and publish no host ports — only Caddy publishes :80/:443.

Prerequisites

  • -ingress-enabled and -ingress-acme-email. (Routes are derived from the always-on desired-state store, so no extra state flag is needed.)
  • The host must allow rootless privileged-port binding — see Provisioning-a-Podman-Host § 7 (net.ipv4.ip_unprivileged_port_start=80).
  • Each domain must already resolve to the host (operator-managed DNS for v1), and the host's :80/:443 must be internet-reachable for HTTP-01.

Flags (defaults in parentheses)

Flag Purpose
-ingress-enabled (false) turn the per-host Caddy ingress on
-ingress-network (podman-api-ingress) shared network app pods join
-ingress-caddy-image (docker.io/library/caddy:2) Caddy image for the system pod
-ingress-acme-email (required) ACME account email for Let's Encrypt
-ingress-reconcile-interval (5m) periodic drift-correction sweep per host; 0 disables

How routing works

A template opts in by declaring an ingress: block in its meta (container + port). An instance becomes reachable by giving it one or more domains on apply. On every create/delete/upgrade — and on the periodic sweep — podman-api derives the host's routes from the store, renders a Caddyfile, copies it into the running Caddy pod, and caddy reloads with zero downtime. The backend is the pod name (<template>-<slug>:<port>), which resolves on the shared network; app containers expose no host ports.

An apply is rejected up front (no pod played, nothing persisted) when it carries domains but ingress is disabled, the template declares no ingress:, or a domain is already claimed by another instance on the host.

Certificates persist on the podman-api-caddy-data named volume and survive Caddy restarts with no re-issue, so you stay clear of Let's Encrypt rate limits. Include that volume in host backups.

Inspecting — the Caddy pod is podman-api-ingress-caddy and its config is /etc/caddy/Caddyfile inside the container:

podman pod inspect podman-api-ingress-caddy
podman logs podman-api-ingress-caddy-caddy        # ACME + reload activity
podman exec podman-api-ingress-caddy-caddy cat /etc/caddy/Caddyfile

Related

Appendix: Manual evacuate runbook

This procedure was exercised live during the 2026-06-14 dogfooding session (engine-2 → engine-1, 3 instances, cold-volume transport). It reproduces the full real-host path — deploy, plan, evacuate, assert placement, confirm rollback on partial failure — outside CI, on any two podman hosts.

Prerequisites

  • Two hosts (node-a, node-b) each running podman system service. Both registered in the daemon's hosts/ config directory.
  • The evacuating host must have at least one deployed, running instance.
  • State store opened with -spec-key-file if instances carry secrets.
  • -migrate-verify-volumes set to true (default) so the copy-time-manifest path is exercised.

1. Confirm source state

# Which instances are on the host-to-be-evacuated?
curl -sS https://api.example/hosts/node-a/instances \
  -H "Authorization: Bearer $TOKEN" | jq '.'

Each instance's template, slug, pod.status, and warnings (if any) are shown.

2. Pick destinations

Read host capacity to decide where each instance goes:

curl -sS https://api.example/hosts \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {id, instance_count, draining}'

Avoid drained hosts as destinations (they reject new creates). Note which instances carry domains (for ingress routes — the domain follows the instance on evacuate) so you know the DNS target may change.

3. Plan

Dry-run the evacuate before running it:

curl -sS -X POST https://api.example/evacuate/plan \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{
    "from_host": "node-a",
    "moves": [
      {"template": "postgres", "slug": "db1", "to_host": "node-b"},
      {"template": "redis",    "slug": "cache", "to_host": "node-b"},
      {"template": "web",      "slug": "app1",  "to_host": "node-b"}
    ]
  }' | jq .

The response lists each move with ok: true/false and any issues. Fix all blocked moves before proceeding (resolve port conflicts, seed missing host secrets, clear destination instances, etc.).

4. Evacuate

Run the evacuate with the same moves:

RESP=$(curl -sS -X POST https://api.example/evacuate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{
    "from_host": "node-a",
    "moves": [
      {"template": "postgres", "slug": "db1", "to_host": "node-b"},
      {"template": "redis",    "slug": "cache", "to_host": "node-b"},
      {"template": "web",      "slug": "app1",  "to_host": "node-b"}
    ]
  }')
JOB_ID=$(echo "$RESP" | jq -r '.job_id')
echo "Evacuate parent job: $JOB_ID"

The POST returns 202 with a job_id — the parent evacuate job. Children are created inside it.

5. Watch the parent

Poll until the parent reaches a terminal state:

while :; do
  JOB=$(curl -sS https://api.example/jobs/"$JOB_ID" \
    -H "Authorization: Bearer $TOKEN")
  STATE=$(echo "$JOB" | jq -r '.state')
  echo "[$(date +%H:%M:%S)] parent: $STATE"
  [ "$STATE" = succeeded ] && break
  [ "$STATE" = failed ] && echo "FAILED: $(echo "$JOB" | jq -r '.error // ""')" && break
  [ "$STATE" = canceled ] && echo "CANCELED" && break
  sleep 2
done

On failure, the error includes per-child detail (2/3 migrations failed: db1: ... ; cache: ...).

6. Inspect children

List the child migrate jobs:

curl -sS "https://api.example/jobs?parent_id=$JOB_ID" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

Each child should be succeeded or failed. A failed child means that move rolled back — the source instance is still running on node-a and the partial destination was cleaned up.

7. Assert placement

For each succeeded child, confirm the instance is running on the destination:

curl -sS https://api.example/hosts/node-b/instances/postgres/db1 \
  -H "Authorization: Bearer $TOKEN" | jq '.pod.status'
# Should show "Running"

For each failed/rolled-back child, confirm it is still on the source:

curl -sS https://api.example/hosts/node-a/instances/postgres/db1 \
  -H "Authorization: Bearer $TOKEN" | jq '.pod.status'
# Should show "Running" (source intact)

Source instances of succeeded children should now return 404.

8. Confirm ingress

If any moved instance carried domains, verify traffic reaches the new host (e.g. curl -sS https://app1.example.com or check the ingress logs).

9. Retry on partial failure

After diagnosing and fixing the root cause (e.g. a missing host secret, a port conflict, a destination that was unreachable), re-issue the evacuate covering only the remaining instances on node-a:

curl -sS -X POST https://api.example/evacuate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{
    "from_host": "node-a",
    "moves": [
      {"template": "web", "slug": "app1", "to_host": "node-b"}
    ]
  }'

The bijection check ensures every move names an instance still on the source; already-moved instances are excluded automatically because their specs were deleted during commit.

10. Verify rollback on forced failure

To manually exercise the rollback path, run a single-instance migrate to a host where the instance already exists:

# Seed a blocker on the destination
curl -sS -X PUT https://api.example/hosts/node-b/instances/web/staller \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"template":"web","slug":"staller","parameters":{"slug":"staller","image":"docker.io/library/alpine:latest"}}'

# Attempt to move the same instance to node-b — destination preflight detects
# existing pod and fails before any mutation.
curl -sS -X POST https://api.example/migrate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","to_host":"node-b","template":"web","slug":"staller"}'
# → 202, then the job fails with "instance already exists"

Confirm the source pod is still Running and its spec still exists on node-a. This validates preflight-only failure (no mutation). For copy-failure or verify-failure rollback, introduce a network partition mid-copy (e.g. firewall iptables -A OUTPUT -d <dest-ip> -j DROP between two SSH-addressed hosts) and observe the source restart and partial destination get reaped.

Clone this wiki locally