Skip to content

Operating

Jump to bottom
Tej Pochiraju edited this page Jun 4, 2026 · 17 revisions

Operating

Day-2 concerns: rotating keys, shipping the audit log, scraping metrics, and health checks.

Rotate a bearer key without dropping log streams

Keys reload on SIGHUP, applied on the next inbound request:

$EDITOR /etc/podman-api/keys.yaml
systemctl reload podman-api        # or: kill -HUP $(pidof podman-api)

In-flight log streams are not interrupted. A bad reload (parse error or zero keys) is logged and the previous key list stays live — a fat-fingered edit can't lock you out.

Audit log

Every state-changing request (POST/PUT/DELETE) emits one JSON line with method, path, host, template, slug, status, duration, key_id, and any error. By default it goes to stdout; with -audit-log-file=/var/log/podman-api/audit.log it goes to that file (the fd is held open across rotations).

A) systemd / journald (default)

stdout is captured by journald. Cap on-disk size and extract just the audit lines:

# /etc/systemd/journald.conf.d/podman-api.conf
[Journal]
SystemMaxUse=2G
MaxFileSec=1day
journalctl -u podman-api -o cat | jq -c 'select(.method)'

B) On-disk file with logrotate

# /etc/logrotate.d/podman-api
/var/log/podman-api/audit.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    copytruncate    # binary keeps the fd open; copytruncate avoids a restart
}

Use copytruncate because the process holds the file open. If you prefer create, add a postrotate hook that restarts the service (you lose in-flight log streams).

C) External collector (Vector / Promtail / Fluent Bit)

Keep audit on stdout and let the collector tail journald. Vector sketch:

[sources.podman_api]
type = "journald"
include_units = ["podman-api.service"]

[transforms.parse]
type = "remap"
inputs = ["podman_api"]
source = '. = parse_json!(.message)'

[sinks.loki]
type = "loki"
inputs = ["parse"]
endpoint = "http://loki.internal:3100"
labels = {job = "podman-api", host = "{{ host }}", template = "{{ template }}"}

Metrics

Only exposed when -metrics-addr is set, on its own listener:

  • podman_api_requests_total{host,template,method,status} — counter
  • podman_api_request_duration_seconds{host,template,method} — histogram
  • plus standard Go/process metrics.

Labels include host/template/path, so do not expose this publicly — scrape over an SSH tunnel or bind to a VPC-internal address.

Health checks

  • GET /healthz — no auth, liveness of the daemon itself.
  • GET /hosts/{host}/healthz — hosts:read, pings the target's podman over the SSH tunnel.
  • GET /hosts/{host}/ports-in-use — hosts:read, what host ports are bound (useful before choosing a hostPort).

Draining a host

A host can be marked draining (it still allows lifecycle ops and deletes, just signals intent via /hosts). Combine with instance_count to decide when a host is safe to take out of rotation. Draining also blocks a host from being a migrate/evacuate destination (you don't want to fill a host you're emptying).

Migrating & evacuating instances

These require the desired-state store (-state-db + -spec-key-file; see Deploying). Without it both endpoints return 501. With it the daemon is a stateful controller — it holds each instance's parameters and encrypted secrets, so it can re-create an instance on another host by name, without the client re-supplying secrets. Moves are cold: stop → copy volumes → re-apply on the destination → verify → reap the source. Placement is the client's choice; the daemon executes it (there is no scheduler).

Move one instance — POST /migrate 

curl -sS -X POST https://api.example/migrate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","to_host":"node-b","template":"postgres","slug":"acme"}'
# -> 202 {"job_id":"..."}

Synchronous validation (unknown host, same source/destination, no stored spec) fails fast as 4xx. Everything else — host-port conflict, missing per-host secret, draining destination, an instance already present, a destination that won't come up — surfaces as a job failure, not the POST status. Before it reaps the source the job (a) content-verifies every copied volume against the source and (b) waits for the destination to be ready; if either fails, the move rolls back (restart the source, reap the partial destination) and the source is left intact.

Readiness gate. "Ready" means the pod and every container are Running and every container that declares a healthcheck reports healthy — so a move won't commit while an app is still warming up (DB WAL replay, cache priming). Containers without a healthcheck are gated on liveness alone, exactly as before. The wait is bounded by -migrate-verify-timeout (default 60s); if readiness isn't reached in time the move rolls back. If you already run templates with healthchecks, review this timeout before rolling out — an app whose healthcheck start_period / warm-up exceeds 60s will now time out and roll back where a liveness-only check previously committed. Raise the flag to fit your slowest instance.

Volume integrity. Each cold-copied volume is verified by re-exporting source and destination and comparing a content manifest (every file's path + size + SHA-256); a mismatch fails the move, naming the offending volume and path. This is a content check — it deliberately ignores mode/uid/gid/mtime (a cross-host export need not preserve them), so it is not a permissions/ownership guarantee. Disable it with -migrate-verify-volumes=false to skip the extra source+dest re-export on very large volumes (trading the integrity guarantee for speed).

Clear a whole host — POST /evacuate 

curl -sS -X POST https://api.example/evacuate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","map":{"acme":"node-b","globex":"node-c"}}'
# -> 202 {"job_id":"..."}   (the PARENT job)

map is slug → destination host and must cover exactly the instances on from_host (every instance mapped, no stray keys) — otherwise 400. The same 400 covers any bad map content: an unmapped/extra/ambiguous slug, a same-host destination, or an unknown destination host. (404 is reserved for an unknown from_host — the host being operated on.) The parent job fans out one child migrate per instance at bounded concurrency; a sibling failure does not abort the others. The parent succeeds only if every child succeeds, else it fails with per-child detail.

Tuning concurrency. migrate is heavy (stop → volume cold-copy → apply → verify), so the fan-out runs a bounded number of children at once — 2 by default. Raise the daemon-wide default with -evacuate-concurrency <n> (see Deploying), or override per call by adding "concurrency": <n> to the POST /evacuate body. Either value is clamped to [1,32]. Higher concurrency finishes a large host faster but multiplies disk/network load on the destinations.

A parent evacuate occupies one job-runner worker for its entire fan-out (its children run inside the parent handler, not as separate pool jobs), so several concurrent evacuates can tie up workers and delay plain migrate/other jobs. The pool defaults to 8 workers; raise it with -job-workers <n> (see Deploying) if you routinely run many evacuates at once. (This is headroom, not a hard guarantee — it raises the threshold rather than isolating orchestration onto a separate pool.)

Typical workflow to retire a host:

  1. PATCH/configure the host as draining so nothing new lands on it, then reload (SIGHUP). Read /hosts to pick destinations with capacity (instance_count, ports-in-use).
  2. POST /evacuate with a map placing each instance on a chosen destination.
  3. Poll the parent job; on partial failure, re-issue evacuate with a map covering only what remains on the host (already-moved instances are gone from the source, so re-including them fails the bijection check).

Watching jobs

curl -sS https://api.example/jobs/$JOB_ID -H "Authorization: Bearer $TOKEN"
curl -sS "https://api.example/jobs?parent_id=$PARENT" -H "Authorization: Bearer $TOKEN"

GET /jobs/{id} returns the job's state, progress steps, and error. GET /jobs?parent_id=<id> lists an evacuate's child migrations (also filter by state / kind). Scope jobs:read.

The list is paginated, newest-first: ?limit= defaults to 100 (max 1000) and ?before=<job_id> is a cursor — pass the previous page's last id, and stop when fewer than limit rows come back. This means an evacuate fanning out more than 100 child migrations is only fully visible by paging: GET /jobs?parent_id=<id> shows the 100 newest children; page with before to see the rest.

# walk every child of a large evacuate
before=""
while :; do
  page=$(curl -sS "https://api.example/jobs?parent_id=$PARENT&limit=100${before:+&before=$before}" \
    -H "Authorization: Bearer $TOKEN")
  [ "$(echo "$page" | jq 'length')" -eq 0 ] && break
  echo "$page" | jq -r '.[].id'
  before=$(echo "$page" | jq -r '.[-1].id')
done

Previewing an evacuate

curl -sS -X POST https://api.example/evacuate/plan \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","map":{"acme":"node-b","globex":"node-c"}}'
# -> 200 {plan}

POST /evacuate/plan (scope instances:read) is a dry-run: it resolves the map and runs the same live destination checks the real move runs, but enqueues nothing and changes nothing. It answers "would this evacuate succeed?" before you commit.

The static validation is identical to POST /evacuate, so a request that would 4xx there 4xxes here the same way (unmapped/extra/ambiguous slug, unknown destination → 400; unknown from_host → 404; no state store → 501). Once the map resolves you get 200 with one entry per instance:

{
  "from_host": "node-a",
  "moves": [
    {"slug":"acme","template":"postgres","to_host":"node-b","ok":true,"issues":[]},
    {"slug":"globex","template":"redis","to_host":"node-c","ok":false,
     "issues":[{"code":"port_conflict","message":"required host port already in use: 6379"}]}
  ]
}

A move is ok only when issues is empty. Issue codes:

  • destination_draining — the destination host is draining; evacuate refuses it.
  • instance_exists — an instance with this name is already on the destination.
  • host_secret_missing — a required per-host secret is not seeded on the destination.
  • port_conflict — a host port the pod binds is already taken on the destination.
  • invalid_parameters — the stored spec's parameters/secrets no longer satisfy the template's contract (e.g. the template added a required parameter after the instance was created); the real evacuate would fail when re-applying the spec.
  • check_error — the check was inconclusive (e.g. the destination was unreachable); the preview cannot vouch for the move, treat it as not-ready.

A blocked plan does not stop you fixing the named problems (seed the secret, free the port, fix the spec, pick another destination) and re-previewing until every move is ok.

Scope note. Because scopes are not hierarchical, an instances:write-only key can run the destructive POST /evacuate but not this read-only preview — grant instances:read (or instances:*) to operators who should be able to dry-run.

Cancelling a job

curl -sS -X POST https://api.example/jobs/$JOB_ID/cancel \
  -H "Authorization: Bearer $TOKEN"
# -> 202 {job}   (cancellation accepted)

POST /jobs/{id}/cancel (scope instances:write, like migrate/evacuate) stops a job. A queued job is cancelled immediately and moves to the terminal state canceled. A running job is signalled to stop: cancellation is asynchronous — the handler unwinds (a migrate rolls back, leaving the source intact, exactly as in the verify-fail path) and the canceled state is recorded shortly after, so the 202 body may still show running. Poll the job to confirm.

  • Cancel the parent to stop an evacuate. A child migrate runs inside the parent handler, so it is not independently cancellable — POST /jobs/{childID}/cancel returns 409 while it runs. Cancel the parent evacuate: its in-flight children roll back and are recorded canceled too.
  • Responses. 202 accepted; 404 unknown job; 409 job_terminal when the job already finished; 409 job_not_cancelable (transient — the job was just claimed but isn't registered yet) — retry shortly; 501 when the state store is disabled.
  • Restart caveat. canceled is a terminal state and survives like any other. But a job interrupted by a daemon restart (not an explicit cancel) is still marked failed on boot, not canceled.

Pruning old jobs

The jobs table is unbounded by default — terminal (succeeded/failed/canceled) rows accumulate (one parent + N children per evacuate). Set -jobs-retention <dur> (e.g. 168h) to enable a background sweep that, hourly, deletes terminal jobs older than the duration. Parent/child families stay intact: a parent is removed only once it has no surviving child, so an in-flight or recent move is never orphaned. The default 0 disables pruning (the table grows until you clean it up manually). Retention also self-heals job rows written by a pre-pagination build (their timestamps render near 1970 until the first sweep removes them).

No auto-resume across restarts. If the daemon restarts mid-move, in-flight jobs (parent and children) are marked failed on boot; an in-flight migrate rolls back during graceful shutdown. Re-issue the operation — the strict bijection means a re-issued evacuate naturally covers only the instances still on the host.

Scheduled prune (host-health automation)

An opt-in, headless scheduler that keeps hosts from filling their container-storage partition by running podman prune on a safe policy. Each run is an auditable prune job (see Watching jobs), so you can see exactly what was reclaimed and when. It is off by default — nothing is auto-deleted until you enable it — and, like migrate/evacuate, it requires -state-db; the daemon refuses to start if -prune-enabled is set without a store.

How it triggers

A scheduler evaluates every prune-enabled host roughly once a minute and enqueues a prune when either:

  • the interval has elapsed since that host's last successful prune, or
  • the host's disk crosses a high-water threshold (whichever comes first).

The disk-usage probe behind the threshold is throttled to about once every five minutes per host, so the per-minute tick stays cheap. A host that already has a prune in flight is skipped, and a host whose last prune failed is held off for an hour before retrying rather than hammering every tick.

Flags (global defaults)

Each host may override these via a prune: block in hosts/*.yaml (below).

Flag Default Meaning
-prune-enabled false Turn the feature on.
-prune-interval <dur> 24h Routine sweep interval per host. 0 disables the interval trigger (threshold-only).
-prune-disk-threshold <pct> 85 Disk used-% that triggers an early prune before the interval is due. 0 disables the threshold trigger.
-prune-scope <list> dangling Comma-separated scopes (see below). Only dangling runs unless you opt into more.
-prune-dry-run false Remove nothing; report what would be reclaimed.

Scopes: dangling (dangling image layers — the only default), all-images (also unused tagged images — costs a re-pull on next deploy), containers (exited containers), build-cache, volumes (unused/unattached volumes). Everything except dangling is strictly opt-in.

Per-host override

id: web1
addr: user@web1
socket: /run/user/1000/podman/podman.sock
prune:
  enabled: true
  interval: 12h
  disk_threshold_pct: 70
  scope: [dangling, build-cache]
  dry_run: false

Safety

Prune leans on podman's safe-by-default semantics — only dangling/unused objects are removed, never anything in use, and nothing is force-removed. Two extra guards protect stateful workloads:

  • The scheduler will not start a prune for a host that already has a running migrate or evacuate job, and while any such move is in flight the volumes scope is dropped from that run (re-checked at run time, just before the volume prune) — so a migration's transiently detached volume can't be reaped.
  • The volumes scope skips any volume carrying the protect label podman-api.protect=true. Volumes are opt-in regardless.

Operating it

  • Start cautiously. Enable with -prune-dry-run first, or set a high -prune-disk-threshold and a long -prune-interval, and watch a few runs before letting it remove anything. A dry run reports volume-reclaimable bytes (from system df) when the volumes scope is on; image/build-cache sizes aren't available without actually pruning.
  • Inspect runs via the jobs API: GET /jobs?kind=prune (add &state=succeeded). Each job records the scopes run and bytes reclaimed.
  • Metrics: the counters podman_api_prune_runs_total{host,result} and podman_api_prune_reclaimed_bytes_total{host,scope} are exported on /metrics (see Metrics).

SIGHUP caveat. The podman client is fixed at daemon startup, so a host added via a config reload (SIGHUP) is not pruned until the daemon restarts — the scheduler logs and skips such hosts rather than enqueuing prunes that can only fail. The same limitation affects migrate/evacuate for reload-added hosts.

Registry-down behaviour on apply

Apply pulls every image before playing the pod. A pull failure aborts before any secret is written, returns 502 with the registry's message, and leaves no orphan state on the target. Use ?skip_pull=true only when the image is known to be local (CI).

Related

Clone this wiki locally