Skip to content

Operating

Jump to bottom
Tej Pochiraju edited this page Jun 4, 2026 · 17 revisions

Operating

Day-2 concerns: rotating keys, shipping the audit log, scraping metrics, and health checks.

Rotate a bearer key without dropping log streams

Keys reload on SIGHUP, applied on the next inbound request:

$EDITOR /etc/podman-api/keys.yaml
systemctl reload podman-api        # or: kill -HUP $(pidof podman-api)

In-flight log streams are not interrupted. A bad reload (parse error or zero keys) is logged and the previous key list stays live — a fat-fingered edit can't lock you out.

Audit log

Every state-changing request (POST/PUT/DELETE) emits one JSON line with method, path, host, template, slug, status, duration, key_id, and any error. By default it goes to stdout; with -audit-log-file=/var/log/podman-api/audit.log it goes to that file (the fd is held open across rotations).

A) systemd / journald (default)

stdout is captured by journald. Cap on-disk size and extract just the audit lines:

# /etc/systemd/journald.conf.d/podman-api.conf
[Journal]
SystemMaxUse=2G
MaxFileSec=1day
journalctl -u podman-api -o cat | jq -c 'select(.method)'

B) On-disk file with logrotate

# /etc/logrotate.d/podman-api
/var/log/podman-api/audit.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    copytruncate    # binary keeps the fd open; copytruncate avoids a restart
}

Use copytruncate because the process holds the file open. If you prefer create, add a postrotate hook that restarts the service (you lose in-flight log streams).

C) External collector (Vector / Promtail / Fluent Bit)

Keep audit on stdout and let the collector tail journald. Vector sketch:

[sources.podman_api]
type = "journald"
include_units = ["podman-api.service"]

[transforms.parse]
type = "remap"
inputs = ["podman_api"]
source = '. = parse_json!(.message)'

[sinks.loki]
type = "loki"
inputs = ["parse"]
endpoint = "http://loki.internal:3100"
labels = {job = "podman-api", host = "{{ host }}", template = "{{ template }}"}

Metrics

Only exposed when -metrics-addr is set, on its own listener:

  • podman_api_requests_total{host,template,method,status} — counter
  • podman_api_request_duration_seconds{host,template,method} — histogram
  • plus standard Go/process metrics.

Labels include host/template/path, so do not expose this publicly — scrape over an SSH tunnel or bind to a VPC-internal address.

Health checks

  • GET /healthz — no auth, liveness of the daemon itself.
  • GET /hosts/{host}/healthz — hosts:read, pings the target's podman over the SSH tunnel.
  • GET /hosts/{host}/ports-in-use — hosts:read, what host ports are bound (useful before choosing a hostPort).

Draining a host

A host can be marked draining (it still allows lifecycle ops and deletes, just signals intent via /hosts). Combine with instance_count to decide when a host is safe to take out of rotation. Draining also blocks a host from being a migrate/evacuate destination (you don't want to fill a host you're emptying).

Migrating & evacuating instances

These require the desired-state store (-state-db + -spec-key-file; see Deploying). Without it both endpoints return 501. With it the daemon is a stateful controller — it holds each instance's parameters and encrypted secrets, so it can re-create an instance on another host by name, without the client re-supplying secrets. Moves are cold: stop → copy volumes → re-apply on the destination → verify → reap the source. Placement is the client's choice; the daemon executes it (there is no scheduler).

Move one instance — POST /migrate 

curl -sS -X POST https://api.example/migrate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","to_host":"node-b","template":"postgres","slug":"acme"}'
# -> 202 {"job_id":"..."}

Synchronous validation (unknown host, same source/destination, no stored spec) fails fast as 4xx. Everything else — host-port conflict, missing per-host secret, draining destination, an instance already present, a destination that won't come up — surfaces as a job failure, not the POST status. Before it reaps the source the job (a) content-verifies every copied volume against the source and (b) waits for the destination to be ready; if either fails, the move rolls back (restart the source, reap the partial destination) and the source is left intact.

Readiness gate. "Ready" means the pod and every container are Running and every container that declares a healthcheck reports healthy — so a move won't commit while an app is still warming up (DB WAL replay, cache priming). Containers without a healthcheck are gated on liveness alone, exactly as before. The wait is bounded by -migrate-verify-timeout (default 60s); if readiness isn't reached in time the move rolls back. If you already run templates with healthchecks, review this timeout before rolling out — an app whose healthcheck start_period / warm-up exceeds 60s will now time out and roll back where a liveness-only check previously committed. Raise the flag to fit your slowest instance.

Volume integrity. Each cold-copied volume is verified by re-exporting source and destination and comparing a content manifest (every file's path + size + SHA-256); a mismatch fails the move, naming the offending volume and path. This is a content check — it deliberately ignores mode/uid/gid/mtime (a cross-host export need not preserve them), so it is not a permissions/ownership guarantee. Disable it with -migrate-verify-volumes=false to skip the extra source+dest re-export on very large volumes (trading the integrity guarantee for speed).

Clear a whole host — POST /evacuate 

curl -sS -X POST https://api.example/evacuate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","map":{"acme":"node-b","globex":"node-c"}}'
# -> 202 {"job_id":"..."}   (the PARENT job)

map is slug → destination host and must cover exactly the instances on from_host (every instance mapped, no stray keys) — otherwise 400. The same 400 covers any bad map content: an unmapped/extra/ambiguous slug, a same-host destination, or an unknown destination host. (404 is reserved for an unknown from_host — the host being operated on.) The parent job fans out one child migrate per instance at bounded concurrency; a sibling failure does not abort the others. The parent succeeds only if every child succeeds, else it fails with per-child detail.

Tuning concurrency. migrate is heavy (stop → volume cold-copy → apply → verify), so the fan-out runs a bounded number of children at once — 2 by default. Raise the daemon-wide default with -evacuate-concurrency <n> (see Deploying), or override per call by adding "concurrency": <n> to the POST /evacuate body. Either value is clamped to [1,32]. Higher concurrency finishes a large host faster but multiplies disk/network load on the destinations.

A parent evacuate occupies one job-runner worker for its entire fan-out (its children run inside the parent handler, not as separate pool jobs), so several concurrent evacuates can tie up workers and delay plain migrate/other jobs. The pool defaults to 8 workers; raise it with -job-workers <n> (see Deploying) if you routinely run many evacuates at once. (This is headroom, not a hard guarantee — it raises the threshold rather than isolating orchestration onto a separate pool.)

Typical workflow to retire a host:

  1. PATCH/configure the host as draining so nothing new lands on it, then reload (SIGHUP). Read /hosts to pick destinations with capacity (instance_count, ports-in-use).
  2. POST /evacuate with a map placing each instance on a chosen destination.
  3. Poll the parent job; on partial failure, re-issue evacuate with a map covering only what remains on the host (already-moved instances are gone from the source, so re-including them fails the bijection check).

Watching jobs

curl -sS https://api.example/jobs/$JOB_ID -H "Authorization: Bearer $TOKEN"
curl -sS "https://api.example/jobs?parent_id=$PARENT" -H "Authorization: Bearer $TOKEN"

GET /jobs/{id} returns the job's state, progress steps, and error. GET /jobs?parent_id=<id> lists an evacuate's child migrations (also filter by state / kind). Scope jobs:read.

The list is paginated, newest-first: ?limit= defaults to 100 (max 1000) and ?before=<job_id> is a cursor — pass the previous page's last id, and stop when fewer than limit rows come back. This means an evacuate fanning out more than 100 child migrations is only fully visible by paging: GET /jobs?parent_id=<id> shows the 100 newest children; page with before to see the rest.

# walk every child of a large evacuate
before=""
while :; do
  page=$(curl -sS "https://api.example/jobs?parent_id=$PARENT&limit=100${before:+&before=$before}" \
    -H "Authorization: Bearer $TOKEN")
  [ "$(echo "$page" | jq 'length')" -eq 0 ] && break
  echo "$page" | jq -r '.[].id'
  before=$(echo "$page" | jq -r '.[-1].id')
done

Cancelling a job

curl -sS -X POST https://api.example/jobs/$JOB_ID/cancel \
  -H "Authorization: Bearer $TOKEN"
# -> 202 {job}   (cancellation accepted)

POST /jobs/{id}/cancel (scope instances:write, like migrate/evacuate) stops a job. A queued job is cancelled immediately and moves to the terminal state canceled. A running job is signalled to stop: cancellation is asynchronous — the handler unwinds (a migrate rolls back, leaving the source intact, exactly as in the verify-fail path) and the canceled state is recorded shortly after, so the 202 body may still show running. Poll the job to confirm.

  • Cancel the parent to stop an evacuate. A child migrate runs inside the parent handler, so it is not independently cancellable — POST /jobs/{childID}/cancel returns 409 while it runs. Cancel the parent evacuate: its in-flight children roll back and are recorded canceled too.
  • Responses. 202 accepted; 404 unknown job; 409 job_terminal when the job already finished; 409 job_not_cancelable (transient — the job was just claimed but isn't registered yet) — retry shortly; 501 when the state store is disabled.
  • Restart caveat. canceled is a terminal state and survives like any other. But a job interrupted by a daemon restart (not an explicit cancel) is still marked failed on boot, not canceled.

Pruning old jobs

The jobs table is unbounded by default — terminal (succeeded/failed/canceled) rows accumulate (one parent + N children per evacuate). Set -jobs-retention <dur> (e.g. 168h) to enable a background sweep that, hourly, deletes terminal jobs older than the duration. Parent/child families stay intact: a parent is removed only once it has no surviving child, so an in-flight or recent move is never orphaned. The default 0 disables pruning (the table grows until you clean it up manually). Retention also self-heals job rows written by a pre-pagination build (their timestamps render near 1970 until the first sweep removes them).

No auto-resume across restarts. If the daemon restarts mid-move, in-flight jobs (parent and children) are marked failed on boot; an in-flight migrate rolls back during graceful shutdown. Re-issue the operation — the strict bijection means a re-issued evacuate naturally covers only the instances still on the host.

Registry-down behaviour on apply

Apply pulls every image before playing the pod. A pull failure aborts before any secret is written, returns 502 with the registry's message, and leaves no orphan state on the target. Use ?skip_pull=true only when the image is known to be local (CI).

Related

Clone this wiki locally