Skip to content

Operating

Jump to bottom
Tej Pochiraju edited this page Jun 4, 2026 · 17 revisions

Operating

Day-2 concerns: rotating keys, shipping the audit log, scraping metrics, and health checks.

Rotate a bearer key without dropping log streams

Keys reload on SIGHUP, applied on the next inbound request:

$EDITOR /etc/podman-api/keys.yaml
systemctl reload podman-api        # or: kill -HUP $(pidof podman-api)

In-flight log streams are not interrupted. A bad reload (parse error or zero keys) is logged and the previous key list stays live — a fat-fingered edit can't lock you out.

Audit log

Every state-changing request (POST/PUT/DELETE) emits one JSON line with method, path, host, template, slug, status, duration, key_id, and any error. By default it goes to stdout; with -audit-log-file=/var/log/podman-api/audit.log it goes to that file (the fd is held open across rotations).

A) systemd / journald (default)

stdout is captured by journald. Cap on-disk size and extract just the audit lines:

# /etc/systemd/journald.conf.d/podman-api.conf
[Journal]
SystemMaxUse=2G
MaxFileSec=1day
journalctl -u podman-api -o cat | jq -c 'select(.method)'

B) On-disk file with logrotate

# /etc/logrotate.d/podman-api
/var/log/podman-api/audit.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    copytruncate    # binary keeps the fd open; copytruncate avoids a restart
}

Use copytruncate because the process holds the file open. If you prefer create, add a postrotate hook that restarts the service (you lose in-flight log streams).

C) External collector (Vector / Promtail / Fluent Bit)

Keep audit on stdout and let the collector tail journald. Vector sketch:

[sources.podman_api]
type = "journald"
include_units = ["podman-api.service"]

[transforms.parse]
type = "remap"
inputs = ["podman_api"]
source = '. = parse_json!(.message)'

[sinks.loki]
type = "loki"
inputs = ["parse"]
endpoint = "http://loki.internal:3100"
labels = {job = "podman-api", host = "{{ host }}", template = "{{ template }}"}

Metrics

Only exposed when -metrics-addr is set, on its own listener:

  • podman_api_requests_total{host,template,method,status} — counter
  • podman_api_request_duration_seconds{host,template,method} — histogram
  • plus standard Go/process metrics.

Labels include host/template/path, so do not expose this publicly — scrape over an SSH tunnel or bind to a VPC-internal address.

Health checks

  • GET /healthz — no auth, liveness of the daemon itself.
  • GET /hosts/{host}/healthz — hosts:read, pings the target's podman over the SSH tunnel.
  • GET /hosts/{host}/ports-in-use — hosts:read, what host ports are bound (useful before choosing a hostPort).

Draining a host

A host can be marked draining (it still allows lifecycle ops and deletes, just signals intent via /hosts). Combine with instance_count to decide when a host is safe to take out of rotation. Draining also blocks a host from being a migrate/evacuate destination (you don't want to fill a host you're emptying).

Migrating & evacuating instances

These require the desired-state store (-state-db + -spec-key-file; see Deploying). Without it both endpoints return 501. With it the daemon is a stateful controller — it holds each instance's parameters and encrypted secrets, so it can re-create an instance on another host by name, without the client re-supplying secrets. Moves are cold: stop → copy volumes → re-apply on the destination → verify → reap the source. Placement is the client's choice; the daemon executes it (there is no scheduler).

Move one instance — POST /migrate 

curl -sS -X POST https://api.example/migrate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","to_host":"node-b","template":"postgres","slug":"acme"}'
# -> 202 {"job_id":"..."}

Synchronous validation (unknown host, same source/destination, no stored spec) fails fast as 4xx. Everything else — host-port conflict, missing per-host secret, draining destination, an instance already present, a destination that won't come up — surfaces as a job failure, not the POST status. Before it reaps the source the job (a) content-verifies every copied volume against the source and (b) waits for the destination to be ready; if either fails, the move rolls back (restart the source, reap the partial destination) and the source is left intact.

Readiness gate. "Ready" means the pod and every container are Running and every container that declares a healthcheck reports healthy — so a move won't commit while an app is still warming up (DB WAL replay, cache priming). Containers without a healthcheck are gated on liveness alone, exactly as before. The wait is bounded by -migrate-verify-timeout (default 60s); if readiness isn't reached in time the move rolls back. If you already run templates with healthchecks, review this timeout before rolling out — an app whose healthcheck start_period / warm-up exceeds 60s will now time out and roll back where a liveness-only check previously committed. Raise the flag to fit your slowest instance.

Volume integrity. Each cold-copied volume is verified by re-exporting source and destination and comparing a content manifest (every file's path + size + SHA-256); a mismatch fails the move, naming the offending volume and path. This is a content check — it deliberately ignores mode/uid/gid/mtime (a cross-host export need not preserve them), so it is not a permissions/ownership guarantee. Disable it with -migrate-verify-volumes=false to skip the extra source+dest re-export on very large volumes (trading the integrity guarantee for speed).

Clear a whole host — POST /evacuate 

curl -sS -X POST https://api.example/evacuate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","map":{"acme":"node-b","globex":"node-c"}}'
# -> 202 {"job_id":"..."}   (the PARENT job)

map is slug → destination host and must cover exactly the instances on from_host (every instance mapped, no stray keys) — otherwise 400. The same 400 covers any bad map content: an unmapped/extra/ambiguous slug, a same-host destination, or an unknown destination host. (404 is reserved for an unknown from_host — the host being operated on.) The parent job fans out one child migrate per instance at bounded concurrency; a sibling failure does not abort the others. The parent succeeds only if every child succeeds, else it fails with per-child detail.

Tuning concurrency. migrate is heavy (stop → volume cold-copy → apply → verify), so the fan-out runs a bounded number of children at once — 2 by default. Raise the daemon-wide default with -evacuate-concurrency <n> (see Deploying), or override per call by adding "concurrency": <n> to the POST /evacuate body. Either value is clamped to [1,32]. Higher concurrency finishes a large host faster but multiplies disk/network load on the destinations.

Typical workflow to retire a host:

  1. PATCH/configure the host as draining so nothing new lands on it, then reload (SIGHUP). Read /hosts to pick destinations with capacity (instance_count, ports-in-use).
  2. POST /evacuate with a map placing each instance on a chosen destination.
  3. Poll the parent job; on partial failure, re-issue evacuate with a map covering only what remains on the host (already-moved instances are gone from the source, so re-including them fails the bijection check).

Watching jobs

curl -sS https://api.example/jobs/$JOB_ID -H "Authorization: Bearer $TOKEN"
curl -sS "https://api.example/jobs?parent_id=$PARENT" -H "Authorization: Bearer $TOKEN"

GET /jobs/{id} returns the job's state, progress steps, and error. GET /jobs?parent_id=<id> lists an evacuate's child migrations (also filter by state / kind). Scope jobs:read.

The list is paginated, newest-first: ?limit= defaults to 100 (max 1000) and ?before=<job_id> is a cursor — pass the previous page's last id, and stop when fewer than limit rows come back. This means an evacuate fanning out more than 100 child migrations is only fully visible by paging: GET /jobs?parent_id=<id> shows the 100 newest children; page with before to see the rest.

# walk every child of a large evacuate
before=""
while :; do
  page=$(curl -sS "https://api.example/jobs?parent_id=$PARENT&limit=100${before:+&before=$before}" \
    -H "Authorization: Bearer $TOKEN")
  [ "$(echo "$page" | jq 'length')" -eq 0 ] && break
  echo "$page" | jq -r '.[].id'
  before=$(echo "$page" | jq -r '.[-1].id')
done

Pruning old jobs

The jobs table is unbounded by default — terminal (succeeded/failed) rows accumulate (one parent + N children per evacuate). Set -jobs-retention <dur> (e.g. 168h) to enable a background sweep that, hourly, deletes terminal jobs older than the duration. Parent/child families stay intact: a parent is removed only once it has no surviving child, so an in-flight or recent move is never orphaned. The default 0 disables pruning (the table grows until you clean it up manually). Retention also self-heals job rows written by a pre-pagination build (their timestamps render near 1970 until the first sweep removes them).

No auto-resume across restarts. If the daemon restarts mid-move, in-flight jobs (parent and children) are marked failed on boot; an in-flight migrate rolls back during graceful shutdown. Re-issue the operation — the strict bijection means a re-issued evacuate naturally covers only the instances still on the host.

Registry-down behaviour on apply

Apply pulls every image before playing the pod. A pull failure aborts before any secret is written, returns 502 with the registry's message, and leaves no orphan state on the target. Use ?skip_pull=true only when the image is known to be local (CI).

Related

Clone this wiki locally