Skip to content

Operating

Jump to bottom
Tej Pochiraju edited this page Jun 4, 2026 · 17 revisions

Operating

Day-2 concerns: rotating keys, shipping the audit log, scraping metrics, and health checks.

Rotate a bearer key without dropping log streams

Keys reload on SIGHUP, applied on the next inbound request:

$EDITOR /etc/podman-api/keys.yaml
systemctl reload podman-api        # or: kill -HUP $(pidof podman-api)

In-flight log streams are not interrupted. A bad reload (parse error or zero keys) is logged and the previous key list stays live — a fat-fingered edit can't lock you out.

Audit log

Every state-changing request (POST/PUT/DELETE) emits one JSON line with method, path, host, template, slug, status, duration, key_id, and any error. By default it goes to stdout; with -audit-log-file=/var/log/podman-api/audit.log it goes to that file (the fd is held open across rotations).

A) systemd / journald (default)

stdout is captured by journald. Cap on-disk size and extract just the audit lines:

# /etc/systemd/journald.conf.d/podman-api.conf
[Journal]
SystemMaxUse=2G
MaxFileSec=1day
journalctl -u podman-api -o cat | jq -c 'select(.method)'

B) On-disk file with logrotate

# /etc/logrotate.d/podman-api
/var/log/podman-api/audit.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    copytruncate    # binary keeps the fd open; copytruncate avoids a restart
}

Use copytruncate because the process holds the file open. If you prefer create, add a postrotate hook that restarts the service (you lose in-flight log streams).

C) External collector (Vector / Promtail / Fluent Bit)

Keep audit on stdout and let the collector tail journald. Vector sketch:

[sources.podman_api]
type = "journald"
include_units = ["podman-api.service"]

[transforms.parse]
type = "remap"
inputs = ["podman_api"]
source = '. = parse_json!(.message)'

[sinks.loki]
type = "loki"
inputs = ["parse"]
endpoint = "http://loki.internal:3100"
labels = {job = "podman-api", host = "{{ host }}", template = "{{ template }}"}

Metrics

Only exposed when -metrics-addr is set, on its own listener:

  • podman_api_requests_total{host,template,method,status} — counter
  • podman_api_request_duration_seconds{host,template,method} — histogram
  • plus standard Go/process metrics.

Labels include host/template/path, so do not expose this publicly — scrape over an SSH tunnel or bind to a VPC-internal address.

Health checks

  • GET /healthz — no auth, liveness of the daemon itself.
  • GET /hosts/{host}/healthz — hosts:read, pings the target's podman over the SSH tunnel.
  • GET /hosts/{host}/ports-in-use — hosts:read, what host ports are bound (useful before choosing a hostPort).

Draining a host

A host can be marked draining (it still allows lifecycle ops and deletes, just signals intent via /hosts). Combine with instance_count to decide when a host is safe to take out of rotation. Draining also blocks a host from being a migrate/evacuate destination (you don't want to fill a host you're emptying).

Migrating & evacuating instances

These require the desired-state store (-state-db + -spec-key-file; see Deploying). Without it both endpoints return 501. With it the daemon is a stateful controller — it holds each instance's parameters and encrypted secrets, so it can re-create an instance on another host by name, without the client re-supplying secrets. Moves are cold: stop → copy volumes → re-apply on the destination → verify → reap the source. Placement is the client's choice; the daemon executes it (there is no scheduler).

Move one instance — POST /migrate 

curl -sS -X POST https://api.example/migrate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","to_host":"node-b","template":"postgres","slug":"acme"}'
# -> 202 {"job_id":"..."}

Synchronous validation (unknown host, same source/destination, no stored spec) fails fast as 4xx. Everything else — host-port conflict, missing per-host secret, draining destination, an instance already present, a destination that won't come up — surfaces as a job failure, not the POST status. The job verifies every container is Running before it reaps the source; if it doesn't, the move rolls back (restart the source, reap the partial destination) and the source is left intact.

Clear a whole host — POST /evacuate 

curl -sS -X POST https://api.example/evacuate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","map":{"acme":"node-b","globex":"node-c"}}'
# -> 202 {"job_id":"..."}   (the PARENT job)

map is slug → destination host and must cover exactly the instances on from_host (every instance mapped, no stray keys) — otherwise 400. The same 400 covers any bad map content: an unmapped/extra/ambiguous slug, a same-host destination, or an unknown destination host. (404 is reserved for an unknown from_host — the host being operated on.) The parent job fans out one child migrate per instance at bounded concurrency; a sibling failure does not abort the others. The parent succeeds only if every child succeeds, else it fails with per-child detail.

Tuning concurrency. migrate is heavy (stop → volume cold-copy → apply → verify), so the fan-out runs a bounded number of children at once — 2 by default. Raise the daemon-wide default with -evacuate-concurrency <n> (see Deploying), or override per call by adding "concurrency": <n> to the POST /evacuate body. Either value is clamped to [1,32]. Higher concurrency finishes a large host faster but multiplies disk/network load on the destinations.

Typical workflow to retire a host:

  1. PATCH/configure the host as draining so nothing new lands on it, then reload (SIGHUP). Read /hosts to pick destinations with capacity (instance_count, ports-in-use).
  2. POST /evacuate with a map placing each instance on a chosen destination.
  3. Poll the parent job; on partial failure, re-issue evacuate with a map covering only what remains on the host (already-moved instances are gone from the source, so re-including them fails the bijection check).

Watching jobs

curl -sS https://api.example/jobs/$JOB_ID -H "Authorization: Bearer $TOKEN"
curl -sS "https://api.example/jobs?parent_id=$PARENT" -H "Authorization: Bearer $TOKEN"

GET /jobs/{id} returns the job's state, progress steps, and error. GET /jobs?parent_id=<id> lists an evacuate's child migrations (also filter by state / kind). Scope jobs:read.

The list is paginated, newest-first: ?limit= defaults to 100 (max 1000) and ?before=<job_id> is a cursor — pass the previous page's last id, and stop when fewer than limit rows come back. This means an evacuate fanning out more than 100 child migrations is only fully visible by paging: GET /jobs?parent_id=<id> shows the 100 newest children; page with before to see the rest.

# walk every child of a large evacuate
before=""
while :; do
  page=$(curl -sS "https://api.example/jobs?parent_id=$PARENT&limit=100${before:+&before=$before}" \
    -H "Authorization: Bearer $TOKEN")
  [ "$(echo "$page" | jq 'length')" -eq 0 ] && break
  echo "$page" | jq -r '.[].id'
  before=$(echo "$page" | jq -r '.[-1].id')
done

Pruning old jobs

The jobs table is unbounded by default — terminal (succeeded/failed) rows accumulate (one parent + N children per evacuate). Set -jobs-retention <dur> (e.g. 168h) to enable a background sweep that, hourly, deletes terminal jobs older than the duration. Parent/child families stay intact: a parent is removed only once it has no surviving child, so an in-flight or recent move is never orphaned. The default 0 disables pruning (the table grows until you clean it up manually). Retention also self-heals job rows written by a pre-pagination build (their timestamps render near 1970 until the first sweep removes them).

No auto-resume across restarts. If the daemon restarts mid-move, in-flight jobs (parent and children) are marked failed on boot; an in-flight migrate rolls back during graceful shutdown. Re-issue the operation — the strict bijection means a re-issued evacuate naturally covers only the instances still on the host.

Registry-down behaviour on apply

Apply pulls every image before playing the pod. A pull failure aborts before any secret is written, returns 502 with the registry's message, and leaves no orphan state on the target. Use ?skip_pull=true only when the image is known to be local (CI).

Related

Clone this wiki locally