Skip to content

Operating

Jump to bottom
Tej Pochiraju edited this page Jun 4, 2026 · 17 revisions

Operating

Day-2 concerns: rotating keys, shipping the audit log, scraping metrics, and health checks.

Rotate a bearer key without dropping log streams

Keys reload on SIGHUP, applied on the next inbound request:

$EDITOR /etc/podman-api/keys.yaml
systemctl reload podman-api        # or: kill -HUP $(pidof podman-api)

In-flight log streams are not interrupted. A bad reload (parse error or zero keys) is logged and the previous key list stays live — a fat-fingered edit can't lock you out.

Audit log

Every state-changing request (POST/PUT/DELETE) emits one JSON line with method, path, host, template, slug, status, duration, key_id, and any error. By default it goes to stdout; with -audit-log-file=/var/log/podman-api/audit.log it goes to that file (the fd is held open across rotations).

A) systemd / journald (default)

stdout is captured by journald. Cap on-disk size and extract just the audit lines:

# /etc/systemd/journald.conf.d/podman-api.conf
[Journal]
SystemMaxUse=2G
MaxFileSec=1day
journalctl -u podman-api -o cat | jq -c 'select(.method)'

B) On-disk file with logrotate

# /etc/logrotate.d/podman-api
/var/log/podman-api/audit.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    copytruncate    # binary keeps the fd open; copytruncate avoids a restart
}

Use copytruncate because the process holds the file open. If you prefer create, add a postrotate hook that restarts the service (you lose in-flight log streams).

C) External collector (Vector / Promtail / Fluent Bit)

Keep audit on stdout and let the collector tail journald. Vector sketch:

[sources.podman_api]
type = "journald"
include_units = ["podman-api.service"]

[transforms.parse]
type = "remap"
inputs = ["podman_api"]
source = '. = parse_json!(.message)'

[sinks.loki]
type = "loki"
inputs = ["parse"]
endpoint = "http://loki.internal:3100"
labels = {job = "podman-api", host = "{{ host }}", template = "{{ template }}"}

Metrics

Only exposed when -metrics-addr is set, on its own listener:

  • podman_api_requests_total{host,template,method,status} — counter
  • podman_api_request_duration_seconds{host,template,method} — histogram
  • plus standard Go/process metrics.

Labels include host/template/path, so do not expose this publicly — scrape over an SSH tunnel or bind to a VPC-internal address.

Health checks

  • GET /healthz — no auth, liveness of the daemon itself.
  • GET /hosts/{host}/healthz — hosts:read, pings the target's podman over the SSH tunnel.
  • GET /hosts/{host}/ports-in-use — hosts:read, what host ports are bound (useful before choosing a hostPort).

Draining a host

A host can be marked draining (it still allows lifecycle ops and deletes, just signals intent via /hosts). Combine with instance_count to decide when a host is safe to take out of rotation. Draining also blocks a host from being a migrate/evacuate destination (you don't want to fill a host you're emptying).

Migrating & evacuating instances

These require the desired-state store (-state-db + -spec-key-file; see Deploying). Without it both endpoints return 501. With it the daemon is a stateful controller — it holds each instance's parameters and encrypted secrets, so it can re-create an instance on another host by name, without the client re-supplying secrets. Moves are cold: stop → copy volumes → re-apply on the destination → verify → reap the source. Placement is the client's choice; the daemon executes it (there is no scheduler).

Move one instance — POST /migrate 

curl -sS -X POST https://api.example/migrate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","to_host":"node-b","template":"postgres","slug":"acme"}'
# -> 202 {"job_id":"..."}

Synchronous validation (unknown host, same source/destination, no stored spec) fails fast as 4xx. Everything else — host-port conflict, missing per-host secret, draining destination, an instance already present, a destination that won't come up — surfaces as a job failure, not the POST status. The job verifies every container is Running before it reaps the source; if it doesn't, the move rolls back (restart the source, reap the partial destination) and the source is left intact.

Clear a whole host — POST /evacuate 

curl -sS -X POST https://api.example/evacuate \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"from_host":"node-a","map":{"acme":"node-b","globex":"node-c"}}'
# -> 202 {"job_id":"..."}   (the PARENT job)

map is slug → destination host and must cover exactly the instances on from_host (every instance mapped, no stray keys) — otherwise 400. The parent job fans out one child migrate per instance at bounded concurrency; a sibling failure does not abort the others. The parent succeeds only if every child succeeds, else it fails with per-child detail.

Typical workflow to retire a host:

  1. PATCH/configure the host as draining so nothing new lands on it, then reload (SIGHUP). Read /hosts to pick destinations with capacity (instance_count, ports-in-use).
  2. POST /evacuate with a map placing each instance on a chosen destination.
  3. Poll the parent job; on partial failure, re-issue evacuate with a map covering only what remains on the host (already-moved instances are gone from the source, so re-including them fails the bijection check).

Watching jobs

curl -sS https://api.example/jobs/$JOB_ID -H "Authorization: Bearer $TOKEN"
curl -sS "https://api.example/jobs?parent_id=$PARENT" -H "Authorization: Bearer $TOKEN"

GET /jobs/{id} returns the job's state, progress steps, and error. GET /jobs?parent_id=<id> lists an evacuate's child migrations (also filter by state / kind). Scope jobs:read.

No auto-resume across restarts. If the daemon restarts mid-move, in-flight jobs (parent and children) are marked failed on boot; an in-flight migrate rolls back during graceful shutdown. Re-issue the operation — the strict bijection means a re-issued evacuate naturally covers only the instances still on the host.

Registry-down behaviour on apply

Apply pulls every image before playing the pod. A pull failure aborts before any secret is written, returns 502 with the registry's message, and leaves no orphan state on the target. Use ?skip_pull=true only when the image is known to be local (CI).

Related

Clone this wiki locally