Skip to content

Operating

Jump to bottom
tej edited this page Jun 2, 2026 · 17 revisions

Operating

Day-2 concerns: rotating keys, shipping the audit log, scraping metrics, and health checks.

Rotate a bearer key without dropping log streams

Keys reload on SIGHUP, applied on the next inbound request:

$EDITOR /etc/podman-api/keys.yaml
systemctl reload podman-api        # or: kill -HUP $(pidof podman-api)

In-flight log streams are not interrupted. A bad reload (parse error or zero keys) is logged and the previous key list stays live — a fat-fingered edit can't lock you out.

Audit log

Every state-changing request (POST/PUT/DELETE) emits one JSON line with method, path, host, template, slug, status, duration, key_id, and any error. By default it goes to stdout; with -audit-log-file=/var/log/podman-api/audit.log it goes to that file (the fd is held open across rotations).

A) systemd / journald (default)

stdout is captured by journald. Cap on-disk size and extract just the audit lines:

# /etc/systemd/journald.conf.d/podman-api.conf
[Journal]
SystemMaxUse=2G
MaxFileSec=1day
journalctl -u podman-api -o cat | jq -c 'select(.method)'

B) On-disk file with logrotate

# /etc/logrotate.d/podman-api
/var/log/podman-api/audit.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    copytruncate    # binary keeps the fd open; copytruncate avoids a restart
}

Use copytruncate because the process holds the file open. If you prefer create, add a postrotate hook that restarts the service (you lose in-flight log streams).

C) External collector (Vector / Promtail / Fluent Bit)

Keep audit on stdout and let the collector tail journald. Vector sketch:

[sources.podman_api]
type = "journald"
include_units = ["podman-api.service"]

[transforms.parse]
type = "remap"
inputs = ["podman_api"]
source = '. = parse_json!(.message)'

[sinks.loki]
type = "loki"
inputs = ["parse"]
endpoint = "http://loki.internal:3100"
labels = {job = "podman-api", host = "{{ host }}", template = "{{ template }}"}

Metrics

Only exposed when -metrics-addr is set, on its own listener:

  • podman_api_requests_total{host,template,method,status} — counter
  • podman_api_request_duration_seconds{host,template,method} — histogram
  • plus standard Go/process metrics.

Labels include host/template/path, so do not expose this publicly — scrape over an SSH tunnel or bind to a VPC-internal address.

Health checks

  • GET /healthz — no auth, liveness of the daemon itself.
  • GET /hosts/{host}/healthz — hosts:read, pings the target's podman over the SSH tunnel.
  • GET /hosts/{host}/ports-in-use — hosts:read, what host ports are bound (useful before choosing a hostPort).

Draining a host

A host can be marked draining (it still allows lifecycle ops and deletes, just signals intent via /hosts). Combine with instance_count to decide when a host is safe to take out of rotation.

Registry-down behaviour on apply

Apply pulls every image before playing the pod. A pull failure aborts before any secret is written, returns 502 with the registry's message, and leaves no orphan state on the target. Use ?skip_pull=true only when the image is known to be local (CI).

Related

Clone this wiki locally