Skip to content

Monitoring and Observability

Jump to bottom
Frank Yglesias Bertheau edited this page Jul 7, 2026 · 1 revision
<h1>ᚾ&nbsp;NornGate — Monitoring & Observability</h1>

Observability here is not one system — it is the three Norns, three time horizons: Verðandi watches the present (live metrics), Urðr keeps the past (the ledger), Skuld reads the future (forecasting). Wire all three. And wire them to act — the Norns tend the tree; they do not merely watch it.

Metric names and config are illustrative of the pipeline; reconcile against your telemetry stack.

ᚨ Before you start

Two principles govern everything below:

  • Telemetry never blocks inference. Metric and audit writes are fire-and-forget — a synchronous write on the hot path would turn observability into a latency tax and couple the request to the telemetry backend's uptime. Cross-cutting concerns fire async.
  • Watch versus feed back. A dashboard that only displays is decoration. The signal that autoscales Muspelheim, pages on budget burn, or tightens a policy is the daily watering — that is the point of the exercise.

ᛋ Verðandi — the present

Live metrics from the sidecars (Istio Telemetry v2) and the gate services, over OpenTelemetry to Prometheus/Grafana.

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata: { name: gate-metrics, namespace: asgard }
spec:
  metrics:
    - providers: [{ name: prometheus }]

Gate services tag every metric with gate, realm, and decision. The signals that matter, per gate and per realm: decision rate, deny rate, timeout rate, gate latency (p50/p95/p99), approval-budget consumption, sandbox warm-pool utilization, and req/s against the ~50K/cluster ceiling (Performance & Scalability).

Alert on SLO burn, not single breaches — p95 automated path ~150ms, per-gate ~12ms, sidecar <5ms at 99.99%.

# G1 deny-rate spike, by realm — illustrative
sum(rate(gate_decisions_total{gate="policy",decision="deny"}[5m])) by (realm)

Why deny/timeout rates are the early warning: a spike in G1 denies is either an attack or a bad policy deploy; a spike in G0 503 timeouts is a policy-store outage failing closed. The two mean different things and page different people — separate them.

ᛈ Urðr — the past

The ledger is the source of truth for forensics, compliance, and billing — not a metrics store, the exact record of every decision. Where Verðandi gives you the rate of denials, Urðr gives you which ones and why, reproducibly.

# forensics: every policy denial in the last hour
norngate-cli audit query --gate policy --decision deny --from now-1h

billing basis: committed billable outcomes per tenant this period


norngate-cli audit query --decision commit --tenant acme --from 2026-07-01 --output json

Why you need both Norns: aggregate metrics tell you something is wrong; the ledger tells you exactly what, tied to a requestId you can audit replay. Note that reads of the ledger are themselves audited — there is no unobserved forensic query.

ᛞ Skuld — the future

Forecasting and anomaly detection over the metric and ledger streams:

  • Budget exhaustion — at the current burn, the approval or deliberation budget runs out at time T; alert before, not after.
  • Capacity — req/s trend against the ~50K ceiling; scale before saturation, not during it.
  • Anomaly — a deviation in deny-rate or turns-to-resolution flags a probable attack or a regression from a deploy.

Why forecast at all: Verðandi tells you it is happening; Skuld tells you it is about to. You scale Muspelheim before the GPU queue backs up, and you catch a credential-stuffing pattern as a deny-rate anomaly while it is still a graph, not an incident.

ᛒ Close the loop

Wire the signals to actions — then prove the wiring.

Feedback actions

  • Autoscale — KEDA scales Muspelheim on GPU queue depth (event-driven, not CPU-thresholded).
  • Page — alert on approval-budget burn or SLO burn.
  • Tighten — an anomaly can trip a circuit or narrow a policy pending review.

Verify — an observability stack you have not tested failing is itself an unmonitored dependency:

  • Confirm metrics are flowing to the dashboard.
  • Force one denial and confirm it appears in both Verðandi (the deny counter increments) and Urðr (a 403 record with a requestId) — the two views must agree.
  • Force an SLO breach and confirm the alert fires and any wired feedback action triggers.

Why this last step matters: the whole premise is that the runtime feeds back on itself. If the feedback path is broken, you have decoration, not observability — and you will not know until the outage you built it to catch.

ᚱ Next steps

Iconography

Section glyphs are Elder Futhark runes (Unicode Runic block, U+16A0–U+16FF) — semantic, not emoji. Full set on Home:

Rune Name Gloss Marks
ᚾ Nauðiz need, constraint the platform mark
ᚨ Ansuz the word, instruction before you start
ᛋ Sowilō the sun, the present Verðandi — metrics
ᛈ Perthro the well of Urd Urðr — the ledger
ᛞ Dagaz dawn, the coming day Skuld — forecasting
ᛒ Berkanan growth, the tending close the loop
ᚱ Raidō the ride, the road next steps

References. OpenTelemetry; Istio Telemetry v2 (per The Spine); Prometheus / Grafana; KEDA (event-driven autoscaling); Google SRE — the golden signals and SLO / error-budget model; fire-and-forget cross-cutting — Kaman. Naming doctrine: Prose Edda / Poetic Edda, per Norse Cosmology & Platform Design.

ᚾ NornGate — Monitoring & Observability

Observability here is not one system — it is the three Norns, three time horizons: Verðandi watches the present (live metrics), Urðr keeps the past (the ledger), Skuld reads the future (forecasting). Wire all three. And wire them to act — the Norns tend the tree; they do not merely watch it.

Metric names and config are illustrative of the pipeline; reconcile against your telemetry stack.

ᚨ Before you start

Two principles govern everything below:

  • Telemetry never blocks inference. Metric and audit writes are fire-and-forget — a synchronous write on the hot path would turn observability into a latency tax and couple the request to the telemetry backend's uptime. Cross-cutting concerns fire async.
  • Watch versus feed back. A dashboard that only displays is decoration. The signal that autoscales Muspelheim, pages on budget burn, or tightens a policy is the daily watering — that is the point of the exercise.

ᛋ Verðandi — the present

Live metrics from the sidecars (Istio Telemetry v2) and the gate services, over OpenTelemetry to Prometheus/Grafana.

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata: { name: gate-metrics, namespace: asgard }
spec:
  metrics:
    - providers: [{ name: prometheus }]

Gate services tag every metric with gate, realm, and decision. The signals that matter, per gate and per realm: decision rate, deny rate, timeout rate, gate latency (p50/p95/p99), approval-budget consumption, sandbox warm-pool utilization, and req/s against the ~50K/cluster ceiling ([Performance & Scalability](Performance-and-Scalability)).

Alert on SLO burn, not single breaches — p95 automated path ~150ms, per-gate ~12ms, sidecar <5ms at 99.99%.

# G1 deny-rate spike, by realm — illustrative
sum(rate(gate_decisions_total{gate="policy",decision="deny"}[5m])) by (realm)

Why deny/timeout rates are the early warning: a spike in G1 denies is either an attack or a bad policy deploy; a spike in G0 503 timeouts is a policy-store outage failing closed. The two mean different things and page different people — separate them.

ᛈ Urðr — the past

The [ledger](Urd-Ledger) is the source of truth for forensics, compliance, and billing — not a metrics store, the exact record of every decision. Where Verðandi gives you the rate of denials, Urðr gives you which ones and why, reproducibly.

# forensics: every policy denial in the last hour
norngate-cli audit query --gate policy --decision deny --from now-1h

# billing basis: committed billable outcomes per tenant this period
norngate-cli audit query --decision commit --tenant acme --from 2026-07-01 --output json

Why you need both Norns: aggregate metrics tell you something is wrong; the ledger tells you exactly what, tied to a requestId you can audit replay. Note that reads of the ledger are themselves audited — there is no unobserved forensic query.

ᛞ Skuld — the future

Forecasting and anomaly detection over the metric and ledger streams:

  • Budget exhaustion — at the current burn, the approval or deliberation budget runs out at time T; alert before, not after.
  • Capacity — req/s trend against the ~50K ceiling; scale before saturation, not during it.
  • Anomaly — a deviation in deny-rate or turns-to-resolution flags a probable attack or a regression from a deploy.

Why forecast at all: Verðandi tells you it is happening; Skuld tells you it is about to. You scale Muspelheim before the GPU queue backs up, and you catch a credential-stuffing pattern as a deny-rate anomaly while it is still a graph, not an incident.

ᛒ Close the loop

Wire the signals to actions — then prove the wiring.

Feedback actions

  • Autoscale — KEDA scales Muspelheim on GPU queue depth (event-driven, not CPU-thresholded).
  • Page — alert on approval-budget burn or SLO burn.
  • Tighten — an anomaly can trip a circuit or narrow a policy pending review.

Verify — an observability stack you have not tested failing is itself an unmonitored dependency:

  • Confirm metrics are flowing to the dashboard.
  • Force one denial and confirm it appears in both Verðandi (the deny counter increments) and Urðr (a 403 record with a requestId) — the two views must agree.
  • Force an SLO breach and confirm the alert fires and any wired feedback action triggers.

Why this last step matters: the whole premise is that the runtime feeds back on itself. If the feedback path is broken, you have decoration, not observability — and you will not know until the outage you built it to catch.

ᚱ Next steps

Iconography

Section glyphs are Elder Futhark runes (Unicode Runic block, U+16A0–U+16FF) — semantic, not emoji. Full set on [Home](Home#iconography):

Rune Name Gloss Marks
ᚾ Nauðiz need, constraint the platform mark
ᚨ Ansuz the word, instruction before you start
ᛋ Sowilō the sun, the present Verðandi — metrics
ᛈ Perthro the well of Urd Urðr — the ledger
ᛞ Dagaz dawn, the coming day Skuld — forecasting
ᛒ Berkanan growth, the tending close the loop
ᚱ Raidō the ride, the road next steps

References. [OpenTelemetry](https://opentelemetry.io/); Istio Telemetry v2 (per [The Spine](The-Spine)); Prometheus / Grafana; [KEDA](https://keda.sh/) (event-driven autoscaling); Google SRE — the golden signals and SLO / error-budget model; fire-and-forget cross-cutting — Kaman. Naming doctrine: Prose Edda / Poetic Edda, per [Norse Cosmology & Platform Design](Norse-Cosmology-and-Platform-Design).

Welcome

Getting Started

Core Concepts

Guides

Reference

Silence means no.

Clone this wiki locally