Matelot v0.1.0-rc.22

No CHANGELOG section is titled ## v0.1.0-rc.22; these are the latest changes on the branch.

v0.1.0 — first self-hosted release

Images ghcr.io/matelot-cloud/{api,web,worker,runner} and the chart oci://ghcr.io/matelot-cloud/charts/secops-cell, signed, with SBOM attestations; install guide at https://matelot-cloud.github.io/website/docs/latest/install.html. The sections below are the drops that make up the release, newest first.

Observability drop 3 acceptance fixes (kind rc.21): latest-backup rule, one alert per cell, no NaN series, Alertmanager on kind, canonical runbook URLs

• MatelotBackupJobFailed fires only when the newest backup Job (by kube_job_status_start_time, scheduled or
  -backup-now) failed: the CronJob keeps its last failed Jobs, and one failure followed by good nightly runs kept the
  old rule firing for days. promtool test: failed then succeeded → quiet; succeeded then failed → fires; a second cell
  stays quiet (rules/matelot-kubernetes_test.yaml, run with the other rule tests).
• One alert per cell: MatelotApiDown, MatelotWorkerDown, MatelotKeycloakDown are min by (cell) (up{…}) == 0
  (a rollout has two up series and produced two alerts); the other rules already aggregate by cell or are per object
  (a certificate, a PVC). promtool test with two api and two Keycloak series → one alert each.
• Dashboards: every histogram_quantile drops NaN (… >= 0), so the kinds that never ran (all fourteen exist at
  zero) draw no empty series; the Pipelines kind variable lists only the kinds with at least one finished job.
  tools/dashboards/test.py fails on a quantile without the filter.
• kind: the kube-prometheus-stack values run a minimal Alertmanager (v0.34.1, the chart's) with a matelot route
  for Matelot* alerts and the stack's Watchdog to a null receiver — PrometheusNotConnectedToAlertmanagers no longer
  fires; config and routing checked with amtool. kind-observability.sh prints its port-forward.
• Runbook and dashboard links point at the canonical docs, https://matelot.cloud/docs/latest/…, not the github.io
  mirror: one constant, DOCS_URL in tools/docs-site/build.mjs, read by tools/dashboards/build.py; the checker fails
  on any docs link in the rules, dashboards, the chart's runbookBaseUrl / web.docsUrl or the Compose default that
  is not under it.
• Stray deploy/deploy/ tree: tools/dashboards/build.py (drop 3) wrote its chart / Compose copies under
  deploy/deploy/… (its repository root was one level short), so drops 3 and later carried that tree and nothing
  compared it. It writes to the real copies now, refuses a destination that does not exist, and the checker fails if
  deploy/deploy exists. Remove it from the repository: git rm -r deploy/deploy (rsync does not delete).

Compose smoke: the checks are Python files with recorded answers; the "every target up" step really checks

• The red step. smoke-observability.sh checked the Prometheus targets with a python3 -c '…' snippet whose
  f-strings carried \": inside bash single quotes that reaches Python as a backslash, so every attempt was a
  SyntaxError and the targets were never checked (the Caddy diagnostic snippet had the same flaw). All the inline
  Python of smoke.sh and smoke-observability.sh moved to deploy/compose/smoke_checks.py: one subcommand per check
  (discovery, token, Grafana datasources / dashboards / inputs, rules, targets, build_info, series, the pipeline's
  trace id, Tempo, Loki, the backup gauge, Caddy's running routes), the answer on stdin, a one-line verdict, a reason
  on failure — never a traceback.
• Tested without Docker. deploy/compose/smoke_checks_test.py (run by compose-init.test.sh, CI job layout)
  runs every check against recorded answers in deploy/compose/test-fixtures/ — Prometheus 3.15.0 (targets all up and
  with the registry down, rules, the smoke's queries), Tempo 2.9.0 (/api/traces and /api/v2/traces of a
  pipeline-shaped trace), Loki 3.6.12 (query_range by trace_id structured metadata), Caddy 2.9.1's admin API,
  recorded from the real binaries running the profile's own configs; Grafana's answers are built from the
  provisioning files compose-init renders. Each check passes its good answer and fails its broken one with the
  expected reason. The same test extracts every Python snippet still in a shell script of the stack and the
  operator scripts (python3 -c '…' / "…", heredocs, cli_json expressions, with bash's quoting and '…'"$VAR"'…'
  concatenation) and compiles it as bash hands it over — it reports the CI failure on the previous
  smoke-observability.sh (line 86: "unexpected character after line continuation character").
• The steps after the targets (build_info per job, the exporters' series, the pipeline trace in Tempo, its logs in
  Loki, the backup gauge) have now run against those recorded answers; they have not yet run in CI.

Compose: Caddy reloads its rewritten Caddyfile; smoke.sh really tears down between passes; Grafana without plugin downloads; helm pinned

• Root cause of the red compose job (/grafana answered by the web app). smoke.sh's cleanup deleted
  .env.smoke and then ran docker compose down -v; every compose call reads that file (COMPOSE_ENV_FILES), so
  compose refused ("couldn't find env file"), the error went to /dev/null and the plain pass's stack stayed up. The
  --observability pass then came up over it: init was recreated (its environment changed) and rewrote the
  Caddyfile with the /grafana route, but compose does not recreate Caddy (its own config did not change), and Caddy
  kept the routes it had loaded — /grafana/api/health went to the catch-all, our index.html. Reproduced with
  Caddy 2.9.1 (file with the route, running config without it, the SPA answering).
• Fixes. Caddy runs with --watch and reloads the Caddyfile whenever init rewrites it — this was a product bug
  too: turning the observability profile on (or changing the domain) on an existing Compose cell left Caddy on its old
  routes until a restart. smoke.sh tears down before deleting its env file and says so when down fails, and starts
  by removing any stack of the project left over by an earlier run. On a Grafana failure the step prints the routes
  Caddy is running (admin API) next to the file's.
• Grafana: GF_PLUGINS_PREINSTALL_DISABLED=true and no plugin update checks on Compose; the same in the
  kube-prometheus-stack values (grafana.env), documented — no downloads at every start, air-gapped cells start clean.
• helm pinned to v4.3.1 in ci.yml and release.yml (check-workflows.sh fails on an unpinned setup-helm or
  on CI and release disagreeing).

CI fix after drops 2–3: generated BACKUP_REPORT_TOKEN pinned in chart-test; Grafana step of the Compose smoke readable

• chart job, "generated secrets: install == upgrade". Drop 2's BACKUP_REPORT_TOKEN went through the
  keep-existing helper (cell.generated, lookup of <release>-api-secrets) but with "value" "": no values
  override, so chart-test — which simulates the upgrade by feeding every generated value back as a value, because
  helm template has no cluster — could not pin it, and the api's checksum/secret differed between its install
  and upgrade renders. On a cluster lookup returned the stored token, so upgrades did not rotate it (the kind runs
  reported backups through the API after upgrades); the test was what broke. Now backup.reportToken (empty =
  generated, then kept) is the override, chart-test pins it, and chart-test fails on any cell.generated caller
  without a values override — before comparing checksums, with the secret's name. Reproduced and verified with a
  real helm (the drop 2–3 renders were done with a stand-in that hid it).
• compose job, smoke.sh --observability. The Grafana step parsed whatever curl -f printed: on any non-200
  that was nothing, hence JSONDecodeError … char 0 and no cause in the log. Every Grafana call now goes through gf,
  which prints the URL, the HTTP status and the first 300 bytes of the body (like login-page-check.sh) whenever the
  answer is not JSON with 200; the health wait gives up after 180 s with the Caddyfile's /grafana route, the
  container's state (OOMKilled, exit code, restarts, health) and the last log lines of Grafana and Caddy. Grafana's
  memory limit goes from 256 MiB to 512 MiB (its documented minimum; a first start runs its database migrations), it
  gets a healthcheck on /grafana/api/health, and its provisioning tree has the plugins/ and alerting/
  directories it reads at start (empty), so its log carries no "cannot read … provisioning" errors.
• Process. chart-test runs with a real helm before every drop and smoke.sh --observability on a real Docker
  before every drop that touches deploy/compose; a drop that could not do either says so first (CLAUDE.md).

Observability drop 3 — dashboards and alert rules, plus the six fixes from the rc.20 kind run (migration 0051)

• Metrics exist from the first scrape. Every enumerable series — job kind × terminal status, pipeline status ×
  trigger, claim and gate results, queue × result, SBOM / correlation sources, webhook providers, the runner's kinds
  × statuses — is published at zero at startup (KNOWN_LABEL_SETS, seedKnownSeries; counters through add(0),
  histograms as empty buckets rendered by the serializer until observed). rate() now sees the first failed job and
  histogram_quantile is a number after the first pipeline; exposition.spec.ts pins the zero page. The HTTP
  histogram and the per-pool gauges stay lazy (their label sets are not known at startup; documented).
• The claim span fits inside its job. It starts when the job became claimable — the new jobs.queued_at
  (migration 0051: creation for the first stage, the st...

Matelot v0.1.0-rc.21

No CHANGELOG section is titled ## v0.1.0-rc.21; these are the latest changes on the branch.

v0.1.0 — first self-hosted release

Images ghcr.io/matelot-cloud/{api,web,worker,runner} and the chart oci://ghcr.io/matelot-cloud/charts/secops-cell, signed, with SBOM attestations; install guide at https://matelot-cloud.github.io/website/docs/latest/install.html. The sections below are the drops that make up the release, newest first.

Compose smoke: the checks are Python files with recorded answers; the "every target up" step really checks

• The red step. smoke-observability.sh checked the Prometheus targets with a python3 -c '…' snippet whose
  f-strings carried \": inside bash single quotes that reaches Python as a backslash, so every attempt was a
  SyntaxError and the targets were never checked (the Caddy diagnostic snippet had the same flaw). All the inline
  Python of smoke.sh and smoke-observability.sh moved to deploy/compose/smoke_checks.py: one subcommand per check
  (discovery, token, Grafana datasources / dashboards / inputs, rules, targets, build_info, series, the pipeline's
  trace id, Tempo, Loki, the backup gauge, Caddy's running routes), the answer on stdin, a one-line verdict, a reason
  on failure — never a traceback.
• Tested without Docker. deploy/compose/smoke_checks_test.py (run by compose-init.test.sh, CI job layout)
  runs every check against recorded answers in deploy/compose/test-fixtures/ — Prometheus 3.15.0 (targets all up and
  with the registry down, rules, the smoke's queries), Tempo 2.9.0 (/api/traces and /api/v2/traces of a
  pipeline-shaped trace), Loki 3.6.12 (query_range by trace_id structured metadata), Caddy 2.9.1's admin API,
  recorded from the real binaries running the profile's own configs; Grafana's answers are built from the
  provisioning files compose-init renders. Each check passes its good answer and fails its broken one with the
  expected reason. The same test extracts every Python snippet still in a shell script of the stack and the
  operator scripts (python3 -c '…' / "…", heredocs, cli_json expressions, with bash's quoting and '…'"$VAR"'…'
  concatenation) and compiles it as bash hands it over — it reports the CI failure on the previous
  smoke-observability.sh (line 86: "unexpected character after line continuation character").
• The steps after the targets (build_info per job, the exporters' series, the pipeline trace in Tempo, its logs in
  Loki, the backup gauge) have now run against those recorded answers; they have not yet run in CI.

Compose: Caddy reloads its rewritten Caddyfile; smoke.sh really tears down between passes; Grafana without plugin downloads; helm pinned

• Root cause of the red compose job (/grafana answered by the web app). smoke.sh's cleanup deleted
  .env.smoke and then ran docker compose down -v; every compose call reads that file (COMPOSE_ENV_FILES), so
  compose refused ("couldn't find env file"), the error went to /dev/null and the plain pass's stack stayed up. The
  --observability pass then came up over it: init was recreated (its environment changed) and rewrote the
  Caddyfile with the /grafana route, but compose does not recreate Caddy (its own config did not change), and Caddy
  kept the routes it had loaded — /grafana/api/health went to the catch-all, our index.html. Reproduced with
  Caddy 2.9.1 (file with the route, running config without it, the SPA answering).
• Fixes. Caddy runs with --watch and reloads the Caddyfile whenever init rewrites it — this was a product bug
  too: turning the observability profile on (or changing the domain) on an existing Compose cell left Caddy on its old
  routes until a restart. smoke.sh tears down before deleting its env file and says so when down fails, and starts
  by removing any stack of the project left over by an earlier run. On a Grafana failure the step prints the routes
  Caddy is running (admin API) next to the file's.
• Grafana: GF_PLUGINS_PREINSTALL_DISABLED=true and no plugin update checks on Compose; the same in the
  kube-prometheus-stack values (grafana.env), documented — no downloads at every start, air-gapped cells start clean.
• helm pinned to v4.3.1 in ci.yml and release.yml (check-workflows.sh fails on an unpinned setup-helm or
  on CI and release disagreeing).

CI fix after drops 2–3: generated BACKUP_REPORT_TOKEN pinned in chart-test; Grafana step of the Compose smoke readable

• chart job, "generated secrets: install == upgrade". Drop 2's BACKUP_REPORT_TOKEN went through the
  keep-existing helper (cell.generated, lookup of <release>-api-secrets) but with "value" "": no values
  override, so chart-test — which simulates the upgrade by feeding every generated value back as a value, because
  helm template has no cluster — could not pin it, and the api's checksum/secret differed between its install
  and upgrade renders. On a cluster lookup returned the stored token, so upgrades did not rotate it (the kind runs
  reported backups through the API after upgrades); the test was what broke. Now backup.reportToken (empty =
  generated, then kept) is the override, chart-test pins it, and chart-test fails on any cell.generated caller
  without a values override — before comparing checksums, with the secret's name. Reproduced and verified with a
  real helm (the drop 2–3 renders were done with a stand-in that hid it).
• compose job, smoke.sh --observability. The Grafana step parsed whatever curl -f printed: on any non-200
  that was nothing, hence JSONDecodeError … char 0 and no cause in the log. Every Grafana call now goes through gf,
  which prints the URL, the HTTP status and the first 300 bytes of the body (like login-page-check.sh) whenever the
  answer is not JSON with 200; the health wait gives up after 180 s with the Caddyfile's /grafana route, the
  container's state (OOMKilled, exit code, restarts, health) and the last log lines of Grafana and Caddy. Grafana's
  memory limit goes from 256 MiB to 512 MiB (its documented minimum; a first start runs its database migrations), it
  gets a healthcheck on /grafana/api/health, and its provisioning tree has the plugins/ and alerting/
  directories it reads at start (empty), so its log carries no "cannot read … provisioning" errors.
• Process. chart-test runs with a real helm before every drop and smoke.sh --observability on a real Docker
  before every drop that touches deploy/compose; a drop that could not do either says so first (CLAUDE.md).

Observability drop 3 — dashboards and alert rules, plus the six fixes from the rc.20 kind run (migration 0051)

• Metrics exist from the first scrape. Every enumerable series — job kind × terminal status, pipeline status ×
  trigger, claim and gate results, queue × result, SBOM / correlation sources, webhook providers, the runner's kinds
  × statuses — is published at zero at startup (KNOWN_LABEL_SETS, seedKnownSeries; counters through add(0),
  histograms as empty buckets rendered by the serializer until observed). rate() now sees the first failed job and
  histogram_quantile is a number after the first pipeline; exposition.spec.ts pins the zero page. The HTTP
  histogram and the per-pool gauges stay lazy (their label sets are not known at startup; documented).
• The claim span fits inside its job. It starts when the job became claimable — the new jobs.queued_at
  (migration 0051: creation for the first stage, the stage opening for later ones, a retry or a lost lease afterwards;
  existing queued rows backfilled from created_at) — and ends at the hand-out; the runner's own long-poll wait is
  only matelot.claim.waited_ms. The job span gets a queued event, and the queue-wait histogram measures queued_at
  → started_at instead of created_at. The span-budget test fails on any span that starts before or ends after its
  parent.
• worker sweep under matelot-worker. emitCompletedSpan takes service: 'process'; only a pipeline's root and
  job spans go through the matelot-pipeline provider (the budget test asserts the split).
• apply keeps the cell's image tag. cell-provision.sh apply --tag X writes the four image tags into
  deploy/cells/<slug>.values.yaml first (scripts/lib/values-image-tag.py, every other line untouched), so later
  applies without --tag keep X; an apply without --tag on a release that runs a different tag than the file names is
  refused with both ways out — the silent move to the chart's default appVersion (realm-sync without its image, hook
  timeout, release failed) cannot happen again. cell-provision.test.sh covers it.
• Backups wait for Postgres. backup.sh resolves the host and pg_isready for up to two minutes (WAIT_SECONDS)
  before dumping — the CronJob firing right after a node restart failed with "could not translate host name" — and a
  failure report now carries the last error (the DNS / pg_isready message, pg_dump's stderr), so MatelotBackupFailed
  has the reason.
• Polls are never traces. The sampler (telemetry/sampler.ts) drops any parentless GET that carries
  x-matelot-poll: 1: the web app marks every background refetch of a query with a refresh interval (hooks.ts wraps
  useQuery; asPoll in client.ts; test/poll-header.mjs), the CLI its --wait loops (next to the unsampled
  traceparent for older cells). Other parentless GETs are recorded for a share of the trace ids —
  MATELOT_TRACES_GET_RATIO / observability.otlp.getRatio, default 0.1 — mutations always; the base sampler still
  wins when stricter. The span-budget test simulates a web session (ten marked polls of four routes: 0 roots).
  One open tab was ~150 root traces in 10 minutes.
• Dashboards (deploy/observability/dashboards, generated by tools/dashboards/build.py): Cell overview,
  Pipelines (v...

Matelot v0.1.0-rc.20

No CHANGELOG section is titled ## v0.1.0-rc.20; these are the latest changes on the branch.

v0.1.0 — first self-hosted release

Images ghcr.io/matelot-cloud/{api,web,worker,runner} and the chart oci://ghcr.io/matelot-cloud/charts/secops-cell, signed, with SBOM attestations; install guide at https://matelot-cloud.github.io/website/docs/latest/install.html. The sections below are the drops that make up the release, newest first.

Observability drop 2 — delivery: ServiceMonitor, exporters, backup gauges, the Compose observability profile (migration 0050); release steps retried

• Release robustness. cosign sign, syft / cosign attest (.github/scripts/sign-image.sh), the chart
  signature and the GitHub release are retried (3 attempts, 30 s then 60 s, each failure logged as a workflow
  warning; .github/scripts/retry.sh). The release step is .github/scripts/publish-release.sh: an existing release
  (a half-succeeded earlier attempt, or a re-run of the job) is edited and its assets re-uploaded with --clobber
  instead of failing on "already exists". check-workflows.sh guards both and self-tests the retry helper.
• Helm. observability.metrics.serviceMonitor renders one ServiceMonitor over headless metrics Services
  (<release>-<name>-metrics, port metrics, job matelot-<name>, a cell label) for api, worker, runner and, with
  observability.metrics.exporters, postgres_exporter v0.20.0 / redis_exporter v1.94.0 sidecars of the bundled
  Postgres / Redis, Keycloak's metrics (KC_METRICS_ENABLED, management port 9000) and the registry's debug listener
  (5001). The scrape NetworkPolicy admits those ports from scrapeFromNamespaces only; nothing reaches the ingress.
  Defaults render no monitoring CRD.
• OTLP headers only from a Secret. observability.otlp.headersSecret (+ headersSecretKey) puts
  OTEL_EXPORTER_OTLP_HEADERS on every component through a secretKeyRef; observability.otlp.headers is removed and
  refused at render time, and the chart's <release>-otel Secret is gone. Upgrade note: a cell that set
  otlp.headers must move the value into a Secret and name it before the next apply.
• Backup gauges on Helm and Compose. Backup runs report their outcome to POST /api/v1/internal/backup-reports
  (bearer BACKUP_REPORT_TOKEN, generated into the api secrets on both deliveries; constant-time comparison; not in
  the OpenAPI document; table backup_reports, 180 days). The worker publishes
  matelot_backup_last_success_timestamp_seconds, …_last_failure_timestamp_seconds and
  matelot_backup_last_duration_seconds. The report goes over curl or plain bash /dev/tcp (the postgres image has
  no curl) and never fails the backup; the S3 upload reports too.
• Compose observability profile (COMPOSE_PROFILES=observability): OpenTelemetry Collector, Prometheus
  (exemplar storage), Tempo, Loki, Alloy (this project's container logs, trace_id / span_id as structured
  metadata) and Grafana at <domain>/grafana/ behind Caddy, signing in with the cell's Keycloak (client grafana;
  local admin password in init show), datasources provisioned with exemplars → Tempo → Loki by trace_id, plus the
  two exporters. No port is published on the host. realm-sync now creates clients marked
  matelot.createIfMissing in an existing realm and keeps a rendered client secret in sync.
• CI. smoke.sh --observability runs a second Compose pass: every Prometheus target up, matelot_build_info from
  api / worker / runner, a series from each exporter, a checkout pipeline's trace in Tempo and its log lines in Loki
  by trace_id, the backup gauge — all through Caddy and Grafana's datasource proxy. The chart job installs the
  prometheus-operator CRDs (v0.94.1) on kind and dry-runs the everything-on render, fresh and as an upgrade.
  chart-test checks that every ServiceMonitor endpoint reaches a Service port backed by a pod.
• kind. make kind-observability TENANT=<slug> installs kube-prometheus-stack 92.11.0, Tempo 1.24.4, Loki 7.3.0
  (single binary) and Alloy 1.14.1 with deploy/observability/kind/*.values.yaml, then writes the cell's
  observability block (ServiceMonitor with release: kube-prometheus-stack, exporters, OTLP to Tempo) and applies.
  docs/observability.md has the recommended collector configuration and the kube-prometheus-stack recipe.
• Fixed on the way. Compose init never received MATELOT_THEME_*, MATELOT_OTLP_*, MATELOT_ENVIRONMENT or
  MATELOT_TRACES_SAMPLER from .env (missing from x-config-env): the sign-in branding and the OTLP settings of
  .env were silently ignored on Compose. compose-init.test.sh now fails on any variable the script reads that
  compose does not pass. The chart's registry notified http://<release>-api:3000, but the api Service listens on
  80: push notifications never reached the API on Helm.

Observability drop 1 fix-up 3: probe pods without a verdict are retried, client polls and worker timers leave no traces of their own (CLI 2.24.2)

• checks: "the probe pod gave no verdict" right after a rollout (the curl pod's output empty while the API answered
  seconds later) is no longer a failure: probe_cell shares the 60 s budget between the curl loop and the pod
  itself and retries an empty verdict; when it persists the last pod is kept and its phase, last event and logs are
  printed. cell-provision.test.sh covers two empty pods then READY (passes, says so) and a persisting one (fails
  with the diagnostics).
• matelot pipeline --wait (and the correlation wait) send each poll with an unsampled W3C traceparent, like the
  runner's log batches, and the cell's sampler (telemetry/sampler.ts) records no root trace for GET /pipelines/:id, /pipelines/:id/jobs, /jobs/:id, /jobs/:id/logs and the correlation job endpoint from any
  client that sends nothing; a poll inside a traced request stays its child. 69 root traces in 20 minutes → 0.
• The orphan pg.query:SELECT platform root span was the worker's signing-keys load at module initialisation (320
  ms, the first connection). Both the api's and the worker's module initialisation now run with instrumentation
  suppressed (the api starts listening outside that context, so requests stay traced); every worker timer is a root
  span worker <loop> (registry reconcile, exceptions, digest, remediation reconcile, snapshots, risk, usage, image
  sync, enrichment — correlation already was), the every-few-seconds housekeeping (sweeper, gauges, platform
  heartbeat, the policy_eval queue query) is suppressed, and the sweeper emits a worker sweep span only when it
  requeued, failed or marked something offline.
• span-budget.mjs: 8 --wait-style polls (half with the CLI's unsampled header, half with none) must add 0 spans;
  every root span in both processes must be a request, a worker <loop> or a pipeline — the orphan query failed it
  before the fix; the stray check now matches this pipeline's job ids only (a development database's stale jobs no
  longer trip it).

Observability drop 1 fix-up 2: claim polls leave no trace, every job's claim / start / complete in the pipeline trace, make kind-jaeger writes the values file

• Verified on kind (rc.17): the pipeline trace is 42 spans, the root under matelot-pipeline. Left over: every
  runner claim poll was a root trace of its own (67 spans, eight identical traces in 15 minutes with no job), and in
  one trace four of six jobs had their job <kind>, /start and /complete spans outside the pipeline's trace.
• Claim polls: the request span of POST /runner/jobs/claim is dropped by the trimming rules and the polling loop
  runs with instrumentation suppressed (suppressTracing), so an empty poll leaves nothing; a poll that hands out a
  job leaves ONE claim span, started from the job's stored traceparent (never the request's context) in the
  pipeline's trace under the job's span, with matelot.claim.polls, matelot.claim.waited_ms, runner and pool.
• The misparenting could not be reproduced here: with the real runner against the built api and worker (file
  exporter), all six runner jobs had their job <kind>, /start and /complete spans in the pipeline's trace
  (30 spans); the runner sends the job's traceparent on /start (its job span is started from job.traceParent).
  The span-budget test now pins what rc.17 showed broken: for every job of the security template its job <kind>
  span, exactly one claim under it, /start and /complete in the pipeline's trace and none in another, and six
  empty polls adding zero spans — if it recurs on rc.18, the file exporter (OTEL_TRACES_EXPORTER=file) on the
  api pod gives the span-level evidence. Skipped jobs (an earlier stage failed) now get a span too, so the trace shows
  every job of the template.
• make kind-jaeger TENANT=<slug> writes observability.otlp.endpoint into deploy/cells/<slug>.values.yaml
  (scripts/lib/values-observability.py, standard library, replaces or appends the observability: block; tested)
  and runs cell-provision.sh apply, so the setting survives upgrades like every other cell setting; a cell without
  a values file (make deploy) is upgraded in place with --reuse-values and told so. The cell template carries the
  block.

Observability drop 1 fix-up: the pipeline trace trimmed from 1 824 spans to a budget of 150; pipeline spans under one service

• Verified on kind (rc.16): one trace per pipeline across api / runner / worker, metrics on all three — and 1 824
  spans in the trace, most of them Redis commands of the rate limiter and BullMQ housekeeping (incr, multi,
  pexpire, sub-millisecond), pg-pool.connect, and one server span with children per runner /logs batch.
• telemetry/trim.ts, a span processor applied when a span ends: ioredis housekeeping (rl: / bull: keys,
  multi / exec / pexpire / `evalsh...

Matelot v0.1.0-rc.19

No CHANGELOG section is titled ## v0.1.0-rc.19; these are the latest changes on the branch.

v0.1.0 — first self-hosted release

Images ghcr.io/matelot-cloud/{api,web,worker,runner} and the chart oci://ghcr.io/matelot-cloud/charts/secops-cell, signed, with SBOM attestations; install guide at https://matelot-cloud.github.io/website/docs/latest/install.html. The sections below are the drops that make up the release, newest first.

Observability drop 1 fix-up 3: probe pods without a verdict are retried, client polls and worker timers leave no traces of their own (CLI 2.24.2)

• checks: "the probe pod gave no verdict" right after a rollout (the curl pod's output empty while the API answered
  seconds later) is no longer a failure: probe_cell shares the 60 s budget between the curl loop and the pod
  itself and retries an empty verdict; when it persists the last pod is kept and its phase, last event and logs are
  printed. cell-provision.test.sh covers two empty pods then READY (passes, says so) and a persisting one (fails
  with the diagnostics).
• matelot pipeline --wait (and the correlation wait) send each poll with an unsampled W3C traceparent, like the
  runner's log batches, and the cell's sampler (telemetry/sampler.ts) records no root trace for GET /pipelines/:id, /pipelines/:id/jobs, /jobs/:id, /jobs/:id/logs and the correlation job endpoint from any
  client that sends nothing; a poll inside a traced request stays its child. 69 root traces in 20 minutes → 0.
• The orphan pg.query:SELECT platform root span was the worker's signing-keys load at module initialisation (320
  ms, the first connection). Both the api's and the worker's module initialisation now run with instrumentation
  suppressed (the api starts listening outside that context, so requests stay traced); every worker timer is a root
  span worker <loop> (registry reconcile, exceptions, digest, remediation reconcile, snapshots, risk, usage, image
  sync, enrichment — correlation already was), the every-few-seconds housekeeping (sweeper, gauges, platform
  heartbeat, the policy_eval queue query) is suppressed, and the sweeper emits a worker sweep span only when it
  requeued, failed or marked something offline.
• span-budget.mjs: 8 --wait-style polls (half with the CLI's unsampled header, half with none) must add 0 spans;
  every root span in both processes must be a request, a worker <loop> or a pipeline — the orphan query failed it
  before the fix; the stray check now matches this pipeline's job ids only (a development database's stale jobs no
  longer trip it).

Observability drop 1 fix-up 2: claim polls leave no trace, every job's claim / start / complete in the pipeline trace, make kind-jaeger writes the values file

• Verified on kind (rc.17): the pipeline trace is 42 spans, the root under matelot-pipeline. Left over: every
  runner claim poll was a root trace of its own (67 spans, eight identical traces in 15 minutes with no job), and in
  one trace four of six jobs had their job <kind>, /start and /complete spans outside the pipeline's trace.
• Claim polls: the request span of POST /runner/jobs/claim is dropped by the trimming rules and the polling loop
  runs with instrumentation suppressed (suppressTracing), so an empty poll leaves nothing; a poll that hands out a
  job leaves ONE claim span, started from the job's stored traceparent (never the request's context) in the
  pipeline's trace under the job's span, with matelot.claim.polls, matelot.claim.waited_ms, runner and pool.
• The misparenting could not be reproduced here: with the real runner against the built api and worker (file
  exporter), all six runner jobs had their job <kind>, /start and /complete spans in the pipeline's trace
  (30 spans); the runner sends the job's traceparent on /start (its job span is started from job.traceParent).
  The span-budget test now pins what rc.17 showed broken: for every job of the security template its job <kind>
  span, exactly one claim under it, /start and /complete in the pipeline's trace and none in another, and six
  empty polls adding zero spans — if it recurs on rc.18, the file exporter (OTEL_TRACES_EXPORTER=file) on the
  api pod gives the span-level evidence. Skipped jobs (an earlier stage failed) now get a span too, so the trace shows
  every job of the template.
• make kind-jaeger TENANT=<slug> writes observability.otlp.endpoint into deploy/cells/<slug>.values.yaml
  (scripts/lib/values-observability.py, standard library, replaces or appends the observability: block; tested)
  and runs cell-provision.sh apply, so the setting survives upgrades like every other cell setting; a cell without
  a values file (make deploy) is upgraded in place with --reuse-values and told so. The cell template carries the
  block.

Observability drop 1 fix-up: the pipeline trace trimmed from 1 824 spans to a budget of 150; pipeline spans under one service

• Verified on kind (rc.16): one trace per pipeline across api / runner / worker, metrics on all three — and 1 824
  spans in the trace, most of them Redis commands of the rate limiter and BullMQ housekeeping (incr, multi,
  pexpire, sub-millisecond), pg-pool.connect, and one server span with children per runner /logs batch.
• telemetry/trim.ts, a span processor applied when a span ends: ioredis housekeeping (rl: / bull: keys,
  multi / exec / pexpire / evalsha) is dropped whatever its duration, any other Redis command under 5 ms too;
  pg-pool.connect / pg.connect always, pg.query:* under 5 ms — slow queries and commands stay. Dropped spans
  are leaves, nothing is orphaned (trim.spec.ts).
• The runner sends its /logs batches with the sampled flag off: same trace id, no server span at the cell; each
  batch is an event logs shipped {lines, seq} on the job span (runner telemetry.test.ts checks the header).
• The root span was attributed to whichever process finished the pipeline (the worker, after the gate — which is why
  a search under service matelot-api found nothing). Pipeline-level spans (pipeline <name>, job <kind>) now
  come from a second tracer provider with service matelot-pipeline (attribute matelot.emitted_by), the same
  trimming-free export; the Jaeger recipe in docs and make kind-jaeger say so.
• apps/api/test/span-budget.mjs, run by CI's api job after the build (a Redis service was added): the built api and
  worker with the tracing preload and OTEL_TRACES_EXPORTER=file (new), a security pipeline, a fake runner that
  behaves like the real one; fails above 150 spans, on any /logs span, on a root not under matelot-pipeline,
  on fewer than 7 job spans, on a surviving pool checkout or sub-5 ms Redis span. Today: 28 spans from the cell
  (the real runner adds its own ~35).

Observability drop 1 — instrumentation: OTLP traces (one per pipeline), Prometheus metrics with exemplars, JSON logs with trace ids (migration 0049, runner 0.3.3)

• Traces. apps/api/src/telemetry/tracing.ts, preloaded with node -r ./dist/telemetry/tracing.js by the api and
  worker images, Compose and pnpm start: the OpenTelemetry SDK with the http, express (route template only),
  pg, ioredis and undici instrumentations. Off is a no-op (one env read, no SDK module loaded) until
  OTEL_EXPORTER_OTLP_ENDPOINT is set or MATELOT_TRACING_ENABLED=true; export is asynchronous with a bounded queue
  (2048 spans), an unreachable collector costs dropped spans and one warning, never a slow request or a failed
  /health/live; SIGTERM and Nest's shutdown flush with a 5 s cap. Protocols http/protobuf (default), grpc,
  http/json; sampler from OTEL_TRACES_SAMPLER; probes, /metrics and runner heartbeats are never traced.
• One trace per pipeline. Ids are fixed at creation and stored on the rows (pipelines.trace_parent,
  jobs.trace_parent); the claim response hands the runner its job's traceparent; the runner's job run <kind> span
  (phases checkout, tool syft|trivy|semgrep, upload sbom, upload findings, correlate) goes under it and
  every call back carries the traceparent header, so the API side of a job joins the trace; the worker's gate
  evaluation runs under the policy_eval job's span; BullMQ job data carries the enqueuer's traceparent. The
  pipeline's root span and each job's span (start = created, events claimed / started, kind, status, exit code,
  runner) are emitted at their terminal status with the stored ids by whichever process finished them — nothing in
  memory across requests or replicas (PinnedIdGenerator). Verified against a real Jaeger 2.22: a security pipeline
  (checkout, syft, trivy ×2, semgrep, gate) is one trace across api, runner and worker — and the trace already
  showed something the logs never did: semgrep --version taking 105 s in a sandbox before the scan started.
• Metrics. One meter (telemetry/metrics.ts) with explicit buckets through Views (http 5 ms–10 s, pipeline jobs
  1 s–30 min, correlation 100 ms–5 min), served on port 9464 (MATELOT_METRICS_PORT, never on the ingress) by our own
  OpenMetrics serializer (telemetry/exposition.ts) because the SDK's Prometheus exporter renders no trace
  exemplars: a scraper accepting OpenMetrics gets # {trace_id,span_id} value ts per bucket, pipeline histograms
  pointing at the pipeline's trace — Grafana jumps from a slow p95 to the exact trace. exposition.spec.ts pins the
  format (names, _total, unit suffixes, buckets, matelot_build_info = 1, every label in ALLOWED_LABELS —
  an instrument given an id or a name as a label throws). The table of docs/observability.md is implemented:
  http RED by route template, pool connections, BullMQ and pipeline queue depth / age, pipeline jobs and durations
  (+ queue wait), pipelines, runners online / busy / **`matelot_runner_last_seen_timestamp_seconds{...

Matelot v0.1.0-rc.18

No CHANGELOG section is titled ## v0.1.0-rc.18; these are the latest changes on the branch.

v0.1.0 — first self-hosted release

Images ghcr.io/matelot-cloud/{api,web,worker,runner} and the chart oci://ghcr.io/matelot-cloud/charts/secops-cell, signed, with SBOM attestations; install guide at https://matelot-cloud.github.io/website/docs/latest/install.html. The sections below are the drops that make up the release, newest first.

Observability drop 1 fix-up 2: claim polls leave no trace, every job's claim / start / complete in the pipeline trace, make kind-jaeger writes the values file

• Verified on kind (rc.17): the pipeline trace is 42 spans, the root under matelot-pipeline. Left over: every
  runner claim poll was a root trace of its own (67 spans, eight identical traces in 15 minutes with no job), and in
  one trace four of six jobs had their job <kind>, /start and /complete spans outside the pipeline's trace.
• Claim polls: the request span of POST /runner/jobs/claim is dropped by the trimming rules and the polling loop
  runs with instrumentation suppressed (suppressTracing), so an empty poll leaves nothing; a poll that hands out a
  job leaves ONE claim span, started from the job's stored traceparent (never the request's context) in the
  pipeline's trace under the job's span, with matelot.claim.polls, matelot.claim.waited_ms, runner and pool.
• The misparenting could not be reproduced here: with the real runner against the built api and worker (file
  exporter), all six runner jobs had their job <kind>, /start and /complete spans in the pipeline's trace
  (30 spans); the runner sends the job's traceparent on /start (its job span is started from job.traceParent).
  The span-budget test now pins what rc.17 showed broken: for every job of the security template its job <kind>
  span, exactly one claim under it, /start and /complete in the pipeline's trace and none in another, and six
  empty polls adding zero spans — if it recurs on rc.18, the file exporter (OTEL_TRACES_EXPORTER=file) on the
  api pod gives the span-level evidence. Skipped jobs (an earlier stage failed) now get a span too, so the trace shows
  every job of the template.
• make kind-jaeger TENANT=<slug> writes observability.otlp.endpoint into deploy/cells/<slug>.values.yaml
  (scripts/lib/values-observability.py, standard library, replaces or appends the observability: block; tested)
  and runs cell-provision.sh apply, so the setting survives upgrades like every other cell setting; a cell without
  a values file (make deploy) is upgraded in place with --reuse-values and told so. The cell template carries the
  block.

Observability drop 1 fix-up: the pipeline trace trimmed from 1 824 spans to a budget of 150; pipeline spans under one service

• Verified on kind (rc.16): one trace per pipeline across api / runner / worker, metrics on all three — and 1 824
  spans in the trace, most of them Redis commands of the rate limiter and BullMQ housekeeping (incr, multi,
  pexpire, sub-millisecond), pg-pool.connect, and one server span with children per runner /logs batch.
• telemetry/trim.ts, a span processor applied when a span ends: ioredis housekeeping (rl: / bull: keys,
  multi / exec / pexpire / evalsha) is dropped whatever its duration, any other Redis command under 5 ms too;
  pg-pool.connect / pg.connect always, pg.query:* under 5 ms — slow queries and commands stay. Dropped spans
  are leaves, nothing is orphaned (trim.spec.ts).
• The runner sends its /logs batches with the sampled flag off: same trace id, no server span at the cell; each
  batch is an event logs shipped {lines, seq} on the job span (runner telemetry.test.ts checks the header).
• The root span was attributed to whichever process finished the pipeline (the worker, after the gate — which is why
  a search under service matelot-api found nothing). Pipeline-level spans (pipeline <name>, job <kind>) now
  come from a second tracer provider with service matelot-pipeline (attribute matelot.emitted_by), the same
  trimming-free export; the Jaeger recipe in docs and make kind-jaeger say so.
• apps/api/test/span-budget.mjs, run by CI's api job after the build (a Redis service was added): the built api and
  worker with the tracing preload and OTEL_TRACES_EXPORTER=file (new), a security pipeline, a fake runner that
  behaves like the real one; fails above 150 spans, on any /logs span, on a root not under matelot-pipeline,
  on fewer than 7 job spans, on a surviving pool checkout or sub-5 ms Redis span. Today: 28 spans from the cell
  (the real runner adds its own ~35).

Observability drop 1 — instrumentation: OTLP traces (one per pipeline), Prometheus metrics with exemplars, JSON logs with trace ids (migration 0049, runner 0.3.3)

• Traces. apps/api/src/telemetry/tracing.ts, preloaded with node -r ./dist/telemetry/tracing.js by the api and
  worker images, Compose and pnpm start: the OpenTelemetry SDK with the http, express (route template only),
  pg, ioredis and undici instrumentations. Off is a no-op (one env read, no SDK module loaded) until
  OTEL_EXPORTER_OTLP_ENDPOINT is set or MATELOT_TRACING_ENABLED=true; export is asynchronous with a bounded queue
  (2048 spans), an unreachable collector costs dropped spans and one warning, never a slow request or a failed
  /health/live; SIGTERM and Nest's shutdown flush with a 5 s cap. Protocols http/protobuf (default), grpc,
  http/json; sampler from OTEL_TRACES_SAMPLER; probes, /metrics and runner heartbeats are never traced.
• One trace per pipeline. Ids are fixed at creation and stored on the rows (pipelines.trace_parent,
  jobs.trace_parent); the claim response hands the runner its job's traceparent; the runner's job run <kind> span
  (phases checkout, tool syft|trivy|semgrep, upload sbom, upload findings, correlate) goes under it and
  every call back carries the traceparent header, so the API side of a job joins the trace; the worker's gate
  evaluation runs under the policy_eval job's span; BullMQ job data carries the enqueuer's traceparent. The
  pipeline's root span and each job's span (start = created, events claimed / started, kind, status, exit code,
  runner) are emitted at their terminal status with the stored ids by whichever process finished them — nothing in
  memory across requests or replicas (PinnedIdGenerator). Verified against a real Jaeger 2.22: a security pipeline
  (checkout, syft, trivy ×2, semgrep, gate) is one trace across api, runner and worker — and the trace already
  showed something the logs never did: semgrep --version taking 105 s in a sandbox before the scan started.
• Metrics. One meter (telemetry/metrics.ts) with explicit buckets through Views (http 5 ms–10 s, pipeline jobs
  1 s–30 min, correlation 100 ms–5 min), served on port 9464 (MATELOT_METRICS_PORT, never on the ingress) by our own
  OpenMetrics serializer (telemetry/exposition.ts) because the SDK's Prometheus exporter renders no trace
  exemplars: a scraper accepting OpenMetrics gets # {trace_id,span_id} value ts per bucket, pipeline histograms
  pointing at the pipeline's trace — Grafana jumps from a slow p95 to the exact trace. exposition.spec.ts pins the
  format (names, _total, unit suffixes, buckets, matelot_build_info = 1, every label in ALLOWED_LABELS —
  an instrument given an id or a name as a label throws). The table of docs/observability.md is implemented:
  http RED by route template, pool connections, BullMQ and pipeline queue depth / age, pipeline jobs and durations
  (+ queue wait), pipelines, runners online / busy / matelot_runner_last_seen_timestamp_seconds{pool} (so
  NoRunnerOnline is an expression on the scrape), claims, correlation duration, open findings by severity and
  kind, gate evaluations, integration sync failures, webhook lag, build_info; gauges that need a query are
  refreshed by the worker on a timer (WorkerGauges), never on scrape, and their absent series vanish. Optional
  OTLP push with OTEL_METRICS_EXPORTER=prometheus,otlp. The runner serves its own page (RUNNER_METRICS_PORT):
  job durations with exemplars, jobs by status, busy, build_info.
• Logs. One line shape for api, worker and runner (telemetry/log-schema.ts, the runner's JsonLogger):
  time, level, component, cell, msg, context, trace_id, span_id, request_id, …; LOG_LEVEL=info,jobs=debug per
  module; one redaction list applied at any depth, with a test that logs every known secret key nested three levels
  and fails if a value survives. The runner's console.* is gone.
• Delivery. Env in env.ts, the api ConfigMap and compose-init (MATELOT_OTLP_* in .env); chart values
  observability.{metrics,otlp,environment} — metrics ports on api / worker / managed runners, a NetworkPolicy
  admitting scrapers from metrics.scrapeFromNamespaces, OTEL_* on every component once otlp.endpoint is set
  (headers in a Secret); chart-test renders defaults / otlp / metrics-off. make kind-jaeger TENANT=acme puts a
  Jaeger all-in-one on kind and points the cell at it. /api/v1/version carries commit and builtAt (image
  build arguments from the release workflow). The runner ships no dependency: its telemetry.ts (OTLP/HTTP JSON,
  bounded batching, exemplars, redaction) is 300 lines of its own, tested with node --test.
• Not in this drop (drop 2 onwards): ServiceMonitors / PrometheusRule / dashboards, the Compose observability
  profile, backup gauges, OTLP log export, Settings → Observability.

Design: observability milestone (M14) — docs/observability.md, ADR 0004

• The design for platform self-monitoring, OpenTelemetry-first and on the customer's stack: what every process
  emits (OTLP traces, Prometheus metrics on port 9464, JSON logs with trace_id), one trace per pip...

Matelot v0.1.0-rc.17

No CHANGELOG section is titled ## v0.1.0-rc.17; these are the latest changes on the branch.

v0.1.0 — first self-hosted release

Images ghcr.io/matelot-cloud/{api,web,worker,runner} and the chart oci://ghcr.io/matelot-cloud/charts/secops-cell, signed, with SBOM attestations; install guide at https://matelot-cloud.github.io/website/docs/latest/install.html. The sections below are the drops that make up the release, newest first.

Observability drop 1 fix-up: the pipeline trace trimmed from 1 824 spans to a budget of 150; pipeline spans under one service

• Verified on kind (rc.16): one trace per pipeline across api / runner / worker, metrics on all three — and 1 824
  spans in the trace, most of them Redis commands of the rate limiter and BullMQ housekeeping (incr, multi,
  pexpire, sub-millisecond), pg-pool.connect, and one server span with children per runner /logs batch.
• telemetry/trim.ts, a span processor applied when a span ends: ioredis housekeeping (rl: / bull: keys,
  multi / exec / pexpire / evalsha) is dropped whatever its duration, any other Redis command under 5 ms too;
  pg-pool.connect / pg.connect always, pg.query:* under 5 ms — slow queries and commands stay. Dropped spans
  are leaves, nothing is orphaned (trim.spec.ts).
• The runner sends its /logs batches with the sampled flag off: same trace id, no server span at the cell; each
  batch is an event logs shipped {lines, seq} on the job span (runner telemetry.test.ts checks the header).
• The root span was attributed to whichever process finished the pipeline (the worker, after the gate — which is why
  a search under service matelot-api found nothing). Pipeline-level spans (pipeline <name>, job <kind>) now
  come from a second tracer provider with service matelot-pipeline (attribute matelot.emitted_by), the same
  trimming-free export; the Jaeger recipe in docs and make kind-jaeger say so.
• apps/api/test/span-budget.mjs, run by CI's api job after the build (a Redis service was added): the built api and
  worker with the tracing preload and OTEL_TRACES_EXPORTER=file (new), a security pipeline, a fake runner that
  behaves like the real one; fails above 150 spans, on any /logs span, on a root not under matelot-pipeline,
  on fewer than 7 job spans, on a surviving pool checkout or sub-5 ms Redis span. Today: 28 spans from the cell
  (the real runner adds its own ~35).

Observability drop 1 — instrumentation: OTLP traces (one per pipeline), Prometheus metrics with exemplars, JSON logs with trace ids (migration 0049, runner 0.3.3)

• Traces. apps/api/src/telemetry/tracing.ts, preloaded with node -r ./dist/telemetry/tracing.js by the api and
  worker images, Compose and pnpm start: the OpenTelemetry SDK with the http, express (route template only),
  pg, ioredis and undici instrumentations. Off is a no-op (one env read, no SDK module loaded) until
  OTEL_EXPORTER_OTLP_ENDPOINT is set or MATELOT_TRACING_ENABLED=true; export is asynchronous with a bounded queue
  (2048 spans), an unreachable collector costs dropped spans and one warning, never a slow request or a failed
  /health/live; SIGTERM and Nest's shutdown flush with a 5 s cap. Protocols http/protobuf (default), grpc,
  http/json; sampler from OTEL_TRACES_SAMPLER; probes, /metrics and runner heartbeats are never traced.
• One trace per pipeline. Ids are fixed at creation and stored on the rows (pipelines.trace_parent,
  jobs.trace_parent); the claim response hands the runner its job's traceparent; the runner's job run <kind> span
  (phases checkout, tool syft|trivy|semgrep, upload sbom, upload findings, correlate) goes under it and
  every call back carries the traceparent header, so the API side of a job joins the trace; the worker's gate
  evaluation runs under the policy_eval job's span; BullMQ job data carries the enqueuer's traceparent. The
  pipeline's root span and each job's span (start = created, events claimed / started, kind, status, exit code,
  runner) are emitted at their terminal status with the stored ids by whichever process finished them — nothing in
  memory across requests or replicas (PinnedIdGenerator). Verified against a real Jaeger 2.22: a security pipeline
  (checkout, syft, trivy ×2, semgrep, gate) is one trace across api, runner and worker — and the trace already
  showed something the logs never did: semgrep --version taking 105 s in a sandbox before the scan started.
• Metrics. One meter (telemetry/metrics.ts) with explicit buckets through Views (http 5 ms–10 s, pipeline jobs
  1 s–30 min, correlation 100 ms–5 min), served on port 9464 (MATELOT_METRICS_PORT, never on the ingress) by our own
  OpenMetrics serializer (telemetry/exposition.ts) because the SDK's Prometheus exporter renders no trace
  exemplars: a scraper accepting OpenMetrics gets # {trace_id,span_id} value ts per bucket, pipeline histograms
  pointing at the pipeline's trace — Grafana jumps from a slow p95 to the exact trace. exposition.spec.ts pins the
  format (names, _total, unit suffixes, buckets, matelot_build_info = 1, every label in ALLOWED_LABELS —
  an instrument given an id or a name as a label throws). The table of docs/observability.md is implemented:
  http RED by route template, pool connections, BullMQ and pipeline queue depth / age, pipeline jobs and durations
  (+ queue wait), pipelines, runners online / busy / matelot_runner_last_seen_timestamp_seconds{pool} (so
  NoRunnerOnline is an expression on the scrape), claims, correlation duration, open findings by severity and
  kind, gate evaluations, integration sync failures, webhook lag, build_info; gauges that need a query are
  refreshed by the worker on a timer (WorkerGauges), never on scrape, and their absent series vanish. Optional
  OTLP push with OTEL_METRICS_EXPORTER=prometheus,otlp. The runner serves its own page (RUNNER_METRICS_PORT):
  job durations with exemplars, jobs by status, busy, build_info.
• Logs. One line shape for api, worker and runner (telemetry/log-schema.ts, the runner's JsonLogger):
  time, level, component, cell, msg, context, trace_id, span_id, request_id, …; LOG_LEVEL=info,jobs=debug per
  module; one redaction list applied at any depth, with a test that logs every known secret key nested three levels
  and fails if a value survives. The runner's console.* is gone.
• Delivery. Env in env.ts, the api ConfigMap and compose-init (MATELOT_OTLP_* in .env); chart values
  observability.{metrics,otlp,environment} — metrics ports on api / worker / managed runners, a NetworkPolicy
  admitting scrapers from metrics.scrapeFromNamespaces, OTEL_* on every component once otlp.endpoint is set
  (headers in a Secret); chart-test renders defaults / otlp / metrics-off. make kind-jaeger TENANT=acme puts a
  Jaeger all-in-one on kind and points the cell at it. /api/v1/version carries commit and builtAt (image
  build arguments from the release workflow). The runner ships no dependency: its telemetry.ts (OTLP/HTTP JSON,
  bounded batching, exemplars, redaction) is 300 lines of its own, tested with node --test.
• Not in this drop (drop 2 onwards): ServiceMonitors / PrometheusRule / dashboards, the Compose observability
  profile, backup gauges, OTLP log export, Settings → Observability.

Design: observability milestone (M14) — docs/observability.md, ADR 0004

• The design for platform self-monitoring, OpenTelemetry-first and on the customer's stack: what every process
  emits (OTLP traces, Prometheus metrics on port 9464, JSON logs with trace_id), one trace per pipeline carried
  through the runner protocol, the metric table with its cardinality rules, the OTEL_* / MATELOT_* configuration,
  the Helm observability.* block (ServiceMonitors, PrometheusRule, dashboards ConfigMaps — all off by default) and
  the Compose observability profile, the four dashboards and twelve alerts, Settings → Observability and the
  doctor section, and the four drops with their acceptance. No code in this drop.

Fix: apply on an existing cell failed its own server-side dry run ("Job globex-realm-sync … field is immutable")

• The pre-upgrade kubectl apply --dry-run=server included the realm-sync hook Job, which exists from the previous
  run until helm deletes and recreates it (hook-delete-policy: before-hook-creation); a Job's spec.template is
  immutable, so the API server rejected the dry run on every upgrade of every cell, while a fresh install (CI's kind
  dry-run) passed. The rendered chart is now split (scripts/lib/split-hooks.py, standard library only): resources
  without a helm.sh/hook annotation go through apply --dry-run=server as before, hooks through
  kubectl create --dry-run=server under a unique name (<name>-dryrun-<time>), which validates them without
  touching the existing Job (dry_run_rendered in cell-provision.sh).
• cell-provision.test.sh covers the upgrade case with a fake API server that rejects an apply of an existing Job and
  accepts a create under a new name: the raw render fails, dry_run_rendered passes over the existing Job and on a
  fresh namespace, an invalid non-hook resource still fails, the splitter leaves no hook among the resources to apply.
  CI's kind step does the real thing: the four configurations dry-run on the empty namespace, then the default render
  is created for real (hook Job included) and all four are dry-run again — the same filter, so an upgrade-only
  regression cannot pass CI again.

Fix: the Helm chart mounted the login theme one level too high — Keycloak fell back to the stock theme

• The theme ConfigMap was mounted at /opt/keycloak/themes/matelot with the items named relative to login/, so the
  pod held themes/matelot/theme.properties while Keycloak resolves `themes/matelot/login/theme.properti...

Matelot v0.1.0-rc.16

No CHANGELOG section is titled ## v0.1.0-rc.16; these are the latest changes on the branch.

v0.1.0 — first self-hosted release

Images ghcr.io/matelot-cloud/{api,web,worker,runner} and the chart oci://ghcr.io/matelot-cloud/charts/secops-cell, signed, with SBOM attestations; install guide at https://matelot-cloud.github.io/website/docs/latest/install.html. The sections below are the drops that make up the release, newest first.

Observability drop 1 — instrumentation: OTLP traces (one per pipeline), Prometheus metrics with exemplars, JSON logs with trace ids (migration 0049, runner 0.3.3)

• Traces. apps/api/src/telemetry/tracing.ts, preloaded with node -r ./dist/telemetry/tracing.js by the api and
  worker images, Compose and pnpm start: the OpenTelemetry SDK with the http, express (route template only),
  pg, ioredis and undici instrumentations. Off is a no-op (one env read, no SDK module loaded) until
  OTEL_EXPORTER_OTLP_ENDPOINT is set or MATELOT_TRACING_ENABLED=true; export is asynchronous with a bounded queue
  (2048 spans), an unreachable collector costs dropped spans and one warning, never a slow request or a failed
  /health/live; SIGTERM and Nest's shutdown flush with a 5 s cap. Protocols http/protobuf (default), grpc,
  http/json; sampler from OTEL_TRACES_SAMPLER; probes, /metrics and runner heartbeats are never traced.
• One trace per pipeline. Ids are fixed at creation and stored on the rows (pipelines.trace_parent,
  jobs.trace_parent); the claim response hands the runner its job's traceparent; the runner's job run <kind> span
  (phases checkout, tool syft|trivy|semgrep, upload sbom, upload findings, correlate) goes under it and
  every call back carries the traceparent header, so the API side of a job joins the trace; the worker's gate
  evaluation runs under the policy_eval job's span; BullMQ job data carries the enqueuer's traceparent. The
  pipeline's root span and each job's span (start = created, events claimed / started, kind, status, exit code,
  runner) are emitted at their terminal status with the stored ids by whichever process finished them — nothing in
  memory across requests or replicas (PinnedIdGenerator). Verified against a real Jaeger 2.22: a security pipeline
  (checkout, syft, trivy ×2, semgrep, gate) is one trace across api, runner and worker — and the trace already
  showed something the logs never did: semgrep --version taking 105 s in a sandbox before the scan started.
• Metrics. One meter (telemetry/metrics.ts) with explicit buckets through Views (http 5 ms–10 s, pipeline jobs
  1 s–30 min, correlation 100 ms–5 min), served on port 9464 (MATELOT_METRICS_PORT, never on the ingress) by our own
  OpenMetrics serializer (telemetry/exposition.ts) because the SDK's Prometheus exporter renders no trace
  exemplars: a scraper accepting OpenMetrics gets # {trace_id,span_id} value ts per bucket, pipeline histograms
  pointing at the pipeline's trace — Grafana jumps from a slow p95 to the exact trace. exposition.spec.ts pins the
  format (names, _total, unit suffixes, buckets, matelot_build_info = 1, every label in ALLOWED_LABELS —
  an instrument given an id or a name as a label throws). The table of docs/observability.md is implemented:
  http RED by route template, pool connections, BullMQ and pipeline queue depth / age, pipeline jobs and durations
  (+ queue wait), pipelines, runners online / busy / matelot_runner_last_seen_timestamp_seconds{pool} (so
  NoRunnerOnline is an expression on the scrape), claims, correlation duration, open findings by severity and
  kind, gate evaluations, integration sync failures, webhook lag, build_info; gauges that need a query are
  refreshed by the worker on a timer (WorkerGauges), never on scrape, and their absent series vanish. Optional
  OTLP push with OTEL_METRICS_EXPORTER=prometheus,otlp. The runner serves its own page (RUNNER_METRICS_PORT):
  job durations with exemplars, jobs by status, busy, build_info.
• Logs. One line shape for api, worker and runner (telemetry/log-schema.ts, the runner's JsonLogger):
  time, level, component, cell, msg, context, trace_id, span_id, request_id, …; LOG_LEVEL=info,jobs=debug per
  module; one redaction list applied at any depth, with a test that logs every known secret key nested three levels
  and fails if a value survives. The runner's console.* is gone.
• Delivery. Env in env.ts, the api ConfigMap and compose-init (MATELOT_OTLP_* in .env); chart values
  observability.{metrics,otlp,environment} — metrics ports on api / worker / managed runners, a NetworkPolicy
  admitting scrapers from metrics.scrapeFromNamespaces, OTEL_* on every component once otlp.endpoint is set
  (headers in a Secret); chart-test renders defaults / otlp / metrics-off. make kind-jaeger TENANT=acme puts a
  Jaeger all-in-one on kind and points the cell at it. /api/v1/version carries commit and builtAt (image
  build arguments from the release workflow). The runner ships no dependency: its telemetry.ts (OTLP/HTTP JSON,
  bounded batching, exemplars, redaction) is 300 lines of its own, tested with node --test.
• Not in this drop (drop 2 onwards): ServiceMonitors / PrometheusRule / dashboards, the Compose observability
  profile, backup gauges, OTLP log export, Settings → Observability.

Design: observability milestone (M14) — docs/observability.md, ADR 0004

• The design for platform self-monitoring, OpenTelemetry-first and on the customer's stack: what every process
  emits (OTLP traces, Prometheus metrics on port 9464, JSON logs with trace_id), one trace per pipeline carried
  through the runner protocol, the metric table with its cardinality rules, the OTEL_* / MATELOT_* configuration,
  the Helm observability.* block (ServiceMonitors, PrometheusRule, dashboards ConfigMaps — all off by default) and
  the Compose observability profile, the four dashboards and twelve alerts, Settings → Observability and the
  doctor section, and the four drops with their acceptance. No code in this drop.

Fix: apply on an existing cell failed its own server-side dry run ("Job globex-realm-sync … field is immutable")

• The pre-upgrade kubectl apply --dry-run=server included the realm-sync hook Job, which exists from the previous
  run until helm deletes and recreates it (hook-delete-policy: before-hook-creation); a Job's spec.template is
  immutable, so the API server rejected the dry run on every upgrade of every cell, while a fresh install (CI's kind
  dry-run) passed. The rendered chart is now split (scripts/lib/split-hooks.py, standard library only): resources
  without a helm.sh/hook annotation go through apply --dry-run=server as before, hooks through
  kubectl create --dry-run=server under a unique name (<name>-dryrun-<time>), which validates them without
  touching the existing Job (dry_run_rendered in cell-provision.sh).
• cell-provision.test.sh covers the upgrade case with a fake API server that rejects an apply of an existing Job and
  accepts a create under a new name: the raw render fails, dry_run_rendered passes over the existing Job and on a
  fresh namespace, an invalid non-hook resource still fails, the splitter leaves no hook among the resources to apply.
  CI's kind step does the real thing: the four configurations dry-run on the empty namespace, then the default render
  is created for real (hook Job included) and all four are dry-run again — the same filter, so an upgrade-only
  regression cannot pass CI again.

Fix: the Helm chart mounted the login theme one level too high — Keycloak fell back to the stock theme

• The theme ConfigMap was mounted at /opt/keycloak/themes/matelot with the items named relative to login/, so the
  pod held themes/matelot/theme.properties while Keycloak resolves themes/matelot/login/theme.properties: one log
  line (Failed to find LOGIN theme matelot, using built-in themes), the stock page, HTTP 200. The ConfigMap keys and
  volume items now carry the login/ prefix (login__theme.properties → login/theme.properties under the same
  mount); rc.14 is harmless but unbranded, this is in the next release. Verified by projecting the rendered ConfigMap
  through the rendered volume items exactly as kubelet does onto a real Keycloak 26.0.8: themed page, assets 200, no
  log line; the rc.14 shape reproduces the report exactly.
• chart-test asserts the paths the items RESOLVE to (mountPath + item path): every file under
  files/theme/matelot lands at /opt/keycloak/themes/matelot/<path>, login/theme.properties included — not just
  that a mount exists. CI's chart job additionally runs scripts/chart-theme-kind-test.sh: the chart's theme and
  realm ConfigMaps plus one Keycloak pod with exactly the Deployment's theme volume and mount on the kind cluster, then
  test -f …/login/theme.properties in the pod, the sign-in page check, and a grep of the pod log for
  Failed to find LOGIN theme (not runnable in the drop's sandbox: no cluster; the pod-building step is).
• Compose assembles the theme correctly (theme/matelot/login/…, copied as is to /opt/keycloak/themes/matelot; the
  Keycloak service now refuses to start if login/theme.properties is not there, compose-init asserts the same, the
  unit test spells the path out). The CI smoke failure had another cause, found by replaying the exact Compose start
  sequence against a real Keycloak: besides the :80 redirect, the platform-web client requires PKCE, so a request
  without code_challenge never renders the sign-in page — Keycloak bounces to redirect_uri with
  error=invalid_request, and the check read the web app's page. login-page-check.sh now sends a PKCE challenge,
  requires the final URL to be the auth endpoint, and prints requested URL, final URL, status and body on failure;
  `smoke....

Matelot v0.1.0-rc.15

No CHANGELOG section is titled ## v0.1.0-rc.15; these are the latest changes on the branch.

v0.1.0 — first self-hosted release

Images ghcr.io/matelot-cloud/{api,web,worker,runner} and the chart oci://ghcr.io/matelot-cloud/charts/secops-cell, signed, with SBOM attestations; install guide at https://matelot-cloud.github.io/website/docs/latest/install.html. The sections below are the drops that make up the release, newest first.

Fix: the Helm chart mounted the login theme one level too high — Keycloak fell back to the stock theme

• The theme ConfigMap was mounted at /opt/keycloak/themes/matelot with the items named relative to login/, so the
  pod held themes/matelot/theme.properties while Keycloak resolves themes/matelot/login/theme.properties: one log
  line (Failed to find LOGIN theme matelot, using built-in themes), the stock page, HTTP 200. The ConfigMap keys and
  volume items now carry the login/ prefix (login__theme.properties → login/theme.properties under the same
  mount); rc.14 is harmless but unbranded, this is in the next release. Verified by projecting the rendered ConfigMap
  through the rendered volume items exactly as kubelet does onto a real Keycloak 26.0.8: themed page, assets 200, no
  log line; the rc.14 shape reproduces the report exactly.
• chart-test asserts the paths the items RESOLVE to (mountPath + item path): every file under
  files/theme/matelot lands at /opt/keycloak/themes/matelot/<path>, login/theme.properties included — not just
  that a mount exists. CI's chart job additionally runs scripts/chart-theme-kind-test.sh: the chart's theme and
  realm ConfigMaps plus one Keycloak pod with exactly the Deployment's theme volume and mount on the kind cluster, then
  test -f …/login/theme.properties in the pod, the sign-in page check, and a grep of the pod log for
  Failed to find LOGIN theme (not runnable in the drop's sandbox: no cluster; the pod-building step is).
• Compose assembles the theme correctly (theme/matelot/login/…, copied as is to /opt/keycloak/themes/matelot; the
  Keycloak service now refuses to start if login/theme.properties is not there, compose-init asserts the same, the
  unit test spells the path out). The CI smoke failure had another cause, found by replaying the exact Compose start
  sequence against a real Keycloak: besides the :80 redirect, the platform-web client requires PKCE, so a request
  without code_challenge never renders the sign-in page — Keycloak bounces to redirect_uri with
  error=invalid_request, and the check read the web app's page. login-page-check.sh now sends a PKCE challenge,
  requires the final URL to be the auth endpoint, and prints requested URL, final URL, status and body on failure;
  smoke.sh fails on Failed to find LOGIN theme in the Keycloak log before and after the page check and lists
  /opt/keycloak/themes/matelot from the container on failure.
• The tab title is server-side: loginTitle={0} \u00B7 Matelot in the theme's messages_en.properties, rendered by
  Keycloak's own template; nothing in JavaScript.

Fix: the sign-in page check in the Compose smoke test; verified against a real Keycloak 26

• The smoke step asked Keycloak for the sign-in page with redirect_uri=http://localhost:80/ while the client
  registers http://localhost/* — Keycloak answers 400 for an unregistered redirect — and its one-line assertion
  printed an empty title with nothing else to go on. The check is now deploy/compose/login-page-check.sh <base>:
  it drops a default :80 / :443 from the base, follows redirects, and on any failure prints the requested URL,
  the final URL, the HTTP status and the first 300 bytes of the body; smoke.sh calls it through Caddy and dumps
  the Keycloak log when it fails.
• The title is server-side: the theme's messages_en.properties sets loginTitle={0} \u00B7 Matelot (escaped
  middle dot, so every properties reader agrees) and Keycloak's template renders it — no script involved
  (matelot.js only sets the card heading and the icon links). Checked here against Keycloak 26.0.8 from the
  GitHub release tarball, in dev mode and in production mode (start, theme cache on), with a realm importing
  loginTheme=matelot: tab title "Globex Corporation · Matelot", heading "Sign in to Globex Corporation", every
  theme asset 200, screenshots in light and dark. The full Compose smoke (Caddy, the images) cannot run in this
  sandbox (no registry access); the check itself is what CI runs.
• apply's server-side validation uses server-side apply (--server-side --dry-run=server --force-conflicts with
  its own field manager): no more kubectl warning about the hook Job's missing last-applied-configuration; the CI
  kind dry-run does the same.

Fix: Helm upgrade rejected the theme ConfigMap (favicon.svg in both data and binaryData); API-server validation before every apply

• helm upgrade on kind failed with data[resources__img__favicon.svg]: Invalid value: duplicate of key present in binaryData: the theme ConfigMap put every img/* file in binaryData and every non-png/ico file in data, so
  the new favicon.svg landed in both — which helm template renders happily and only the API server rejects. One rule
  now: the file extension decides (png / ico / jpg / gif / woff / ttf → binaryData, everything else → data).
• Guards for what helm template cannot see. chart-test.sh asserts, for every rendered ConfigMap and Secret, that
  no key is in both data and binaryData, every key is [-._a-zA-Z0-9]+ and ≤ 253 characters, and no resource is
  rendered twice. The CI chart job now starts a kind cluster (helm/kind-action, about 40 s) and runs
  kubectl apply --dry-run=server on the rendered chart in four configurations (defaults, TLS, branded theme,
  backups), so API-server validation runs before a tag.
• cell-provision.sh apply validates the rendered chart with kubectl apply --dry-run=server against the cell's
  own cluster before touching an existing release — nothing is changed when the server rejects it — and handles a
  release an earlier apply left behind: failed with a good revision is upgraded over (globex's state on kind),
  pending-* (an interrupted helm) is rolled back to the last good revision first, and failed with no revision ever
  deployed stops with what to do (helm cannot upgrade that). Tested with a fake helm in cell-provision.test.sh.

Favicon designed for small sizes; one icon set for the app and the sign-in page, generated from the SVG

• The tab icon was the full captain's cap scaled to 16 px — a shapeless light blob on a dark tab bar. New mark:
  the anchor alone, two flat colours on a filled rounded square (apps/web/public/favicon.svg, with a
  prefers-color-scheme rule: navy square / sky anchor on light tab bars, sky square / white anchor on dark). Strokes
  3 px at 32 px; the 16 px frame is rendered with thicker strokes and without the fluke tips, not resampled.
• Shipped as favicon.svg (first in index.html), favicon.ico with 16 / 32 / 48 px frames each rendered at its own
  size, favicon-32.png, apple-touch-icon.png (180, the full cap), icon-192.png / icon-512.png with
  manifest.webmanifest (full cap). favicon.png is gone.
• tools/icons/build.mjs (pnpm icons:build, workspace package @secops/icons, sharp) regenerates every size from
  favicon.svg / logo-512.png into apps/web/public and the Keycloak login theme (chart source and compose copy);
  --check (CI, pnpm --filter @secops/icons test) fails when a committed icon differs from what the sources render
  (pixel comparison). chart-test.sh fails when the theme's favicon / apple-touch-icon differ from apps/web/public.
• The login theme links the same set: its script puts favicon.svg before the stock favicon.ico link and adds the
  apple-touch-icon; the smoke test fetches favicon.svg, favicon.ico and favicon-32.png from the sign-in page (200).
• docs/branding.md shows the rendered 16 px and 32 px frames (light and dark) — docs/branding/favicon-review.png,
  written by the build — and the rules the mark follows.

Branded sign-in page: the matelot Keycloak login theme (Helm and Compose)

• The sign-in pages rendered the stock keycloak.v2 theme: Keycloak's favicon, "Sign in to " as tab title, no
  branding. The cell now ships a matelot login theme (deploy/helm/secops-cell/files/theme, copied to
  deploy/compose/init/theme): it extends keycloak.v2 and overrides no template — only theme.properties
  (stylesheets + one script), a messages file, CSS, a small script and images — so OTP, password reset, consent and
  error pages keep Keycloak's layout and a Keycloak upgrade cannot break it. Branding: the web app's favicon
  (favicon.ico built from apps/web/public/favicon.png, plus the apple-touch-icon), the product logo above the
  form, the tenant name in the brand band and as the sign-in heading ("Sign in to Globex Corporation" — set by the
  theme's script on the sign-in page, English only; other pages and locales keep Keycloak's wording), the tab title
  " · Matelot", the app's palette in light and dark (prefers-color-scheme) through PatternFly variables, and
  the Outfit font (Google Fonts; system font offline).
• Per cell: keycloak.theme.logo, keycloak.theme.primaryColor (primaryColorHover) render css/cell.css;
  keycloak.theme.enabled: false is the stock look. Compose: MATELOT_THEME_LOGO, MATELOT_THEME_PRIMARY_COLOR.
  docs/cell-provisioning.md §Branding the sign-in page.
• Helm: one ConfigMap <release>-keycloak-theme holds every theme file (images as binaryData; keys are the
  paths with / → __, mapped back by the volume) mounted at /opt/keycloak/themes/matelot; its checksum is on the
  Keycloak pod, so a theme change restarts Keycloak. Compose: compose-init assembles th...

Matelot v0.1.0-rc.14

No CHANGELOG section is titled ## v0.1.0-rc.14; these are the latest changes on the branch.

v0.1.0 — first self-hosted release

Images ghcr.io/matelot-cloud/{api,web,worker,runner} and the chart oci://ghcr.io/matelot-cloud/charts/secops-cell, signed, with SBOM attestations; install guide at https://matelot-cloud.github.io/website/docs/latest/install.html. The sections below are the drops that make up the release, newest first.

Fix: Helm upgrade rejected the theme ConfigMap (favicon.svg in both data and binaryData); API-server validation before every apply

• helm upgrade on kind failed with data[resources__img__favicon.svg]: Invalid value: duplicate of key present in binaryData: the theme ConfigMap put every img/* file in binaryData and every non-png/ico file in data, so
  the new favicon.svg landed in both — which helm template renders happily and only the API server rejects. One rule
  now: the file extension decides (png / ico / jpg / gif / woff / ttf → binaryData, everything else → data).
• Guards for what helm template cannot see. chart-test.sh asserts, for every rendered ConfigMap and Secret, that
  no key is in both data and binaryData, every key is [-._a-zA-Z0-9]+ and ≤ 253 characters, and no resource is
  rendered twice. The CI chart job now starts a kind cluster (helm/kind-action, about 40 s) and runs
  kubectl apply --dry-run=server on the rendered chart in four configurations (defaults, TLS, branded theme,
  backups), so API-server validation runs before a tag.
• cell-provision.sh apply validates the rendered chart with kubectl apply --dry-run=server against the cell's
  own cluster before touching an existing release — nothing is changed when the server rejects it — and handles a
  release an earlier apply left behind: failed with a good revision is upgraded over (globex's state on kind),
  pending-* (an interrupted helm) is rolled back to the last good revision first, and failed with no revision ever
  deployed stops with what to do (helm cannot upgrade that). Tested with a fake helm in cell-provision.test.sh.

Favicon designed for small sizes; one icon set for the app and the sign-in page, generated from the SVG

• The tab icon was the full captain's cap scaled to 16 px — a shapeless light blob on a dark tab bar. New mark:
  the anchor alone, two flat colours on a filled rounded square (apps/web/public/favicon.svg, with a
  prefers-color-scheme rule: navy square / sky anchor on light tab bars, sky square / white anchor on dark). Strokes
  3 px at 32 px; the 16 px frame is rendered with thicker strokes and without the fluke tips, not resampled.
• Shipped as favicon.svg (first in index.html), favicon.ico with 16 / 32 / 48 px frames each rendered at its own
  size, favicon-32.png, apple-touch-icon.png (180, the full cap), icon-192.png / icon-512.png with
  manifest.webmanifest (full cap). favicon.png is gone.
• tools/icons/build.mjs (pnpm icons:build, workspace package @secops/icons, sharp) regenerates every size from
  favicon.svg / logo-512.png into apps/web/public and the Keycloak login theme (chart source and compose copy);
  --check (CI, pnpm --filter @secops/icons test) fails when a committed icon differs from what the sources render
  (pixel comparison). chart-test.sh fails when the theme's favicon / apple-touch-icon differ from apps/web/public.
• The login theme links the same set: its script puts favicon.svg before the stock favicon.ico link and adds the
  apple-touch-icon; the smoke test fetches favicon.svg, favicon.ico and favicon-32.png from the sign-in page (200).
• docs/branding.md shows the rendered 16 px and 32 px frames (light and dark) — docs/branding/favicon-review.png,
  written by the build — and the rules the mark follows.

Branded sign-in page: the matelot Keycloak login theme (Helm and Compose)

• The sign-in pages rendered the stock keycloak.v2 theme: Keycloak's favicon, "Sign in to " as tab title, no
  branding. The cell now ships a matelot login theme (deploy/helm/secops-cell/files/theme, copied to
  deploy/compose/init/theme): it extends keycloak.v2 and overrides no template — only theme.properties
  (stylesheets + one script), a messages file, CSS, a small script and images — so OTP, password reset, consent and
  error pages keep Keycloak's layout and a Keycloak upgrade cannot break it. Branding: the web app's favicon
  (favicon.ico built from apps/web/public/favicon.png, plus the apple-touch-icon), the product logo above the
  form, the tenant name in the brand band and as the sign-in heading ("Sign in to Globex Corporation" — set by the
  theme's script on the sign-in page, English only; other pages and locales keep Keycloak's wording), the tab title
  " · Matelot", the app's palette in light and dark (prefers-color-scheme) through PatternFly variables, and
  the Outfit font (Google Fonts; system font offline).
• Per cell: keycloak.theme.logo, keycloak.theme.primaryColor (primaryColorHover) render css/cell.css;
  keycloak.theme.enabled: false is the stock look. Compose: MATELOT_THEME_LOGO, MATELOT_THEME_PRIMARY_COLOR.
  docs/cell-provisioning.md §Branding the sign-in page.
• Helm: one ConfigMap <release>-keycloak-theme holds every theme file (images as binaryData; keys are the
  paths with / → __, mapped back by the volume) mounted at /opt/keycloak/themes/matelot; its checksum is on the
  Keycloak pod, so a theme change restarts Keycloak. Compose: compose-init assembles the theme plus cell.css
  into the config volume and the keycloak service copies it into /opt/keycloak/themes at start.
• realm-sync applies loginTheme (and accountTheme / emailTheme when a realm.json carries them) on every run,
  like the URLs. The realm display name is now the plain tenant name (no "Security Platform" / "· Matelot"
  suffix) with displayNameHtml the same; realm-sync migrates the old templated forms once and keeps a display name
  typed by hand.
• Tests: realm-sync.test.js (themes applied, stale names migrated, hand-typed names kept, pure realmSettingsPatch
  cases); compose-init.test.sh (theme assembled, cell.css empty by default and rendered from the variables);
  chart-test.sh (every theme file in the ConfigMap, images byte-identical, the volume maps every key, mount path,
  checksum, realm selects the theme, theme.enabled=false is stock, compose copy identical); the Compose smoke test
  loads the realm's sign-in page and checks the "· Matelot" title and that favicon.ico, apple-touch-icon.png,
  logo.png, matelot.css, cell.css and matelot.js are served (200). The web app's jsdom tests are untouched.
• Not verified here: the page in a real Keycloak 26 (the chart render was checked with a stand-in renderer and the
  CI chart job runs the real helm; the smoke step runs in CI's compose job).

Fix: cell-provision checks declared the cell unhealthy in the endpoint-switch window after a rollout

• Right after "all workloads rolled out", the in-cluster probe (GET http://<release>-api/api/v1/health/ready from a
  curl pod) made one request, hit the moment the Service endpoints switched to the new pod, printed "→ no answer:"
  with nothing after it and exited 1 — while matelot doctor through the ingress answered seconds later. The probe
  pod now retries with backoff (2, 4, 8, 10, 10 … s) for up to READY_TIMEOUT seconds (default 60), the api line
  says how long readiness took ("→ 200 (ready in 1s)", or "→ 200 after 9s and 3 attempts (the first 2 got no answer:
  the Service endpoints were still switching)" with the attempts listed), and only after the retries is the cell
  unhealthy — then with the last HTTP status and the start of the error / body, never an empty "no answer:".
  cell-provision.test.sh runs the probe loop with a fake curl (refuses twice then 200; immediate 200; never) and
  checks the report lines.

Fix: service summary counted no vulnerabilities for an undeployed service; SAST fingerprints; CLI times with offset

• GET /services/:id/summary reported "0 open" next to a gate line saying "1 open critical": it filtered on
  findings.running, which is false for every finding of a service that has no deployment yet. It now counts what the
  gate counts — the open findings linked to the newest SBOM, one per (advisory, source package) — and returns that
  SBOM's id. Spec (service-summary.spec.ts, database): a service with no deployment, two open findings of which
  one is in the newest SBOM → open.total = 1, kev = 1.
• SAST: one finding per issue across runs. Ingest already matched by fingerprint and closed what a scan no longer
  reports (the spec proves it: three ingests with the same fingerprint → 1 open; a changed fingerprint → 1 open,
  1 fixed). What can vary per run is the fingerprint's path when semgrep prints the checkout under another prefix
  (symlinked temp dir) or relative to the runner's cwd (../job-abc123/repo/app/run.py): the per-job workspace then
  lands in the fingerprint. The runner now reduces every form to the repository path (relativeToRepo), folds
  whitespace in the matched lines, and a test fixes the fingerprint across five path forms. The summary also lists
  the open code findings per rule, and matelot service show prints them — "3 code (eval-detected ×3)" is the
  same issue re-opened per run, "3 code (eval-detected, dangerous-eval, …)" is three rules on one line — so the next
  run on kind tells which it was; after it, 1 open and 2 fixed is the expected state.
• CLI 2.24.1: timestamps in service show and in every table are local time with the UTC offset
  ("2026-10-18 10:52+03:00", "…07:52Z"), never bare UTC that reads as local.

From the end-to-end pipeline test: connect with a write token, explicit policy rules, service summary, job names, secret dete...

Matelot v0.1.0-rc.13

No CHANGELOG section is titled ## v0.1.0-rc.13; these are the latest changes on the branch.

v0.1.0 — first self-hosted release

Images ghcr.io/matelot-cloud/{api,web,worker,runner} and the chart oci://ghcr.io/matelot-cloud/charts/secops-cell, signed, with SBOM attestations; install guide at https://matelot-cloud.github.io/website/docs/latest/install.html. The sections below are the drops that make up the release, newest first.

Fix: service summary counted no vulnerabilities for an undeployed service; SAST fingerprints; CLI times with offset

• GET /services/:id/summary reported "0 open" next to a gate line saying "1 open critical": it filtered on
  findings.running, which is false for every finding of a service that has no deployment yet. It now counts what the
  gate counts — the open findings linked to the newest SBOM, one per (advisory, source package) — and returns that
  SBOM's id. Spec (service-summary.spec.ts, database): a service with no deployment, two open findings of which
  one is in the newest SBOM → open.total = 1, kev = 1.
• SAST: one finding per issue across runs. Ingest already matched by fingerprint and closed what a scan no longer
  reports (the spec proves it: three ingests with the same fingerprint → 1 open; a changed fingerprint → 1 open,
  1 fixed). What can vary per run is the fingerprint's path when semgrep prints the checkout under another prefix
  (symlinked temp dir) or relative to the runner's cwd (../job-abc123/repo/app/run.py): the per-job workspace then
  lands in the fingerprint. The runner now reduces every form to the repository path (relativeToRepo), folds
  whitespace in the matched lines, and a test fixes the fingerprint across five path forms. The summary also lists
  the open code findings per rule, and matelot service show prints them — "3 code (eval-detected ×3)" is the
  same issue re-opened per run, "3 code (eval-detected, dangerous-eval, …)" is three rules on one line — so the next
  run on kind tells which it was; after it, 1 open and 2 fixed is the expected state.
• CLI 2.24.1: timestamps in service show and in every table are local time with the UTC offset
  ("2026-10-18 10:52+03:00", "…07:52Z"), never bare UTC that reads as local.

From the end-to-end pipeline test: connect with a write token, explicit policy rules, service summary, job names, secret detectors

• integration:connect is day-to-day work. It left the sudo-only set: a developer's / team lead's write token
  (or a service account) connects a git host or registry to a project, team or workspace. Creating an
  organization-wide connection additionally needs organization:write — an owner, sudo or the web session.
  Disconnecting (forgetting credentials), rotating credentials, on-demand sync and deletion stay integration:write
  (sudo). The token-scope matrix spec, the consent page, the MCP guidance and docs/roles.md say so.
• Policy rules are explicit. matelot policy create sends all seven rules: a block rule not asked for is off
  (--max-severity high --block-kev --action block --enforce no longer produces a block_secrets violation),
  --honour-decisions defaults to on; every boolean flag has a --no-… form; policy update sends only the flags
  given. The API's create fills a missing boolean rule with off too (CREATE_DEFAULTS); rows stored before keep
  reading missing rules as the old defaults so an existing policy blocks what it blocked. policies list /
  in-force show the rules in one column ("max high · kev · secrets · sbom").
• GET /services/:id/summary (vulnerability:read): the service with its latest pipeline and jobs, open
  findings by severity (+ KEV, running ones), open code findings by kind and the newest SBOM — one call.
  CLI 2.24.0: matelot service show reads it and prints five lines (pipeline #, status, ref, jobs with the error
  of a failed one; open findings; code findings; latest SBOM); --json for scripts.
• docs/ci-integration.md: the pipeline jobs table uses the names the API and CLI report (checkout, sbom,
  dependencies, code, secrets, infrastructure, evaluate) with the job type beside them, and a new section lists what
  the secrets job detects — checked with trivy 0.74.0 (the runner image's version) and the runner's exact command on
  a planted repository: the AWS documentation example pair is on trivy's allow list, the AWS secret key fires only
  with its keyword on the line, ghp_ needs exactly 36 characters, and test(s)/, example(s)/, vendor/,
  node_modules/ and *.md are skipped. With random-looking values all three planted secrets are reported.

Fix: cell-provision apply waited for "N pods ready", not for the rollout

• One second after helm upgrade returned the wait printed "all workloads ready (pods 7/8 ready) · waiting:
  globex-api-… (Running 0/1)" and checks ran against the old API pod. cellwait.py tested readyReplicas ≥ n and
  updatedReplicas ≥ n separately: during a rolling update the old pod is the ready one and the new pod the
  updated one, so both passed — a broken new image would have passed the checks. "Done" is now what
  kubectl rollout status means, per Deployment / StatefulSet: observedGeneration ≥ generation, updatedReplicas,
  readyReplicas and availableReplicas all ≥ spec, unavailableReplicas == 0, and no surplus replica
  (status.replicas ≤ updated — the old pod is gone); StatefulSets also need currentRevision == updateRevision.
  Terminating pods (deletionTimestamp) no longer count. The summary reads "N/M workloads rolled out, pods a/b ready",
  and a workload whose pods all look fine but is not rolled out says why ("controller has not observed the new
  generation yet", "1 old replica(s) still terminating", "revision … rolling out").
• Tests (cellwait_test.py): mid-rollout (replicas 2 / updated 1 / ready 1 → not done), generation not observed
  yet → not done, new pod ready but old one terminating → not done, StatefulSet update revision not current → not done.

Header pickers are our own popover, not native selects; Team level in the switcher

• The Workspace / Team / Project / Service pills (and Organization, Refresh) open a TailAdmin-style panel under the
  pill — same surface, border, radius and shadow as the user menu, dark-mode aware — instead of a native <select>,
  which Chrome painted with the OS look (white list, blue highlight) over the dark theme. Items show the name
  (ellipsis, full name on hover) and the slug underneath; the current one carries a check; "All workspaces / All
  projects" (or "Whole workspace / Whole team" inside a scope) sits first above a divider; a type-to-filter field
  appears from 7 items. Keyboard: arrows, Home / End, Enter / Space, Escape (focus returns to the pill), Tab; click
  outside closes. ARIA: combobox + aria-expanded / aria-activedescendant, listbox / option /
  aria-selected. One component, LocationPicker in platform/header/ContextSwitcher.tsx, for every header picker;
  no native select is left in the header (the render test asserts it).
• Team in the switcher at every level: Workspace › Team › Project (› Service inside a project), each list scoped
  to its parent — teams of the workspace, projects of the team, or of the workspace when no team is chosen.
• Native controls that remain on pages (list filters, date fields, scrollbars) follow the theme through
  color-scheme: dark on the dark root, so they no longer render with the OS light look.
• Tests: the render test opens the picker, checks roles, the divider, the check mark, Escape and focus return, a
  keyboard pick (ArrowDown → End → Enter navigates), the team list scoped to the workspace, a mouse pick into the team,
  and click-outside. Checked in Chromium in light and dark mode at 1280 / 1440 px with the filter field.

Navigation: you always know where you are (breadcrumb, scope card, titles, location switcher, team pages)

• Breadcrumb on every page, full path and clickable: Organization › Workspace › Team › Project › (Service) ›
  Page, e.g. "Globex › Platform › Payments team › checkout-api › Members"; each segment opens that scope's overview,
  detail pages add their list ("… › Pipelines › Pipeline"). Middle segments end in an ellipsis with the full name on
  hover; on a phone the trail wraps.
• Headings and tab titles name the scope: "Members — checkout-api (project)" and "Members · checkout-api · Globex".
  The old per-page headings and the workspace / project header blocks are gone (one source: useScopeLocation in
  platform/navigation/scope.tsx).
• Sidebar scope card (initial, name, "Organization" / "Workspace" / "Team" / "Project", criticality on projects)
  with "← Back to " above it; the menu below is that scope's menu.
• Header location switcher shows where you are — "Workspace: Platform ▾ › Team: Payments team ▾ › Project: … ▾",
  inside a project "Workspace ▾ › Project ▾ › Service ▾" — as pills that navigate (the URL changes); "All workspaces /
  All projects" only on organization-wide pages. Visibly different from the filter dropdowns on list pages.
• Teams are a level of their own: Workspace → Teams (list, New team), team pages Overview (its projects, New
  project), Members and Settings (rename, archive) under /workspaces/:id/teams/:teamId, in the breadcrumb and the
  switcher. "Teams & projects" is now "Projects".
• First run: creating a workspace lands inside it with "Next step: Create a team"; creating a team lands inside it
  with "Next step: Create the first project of this team"; creating a project opens it.
• Long names: the product name in the sidebar ("Globex Corporation Security Platform") ends in an ellipsis on
  one line with the full name on hover (it ran off the sidebar edge); the same for the scope card, breadcrumb and
  switcher. The header no longer pushes the page wider ...