release: 0.3.0: production hardening and checks that prove the data paths - #21
Merged
Conversation
…tunnel and access
- exemplar destination trace_id -> traceID, matching what Tempo emits
- HighErrorRate filters span_kind=SPAN_KIND_SERVER on both sides; internal
ASGI child spans inflated the denominator ~3x
- demo load hits /work twice so the demonstrated error rate (~10%) actually
crosses the 5% threshold instead of sitting on it
- infrastructure-logs dashboard queries service_name (the label Loki actually
indexes); drop the env variable no templating query can populate
- storage-s3 overlay re-declares commands with -config.expand-env=true, or the
documented activation reads the \${S3_*} placeholders literally
- drop tempo datasource search.hide, restates the default
- file-backed exporter queues on a new otel_queue volume; buffered telemetry survives collector restarts, retry/queue settings pinned to the documented five-minute window - mem_limit on every service, sized from observed usage; collector memory_limiter switched to absolute values tied to its container limit - prometheus healthcheck with service_healthy ordering for collector and grafana (loki/tempo are distroless, no probe binary) - scrape grafana, loki, tempo, and alertmanager so TargetDown covers them - loki indexes only service.name; service.instance.id becomes structured metadata instead of minting a stream per sender restart - tempo metrics-generator capped at 50k active series - grafana 13.1.4 (CVE-2026-13438, CVE-2026-17183), loki 3.7.6, tempo 3.0.3, prometheus v3.13.2, node-exporter v1.12.1, cloudflared 2026.8.2 - secure cookies behind GRAFANA_COOKIE_SECURE (up-tunnel enforces it); opt-in per-user Grafana login via Cloudflare Access JWT - no-new-privileges on all services; lint containers run --network none; infra-validate keeps tfstate/tfvars out of the container - up-tunnel also guards GRAFANA_ROOT_URL and warns on empty HEARTBEAT_URL; logs/ps/tail see demo services; loki/tempo config checks join just check; smoke asserts every dashboard provisioned
- run just infra-validate; tofu fmt alone let invalid resources merge green - build the demo image so a broken demo app fails CI - pin actions/checkout and setup-just to commit SHAs
…config - delete relab-api.json; the \$service pickers on Service Health and Logs Overview already cover it and scale to future services - delete demo/__init__.py, never copied into the image or imported - allowUiUpdates: false, the :ro dashboards mount made UI saves fail anyway; drop provisioning keys that restate defaults
- fastapi 0.141.1, uvicorn 0.52.4, otel 1.44.0/0.65b0, uv 0.12.1 - python:3.14.7-slim in both stages for reproducible builds - run as nobody
- README: prerequisites, accurate volume list and just-check coverage, dashboard JSON editing workflow, up-tunnel guard description - RUNBOOK: durable queue and just-up volume init, memory ceilings, JWT access setup with the aud-pinning caveat, manually-tracked tool images - ONBOARDING: loki's new label surface (service_name only) - CHANGELOG: Unreleased entry covering the six weeks since 0.2.0 and this batch
- add ADR 0002: hub-and-spoke observability for CML projects, superseding ADR 0001's span-metrics RED decision - move the August 2026 review handover in from the Relab repo as docs/HANDOVER.md, with an owner/review-date tripwire - point the README at both
Verified against the running stack with Relab reporting. - key the rule on project/env as ordinary series labels: Alloy attaches the identity labels to the series, not as OTLP resource attributes, so target_info carries none of them and the previous expression could never match - correct the environment to staging; no production series exist - record the live findings in the handover: cAdvisor sends only the root cgroup, so ContainerRestarting matches nothing and the crash-loop incident is still undetected; 30% of OTLP metric writes were being rejected as out-of-order until the window this branch carries was actually loaded
- remove the generator, its remote-write path and the cardinality backstop - drop Prometheus' remote-write receiver, which existed only for it - unwire tracesToMetrics/serviceMap/nodeGraph rather than leave an empty service graph reading as "no traffic"
- provision rules, contact points and the routing tree from config/grafana/alerting/; one engine that can query Loki as well - drop Alertmanager, its url_file entrypoint hack and its volume - add AlertDeliveryFailing, and make ALERT_WEBHOOK_URL a hard guard in up-tunnel: empty delivery fails silently while the heartbeat stays green
Template variables are stream selectors and label_values() cannot see structured metadata. All four are bounded, so indexing them costs nothing.
- replace infrastructure-logs and logs-overview with one logs dashboard - rework service-health and stack-health onto the OTLP-native series
- one Alloy agent per host: container logs, host metrics, cAdvisor, GPU - bootstrap renders the keystone rule, regenerates the coverage backstop, creates healthchecks.io checks and prints the .env and vendoring curls - replace the ONBOARDING templates that pushed straight to Loki, which needed Loki exposed without authentication
- mark ADR 0002 complete and release the spoke-watchdog tripwire - delete HANDOVER.md: it described a transition, not a system - point onboarding at templates/README.md
…ed rule count - config/alerts/ went with Alertmanager, but Prometheus and the promtool validator still bind-mounted it, so Docker kept recreating it root-owned and promtool checked a config with no rules - assert in `just smoke` that as many alert rules provisioned as there are on disk; a malformed file provisions none of its own group, so a non-zero check would miss it
A project bootstrapped for one environment read as covered while its other environments had no keystone rule — the same silent gap the rule exists to close, one level in. - read the pair back from a COVERS marker in each rendered file rather than splitting the filename, which is ambiguous when either half has a dash - match with `unless on (project, env)` over the rendered pairs - correct the template header: rendered flat, and the file is coverage.yaml
- nothing detects an empty ALERT_WEBHOOK_URL: AlertDeliveryFailing routes to that same webhook and the heartbeat stays green, so the up-tunnel guard is the only control - ONBOARDING no longer carries the Loki-driver and Alloy templates - move the Cloudflare seat-count caveat to the access bullet it qualifies
- overlays are host config: every recipe (up, down, logs, ps, backup) acts on the same COMPOSE_FILE set, so tunnel services are visible to logs/ps on the hosts that run them - drop 'just up-tunnel': 'just up' runs the exposure guards automatically whenever the tunnel overlay is active - document the mechanism in .env.example, the README, and the runbook"
- render the ProjectsUncovered rule from templates/alerting/coverage.yaml.tmpl instead of a shell heredoc, and re-render every covered pair on each bootstrap run so template fixes propagate to all projects - collapse the identical FIRING threshold node (rules.yaml) and the exporter queue/retry blocks (otel-collector.yaml) into YAML anchors - share the compose file set and dashboard paths via justfile variables, run 'docker compose config --images' once in check, and verify dashboards in smoke with one search call instead of per-uid polling loops - scope the Service Health logs panel to the selected service and use the indexed label_values(project) in the logs dashboard - x-logging anchor in the telemetry overlay, ping_fail() helper in run_scheduled.sh, HC_JOBS override for the healthchecks job list - fix docs contradicting the config: Loki's four identity index labels, Tempo's removed metrics generator, the Alertmanager removal, and the now centrally provisioned GPU/container dashboards"
- drop the JWK-set fallback domain: GRAFANA_JWT_AUTH now requires CF_ACCESS_TEAM_DOMAIN and CF_ACCESS_AUD, with the aud pinned via GF_AUTH_JWT_EXPECT_CLAIMS and exported from infra as grafana_access_aud - pass the tunnel token to cloudflared via TUNNEL_TOKEN env, not argv - cap_drop ALL on every service; node-exporter read-only with a pids limit - digest-pin all hub images, matching the client templates - bootstrap: healthchecks API key via curl config-from-stdin, vendoring refuses a tag without templates/ and prints sha256sum -c lines - grafana healthcheck so compose --wait has something to block on
- aggregating by job alone let a healthy prod service dilute a broken staging one sharing the job name below the 5% threshold - HostDiskSpaceLow names the host via host_name or instance - wrap long exprs and descriptions as folded scalars
- mirrors the hub collector: a long hub outage now sheds load instead of OOM-killing the agent together with its drop counters - shorten the OTLP endpoint guard message
…use readable secret files - smoke posts through curl's stdin config; busybox wget put the token in argv - _expose-guards fails when .env or the tofu state is group/other readable - the prek gitleaks hook mounts the repo read-only like just lint
TargetDown, HostDiskSpaceLow and HostDiskFilling returned 0 or a negative value on the state they watch; the shared threshold node fires on > 0.
…E_ALERTS - opt-in locally, on in CI: costs the rule's for plus one evaluation - poll() takes POLL_MAX for the longer wait
- set -o pipefail: tar's status was lost behind gzip and a partial archive passed - restore-check exports COMPOSE_FILE so the host's overlays stay out
The readback loop folded a 401 into 'rule not provisioned'. Fail fast on an empty password, die on 401/403, escape the password for curl's config.
…es at 100 lookup_dns_record died inside a command substitution, so an API error still read as 'no record'.
- Fold the repeated hardening, logging and healthcheck blocks into `x-hardened` and `x-healthcheck` anchors; services override after the merge. - Give every alert rule the same `rule_defaults` anchor, and move the capacity group off a 1m interval: nothing there fires in under 15m. - Stamp `department` as a literal in the collector config. One value has ever been correct, and as an indexed Loki stream label a second one would be an index dimension, not a filter; the `DEPARTMENT` variable is gone from `.env`. - Pull the spoke's dummy environment into `spoke_env`, read the Alloy image from the template that pins it, and only pull lint images that are missing. - `bootstrap.sh` reads single keys out of `.env` through one helper and prints the vendoring curls from the same list it hashes. - CI copies `.env.example` with mode 600, or the exposure guards reject it.
- Drop histogram buckets from the Grafana, Loki and Tempo self-scrapes. Of those three jobs' 8,045 series, one metric is read anywhere in this repo; the buckets were 5,264 of them. `_sum` and `_count` stay, so latency is still there to explore. - Run node-exporter with the same nine-collector allowlist the spokes' Alloy uses. The defaults added ~1,000 series of cpufreq, thermal and cooling data that no dashboard, alert or runbook reads. Measured on the hub with one spoke: 14.1k active series to 7.0k, 438 samples/s to 206.
- Keep the four cAdvisor metrics two dashboard panels and the two container-lifecycle alerts use. cAdvisor emits ~59 metric families per container, and per-container-per-host is the fastest-growing block in the series budget. - Drop Alloy's own histogram buckets: 345 of that job's 760 series, none of them read centrally. - Poll Docker every 15s, so a short-lived one-shot is discovered without hitting the socket proxy at the old rate.
The keystone alert and the coverage backstop asked "is this project sending
anything?" by selecting every project-labelled series in the TSDB: a scan that
grows with the fleet, and one that a logs-only or traces-only project is
invisible to, because it reaches no Prometheus series at all.
- Count what arrives at the collector, keyed on the project and env resource
attributes, and emit telemetry_{datapoints,logs,spans,metrics}_total.
- Point ProjectTelemetrySilent and ProjectsUncovered at those counters.
- Convert the connector's delta sums to cumulative: the Prometheus OTLP
receiver refuses delta ("invalid temporality and type combination"), which
would have failed silently at exactly the layer meant to catch silence.
- Count a sender that set no project as "unknown", so an unattributed sender
raises ProjectsUncovered instead of vanishing.
- Assert the counters in `just smoke`, since two alerts now key on the name.
- Show ingest per project on Stack Health.
Both onboarding templates and the demo set env and never project, so an app onboarded by following them shipped telemetry no alert or dashboard could attribute. With the gateway's ingest counters such a sender is now counted as "unknown" and raises ProjectsUncovered, but the docs are where it should not happen in the first place.
- Changelog entries for the counters, the halved series count and the onboarding fix, and drop the DEPARTMENT upgrade step it no longer has. - Re-base the cardinality prose on the measured ~7k baseline: the 100k ceiling is ~14x it now, not ~7x. - Verify a spoke with the ingest counter, which sees every signal.
The printed check still selected every series a project sends, which is both the scan the alert rules moved off and blind to a project that ships only logs or traces. templates/README.md already says the counter.
CF_ACCESS_TEAM_DOMAIN was the one Access value copied by hand from the dashboard, and Grafana builds its JWK set URL from it: unset, it fetches signing keys from a subdomain anyone could claim, which is why the exposure guards refuse the pair. The team name is account-wide and predates this config, so read it through a data source and trim the suffix Grafana appends.
The old comment argued a second department could not be a stream label, which is what the label already is. What the literal actually buys is that it cannot be misconfigured per deploy, and the trigger for a variable is a second hub, not federation: a second department federating into this hub means the upsert has to go, since it would stamp our department onto their data.
A host without the NVIDIA container runtime cannot include the GPU overlay at all, so tooling that assembles its `-f` list from the host's .env needs a switch, and every spoke inventing its own name makes one runbook per spoke. None of the vendored files read it; say so where the name is given.
…lector - validate reuses _image instead of an inline copy of the jq lookup - telemetry.spanevents counts one series, nothing reads it per project - rules, bootstrap and smoke match telemetry_.+_total in one form
- move the queue chown into a one-shot otel-queue-init service, so a plain `docker compose up` works and the volume is created labelled. Pre-creating it with `docker volume create` left it unlabelled, and `down --volumes` then never reclaimed it - refuse to restore when a stateful volume is missing: `docker run -v` would create it empty, making the pre-restore snapshot a tarball of nothing - drop the dead Alertmanager cleanup step and the stale "use just up" warning from the runbook, and note that the alpine pin now lives in two places
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production hardening of the hub and the checks that prove it. Full notes in
CHANGELOG.md.
Breaking, act on merge:
OTEL_EXPORTER_OTLP_ENDPOINTatotel.<domain>.otlp.is gone.COMPOSE_FILEin the hub's.env.just up-tunnelis gone.DEPARTMENTis gone too: the collector stampscml.projectas well asenvin every sender'sOTEL_RESOURCE_ATTRIBUTES. A sender missing either is counted asunknownand raisesProjectsUncovered.templates/on each spoke atv0.3.0.Highlights:
docker compose upworks. Per-project ingest counters at the gateway drive the silence and coverage alerts, so a logs-only or traces-only project is covered. Prometheus scrapes every service. Disk-fill-rate and cardinality alerts.just smokeboots the production shape and asserts provisioning, JWT, scrape targets, and a metric and log round-trip with the identity labels.just checkislintplusvalidate. CI runs on every PR, in two jobs. prek hooks.no-new-privileges, memory and pids limits, backends on an internal network, digest-pinned images.SMOKE_ALERTS=1(on in CI) it stops a scrape target and provesTargetDownfires. That check caught three threshold rules that could never fire (TargetDown,HostDiskSpaceLow,HostDiskFilling); fixed withboolCI on this PR is the first run of the new workflow.