v0.6.0
Open at both ends. Until now the deal was OTLP, from one cluster: you
re-pointed every sender before you could try avuru-obs at all, and a second
cluster meant a second instance with its own store, UI and login. v0.6 opens
both ends. The gateway now speaks Jaeger, Zipkin, Prometheus remote-write
and Loki push beside OTLP — one values flag each, every one default off, and
all of them through the same tenant stage, so per-project ingest keys are
enforced whatever protocol the data arrived on. It can also dual-write
what it ingests to the backend you run today, which makes adopting this a
reversible decision rather than a migration. At the other end, one project can
now span several clusters: a secondary cluster installs the ingest half of
the chart alone against the central store, and every screen answers for the
union. Projects picked up the operational parts they were missing on the way —
a retention window of their own, and their own storage usage beside the
install's — and the compatibility claim is a CI gate now, not a sentence in a
README.
Added
-
One project can now span several clusters. A UI-managed project gains
member projects: pick the projects it aggregates in Settings → General, and
every screen — services, map, traces, logs, metrics, errors, profiles, green,
alert history — answers for the union instead of a single tenant. Nothing
about the members changes: they stay separately queryable and separately
granted, and each viewer sees only the members they already had access to, so
an aggregate is a convenience over existing permissions and never a way
around them. Membership is one level deep (an aggregate cannot contain
another) and it is a read-time view, so the writes that need one tenant —
error triage, ingest keys, profile ingest — are refused on an aggregate with
a message naming the member to use instead. A membership change applies on
the replica that made it immediately and reaches the others within thirty
seconds. Members need not exist yet: an id can be added before its cluster
ships its first span. -
Which nodes the energy numbers actually come from.
/greencounted its
nodes — known, measured, estimated, absent — but never said which node spent
what, so a fleet where one node reports and eleven do not looked the same as
a fleet where all twelve do. The coverage panel now carries a per-node table:
one row per known node with its Wh, the measured/estimated split kept visible
per row, and nodes reporting nothing listed at 0 Wh instead of omitted. The
node names ride the coverage query that was already running, so the table can
never disagree with the counts above it, and the summary still costs the
screen a single request. -
A carbon budget now tells you whether it can actually reach anyone. A
budget with no notification channel, or one pointing at a channel since
deleted, used to render exactly like a wired one — the footnote only checked
whether the alerting module was on. Each budget now resolves its own
deliverability (ok, alerting off, no channel, unknown channel) against the
same channel set the evaluator walks, so the screen cannot promise a delivery
the tick would drop, and every state names its own fix. -
Budgets aimed at a group that does not exist say so. A budget whose
service-health group nothing rolls up to evaluates forever at zero and can
never fire, which on a dashboard is indistinguishable from a quiet month. The
budgets response now returns warnings for those, and the evaluator logs the
same finding once an hour per budget. "Known" means the configured groups
(chart and UI) plus the groups services actually landed in, so namespace
auto-groups stay legitimate targets and a zero-config install never warns
about its own working budgets. -
A runbook for validating green against real RAPL hardware.
docs/runbooks/green-rapl-validation.md
is the procedure the two green AEPs were missing: confirm a node exposes
RAPL, pin the sensor to it, then check every hop — Kepler's endpoint, the
metric names and their labels, measured-quality rows in ClickHouse, the API,
the screen — because a silent upstream rename is the likeliest failure and it
fails by showing nothing. Stage 5 measures the estimator's error band against
Kepler on the same node, so the ±30-50% figure can be replaced with an
observed one. -
Bring the senders you already run. The drop-in promise stopped at OTLP:
a fleet with Jaeger, Zipkin, Prometheus remote-write or Loki senders had to
re-point or re-instrument them before it could try avuru-obs at all. The
gateway now speaks all four natively — Jaeger gRPC (:14250) and
thrift-HTTP (:14268), Zipkin (:9411), Prometheus remote-write
(:9291), Loki push (:3100) — each behind its own values flag, every one
default off, so an install that wants none renders byte-identically to
before. They are not a side door: every enabled receiver joins the same
tenant stage as OTLP, so per-project ingest keys are enforced the same
way whatever protocol the data arrived on. Loki and remote-write also
respect their signal's module, so enabling a receiver for a signal you do
not store is silently a no-op rather than a surprise. Two deliberate
limits, both documented rather than hidden: Jaeger UDP/thrift is not
offered (it has no authentication hook, and jaeger-agent is deprecated
upstream), and the pinned remote-write receiver is protocol v2 only — a v1
sender is refused with415instead of dropping data quietly. -
Leave the same way you came in. Adopting a backend is a decision people
want to reverse, and evaluating one usually means running two at once.
gateway.forward.otlpandgateway.forward.kafkadual-write what the
gateway ingests to a second destination — your existing backend during a
migration, or a Kafka topic someone else owns — so avuru-obs can sit
alongside what you run today instead of replacing it on day one. The
forwarders always carry a bounded sending queue: a legacy target that goes
down cannot backpressure the ClickHouse path, which is the failure that
makes people distrust dual-write. Kafka SASL credentials come only from an
existing Secret, never inline, so they never land in a ConfigMap. -
The compatibility claim is tested, not asserted.
make e2e-compat
(opt-in, compose) and the kind Helm gate each send a real fixture per
protocol — a genuine Jaeger gRPC batch, Zipkin JSON, snappy-framed
remote-write, a Loki push — through the chart-rendered receivers, then
assert the rows in ClickHouse and the forwarded trace arriving at a
stand-in legacy backend.tools/compatsendis the sender, usable by hand
against any install when you want to check a protocol before committing to
it. -
A project can keep less than the install does. Retention was one number
for the whole install, so a noisy staging tenant held thirty days of traces
because production needed to — the only way out was a second deployment. Any
UI-managed project can now be given a shorter window of its own in Settings →
General, and a background sweep trims that tenant hourly, scoped by project.
It cannot be a ClickHouse TTL: the telemetry tables are shared, and a TTL
expression cannot select the rows of one tenant — so the sweep issues bounded
lightweight mutations instead, skipping a table with a trim still running and
costing one indexed lookup per table once there is nothing left to delete. A
window LONGER than the install-wide one is refused rather than accepted and
quietly ignored: the shared table TTL would drop those rows first whatever the
project asked for. Aggregates are refused too — they own no rows, so a window
there would silently keep everything. -
What one project holds, not just what the install holds. Settings →
Storage answered "how much data is there" for the whole install, which on a
shared instance is the wrong question: nobody could tell whether staging or
production was the reason the disk filled, or whether a project still ships
data at all. The tab now shows the selected project beside the instance-wide
table — rows, an estimated size, the ingest rate over the last hour, how far
back its data goes, and the retention window that actually applies to it
(its own, or the install's, labelled either way). An aggregate reports the
union of the members you may see and names them, and when its members keep
different windows it says "varies" rather than inventing an average. Sizes
are the one estimate: ClickHouse parts hold every project's rows together, so
a project's share can only be apportioned by row count — the column says so
instead of printing an exact-looking number. -
One chart, one instance, many clusters. Every component now has a switch
—hub.enabled,ui.enabled,gateway.enabledbeside the sensor's — so a
second cluster installs the ingest half alone and writes to the central
instance's ClickHouse under its own project. Until now the only way to
observe two clusters was two instances, each with its own store, UI and
login; with member projects (above) one screen already spans them, and this
is what makes the install match. The reductions are real: a secondary
cluster renders no hub, no UI, no auth Secret and — deliberately — no migrate
Job, because two clusters migrating one database is a race. Combinations that
cannot work are refused athelm templatetime with a sentence saying why:
a UI whose hub is elsewhere (its nginx proxies/apito a Service in its own
namespace), a hub-less install writing to the in-chart ClickHouse nothing
would ever query, ingest keys with no hub to validate against, and an Ingress
with nothing left to route. With keys on, the gateway validates against
hub.external.urland the central hub's own internal token — the chart will
not generate a local one, because a token the central hub has never seen
fails closed and looks like a broken sender.
Fixed
-
The hub reported the retention it was built with, not the one you
configured.retention.*reached the migrate Job but never the hub
Deployment, so an install keeping 30 days of traces had a hub still answering
with the 7-day built-in default — Settings → Storage read that as TTL drift
and told operators to re-run a migration that had already worked. Both now
render from one chart helper, and a template assertion fails if either side
loses it. Same values added to the compose hub service for the same reason. -
make version-setleft the hub's embedded chart copy behind. The hub
embeds the sensor-relevant chart files (hub/internal/collection/chart/) so
the runtime-collection applier can render them viago:embed, and a unit
test pins that copy todeploy/helm/avuruobs. Stamping a release version
rewrote the real chart but not the copy, so the very commit that cut a
release failedmake check— caught by CI on the v0.5.0 release commit,
invisible before the stamp because RELEASING.md runs the check first.
version-setnow runssync-hub-chartitself; the two can no longer drift
on a version stamp.