Added
-
The model calls you are already sending. Applications in this estate are
calling models, and because those calls arrive as ordinary spans they were
already in the trace store — one at a time, in a waterfall, with no way to ask
what any of it added up to. A new AI module reads them: per model, the
calls, the tokens in and out, latency, failures, and how often an answer was
cut off at the token ceiling; per calling service, the same numbers with an
owner. Nothing new is collected and no schema changes — it reads the
gen_ai.*attributes on spans you are already storing, so an install that
switches the module on sees its history, not just what arrives next.Four readings do the work, and each of them guards a way of being confidently
wrong. The model that answered wins over the one that was asked for,
because an alias resolves at the provider and the response is what a bill is
computed against — and a row that only knows what was requested says so. Both
spellings of the token counts are read, since a large share of real traffic
still reports the older ones and reading only the current pair would show that
traffic as having spent nothing at all. A call that reported no usage is
counted and left out of the token totals rather than averaged in as a zero.
And a truncated answer is not a failed one: the call succeeded and hit the
ceiling, which is the commonest reason a response comes back unusable, so it
gets a column of its own instead of being hidden or mixed into the error rate.Prices are yours to declare (
ai.prices, per million tokens) and absent by
default, in which case the screen reports tokens and says so. A model with no
rate is named rather than costed at zero, so the total is explicitly a floor.
There is no pricing API — it would be the first outbound call in a product
whose promise is that nothing leaves the cluster — and no bundled price table,
which would be stale within a month while looking exactly as authoritative as
a number you typed yourself.Born off: most installs call no models, and a navigation entry for
something you do not have is noise. -
The shape of one request. A trace could be read span by span — a
waterfall, a tree, a flamegraph — or not at all: at three hundred spans the
services it crossed, and the order it crossed them in, are in there and cannot
be seen. The service map does not help, because it aggregates every trace in
the window and so cannot describe a single one. A new Path view on the
trace answers the question between them: which services this request touched,
what called what, and where its time went.Each service is weighted by the time spent inside it rather than by how
long its span lasted — a caller's span contains its callee's, so duration
would credit the entry point with the whole request no matter where the time
actually went. Dependencies that never sent a span of their own — a database,
a cache, a third-party API — are drawn as the terminal hops they are, named by
the endpoint the caller recorded and marked as measured at the caller, because
the far end never confirmed any of it. A branch that failed is red end to end.Selecting focus on a service reduces the view to what that service caused,
which is what filtering a trace by a parent looks like on a graph. -
A page for one service. Clicking a service — in the inventory or on the
map — used to open a filtered trace list, which answers what it served and
nothing else. Asking the ordinary next questions meant visiting four screens
and re-applying the same filter in each. There is now a page per service: its
health and why, its rate, errors and latency over time, who calls it and
what it depends on in two separate lists, and its traces, logs and error
issues behind tabs.Callers and callees are shown apart on purpose — one is who is affected when
this service breaks, the other is what could be breaking it, and merging them
makes both harder to read. The dependency numbers come from the same edge set
the map draws, so the two screens cannot disagree about what depends on what,
and a dependency the hub had to recover across a proxy says so rather than
passing as directly observed. Latency that was never measured shows as a dash,
never as zero.Nothing new is collected or stored: the page is composed from reads the
product already made. -
Where the traffic actually goes. Every trace view so far returned rows —
which requests, and how slow. None answered how much of what, so working out
that one route carries 60% of the traffic meant exporting the operations table
and adding it up by hand. A new Breakdown tab on Traces draws that
distribution as a treemap and a donut, grouped by service, operation, outcome,
span kind, or any span or resource attribute — an HTTP route, a database, a
namespace, an environment, a business tag. It takes the filter panel already
above it, so the chart and the trace list beneath it always describe the same
traffic, and a slice drills straight into the traces behind it.Two things make it worth trusting. It can be weighted by requests or by
total time, and the two rank differently — a service at 5% of requests and
18% of the wall time is invisible under a request count and is exactly what a
latency investigation is looking for. And it is honest about its tail: the
totals are computed over every matching span before the top-N is cut, so what
the chart does not draw comes back as its own slice instead of the top eight
quietly redrawing themselves as the whole estate.It also separates three questions the product used to answer as one:
requests served (what each service was asked to do), trace entry
points (where traffic entered, one per trace), and all spans. On a
meshed estate the first two differ by a factor of two. Nothing new is
collected — it reads the traces already stored —
AEP -
Refused: a third answer to "did it work?" A server that replies 4xx has
neither failed nor succeeded, and the product only had those two words for
it — so a WAF blocking a request, or an authorization layer turning one away,
was reported asOK. Server-side 4xx is now its own class: an amber badge on
the span, a Refused column beside Errors in both the operations overview
and the trace table, and aRefused (4xx)option in the trace search's status
filter (?status=refusedonGET /api/v1/traces).It is deliberately kept out of the error rate. RED, the service map's
health ring and alerting are unchanged, because folding 4xx into errors would
put every auth challenge and every crawler 404 into the number people page on.
A client-side 4xx is still an error, as the HTTP semantic conventions have it —
there the caller is the one that failed.
Changed
-
Prompts and completions are dropped at the gateway by default.
(Behaviour change — see below if you want them kept.) Model message content
reaches avuru-obs only because an application's own SDK was configured to
capture it; nothing here ever asked for it. But nothing here refused it
either, so on any install whose instrumentation captures content, user text
was being written to the trace store under your ordinary retention and
rendered in the trace view to anyone holding the Viewer role. Nobody chose
that. From this release the gateway deletes those attributes before they are
written.The rule is deliberately narrow and deliberately ungated. It matches only the
gen_ai.*keys that carry message text, anchored so that a token count
under the convention's older spelling is never mistaken for a prompt; a span
event that carried content keeps its name and loses its attributes, so you can
still see that your instrumentation is emitting content without holding any of
it. And it is not tied to the AI module: content arrives whether or not
you run that screen, so making the protection depend on the screen would have
protected only the installs that went looking.It applies from the upgrade forward — text already stored stays until its
retention expires it, and nothing here rewrites your history. If you have
decided you want content kept, and have a retention and access story for it,
setgateway.genai.redactContent=false. Token counts, models, latency and
cost are unaffected either way: those are attributes, not content. -
The service map's node shapes now say what a node is at a glance. An
application is a hexagon rather than a circle, and the database, cache or
broker it depends on is a portrait barrel rather than a hexagon — the two
swapped, so the shape a reader meets most often is the distinctive one and the
datastore gets the glyph that has always meant datastore. Mesh proxies keep
the diamond, undetected peers stay hollow, and health keeps the ring: shape
still carries what a node is and nothing else, so no colour was spent on
this. -
The trace list shows the status code instead of the word "OK". A row whose
span answered 403 now reads403, amber, rather than claiming the request was
fine — the list used to contradict the span detail one panel over.ERR
survives for the trace that answered cleanly at the root and failed somewhere
underneath, where the trace-level verdict is the news.