Repository navigation
Release 3.0.0
Immutable
release. Only release title and notes can be modified.
Added
observability-builder: every attribute, field and label now has a class per location — mandatory, conditional, recommended, optional or prohibited — with who sets it and when (references/data-model.md). The tables cover the resource per platform, trace context, request and identity context, operation spans, logs, metrics, profiles, correlation and AI (GenAI, agents, RAG, MCP, A2A). Maintainer notes on the design are indocs/attribute-standard-design.md.observability-builder: platforms outside Kubernetes: ECS and Fargate, Lambda, Cloud Run services and jobs, VMs and plain Docker, each with its own required resource rows and instance identity, set by platform detectors with a short start-up timeout.observability-builder: A2A (agent-to-agent) calls:workloads/a2a.md, theinvoke_agentclient span and server entry span,app.a2a.*attributes,traceparenton the JSON-RPC call, and thea2a_client_request_duration_seconds/a2a_server_request_duration_secondsmetrics.observability-builder: a shared masking helper for personal data in logs, span attributes and exception messages (emails, phones, names, addresses, government and bank IDs, dates of birth, IPs outside the access and audit logs); it fails closed. A new audit log kind (audit.<action>).observability-builder:references/overhead-and-security.mdandscripts/overhead.py: instrumentation must stay within an overhead budget (latency, CPU, memory, errors, shutdown), measured with telemetry on and off, and must not weaken security (TLS off private hops, credentials from files, metrics and profiling ports bound no wider than the scraper needs, packaged config, pinned dependencies). Overhead testing is now part of acceptance.observability-builder:scripts/receiver/conformance.pychecks captured telemetry against the data model (receive.py --platform,conformance.json). New audit rules:identity-from-header,request-id-attribute,code-namespace,pii-unmasked,profile-label-mismatch,baggage-over-limit,resource-excluded-key,sync-exporter,exporter-no-timeout,plaintext-remote-endpoint,telemetry-credential-literal,metrics-public-bind,attribute-limit-raised,log-extras-passthrough,eager-debug-format.observability-builder: browser RUM with Grafana Faro for single-page apps (spaworkload) and the page side of fullstack apps (references/signals/rum.md,workloads/spa.md). Every self-hosted Faro feature is on: errors, console, web vitals with attribution, performance and resource timing, sessions, views, soft navigation, user actions, frustration, CSP violations, custom signals, browser-to-API tracing and React. Three transports: direct to a public Alloyfaro.receiver(default), a same-origin proxy (instrumentalways asks whether it is needed), or the experimental OTLP transport. Faro starts from runtime config without blocking the render, and every item passes a fail-closedbeforeSendscrubber. Therumkey returns toobservability.ymlwith this new shape, and the release adds tested templates inassets/js/faro, a Grafana dashboard covering every field Faro sends (68 panels, each query checked against live data), thefaro.receivercollector block, RUM queries for Loki and Tempo, and a browser overhead budget withscripts/rum_overhead.mjs. The receiver takes Faro batches on/collectand checks them against the data model. Twelvefaro-*audit rules andrum-access-log-querywere added. Everything was tested end to end on Alloy 1.21, Loki 3.7 and Tempo 3.1.observability-builder: new conditional metrics for RPC (rpc_server/client_duration_seconds), messaging (messaging_process_duration_seconds,messaging_messages_total) and authentication (auth_attempts_total).
Changed
observability-builder: the resource keeps the SDK, runtime and OS rows (telemetry.sdk.language/version,process.runtime.*,os.type) and no longer carriesk8s.pod.uid;service.instance.id, process IDs and command lines,vcs.*,host.idanddeployment.idare never added. The collector matches pods by namespace and pod name, then IP, and adds the owning workload's name.observability-builder: identity comes only from trusted middleware after authentication, never from raw client headers;request.idis not used. Identity rows beyond the core set are added only when the service's code has the value.observability-builder: baggage is limited to an allow-list of at most 8 keys and 512 bytes; exception span events are not added (the error log record carries the exception);gen_ai.usage.costis no longer allowed on the ops path; the GenAI token typecache_writeis nowcache_creation.observability-builder: the Langfuse exporter templates (Go, Java, TypeScript) cap each request at 3 MiB by default and cap content in linear time;hold_port.pynow holds a port reliably on macOS;audit.py --moduleaccepts absolute paths and no longer flags the skill's own scripts or filtered detectors.observability-builder: the Python module's logging file islogs.py, sincelogging.pyshadows the standard library.observability-builder:genai.content.max_charsdefaults to 8000 and stays at or below the span attribute value limit (8192 by default), so the content cap runs before the SDK cuts anything; longer content raises both together as a recorded exception.observability-builder: request context (client address, user agent, headers, cookies, query strings, peer addresses) never goes to Langfuse: the exporter templates and the collector's Langfuse branch drop it in every content mode. The TypeScript Langfuse transport sends OTLP/protobuf, and Langfuse export failures are logged as rate-limited WARN.observability-builder: the overhead budget also accepts up to 1 ms of added CPU per request (cpu_ms_per_request); a function span for an exception that maps to a 4xx response stays UNSET; tracers, meters and loggers are obtained once; Python metrics servers don't reuse a taken port.observability-builder: AI spans on the ops path no longer getuser.idandteam.idfrom the feature stamps; request identity stays on the entry span, and Langfuse keeps the user and team in its own trace context. AI spans keepgen_ai.conversation.id, and spans that aren't AI calls still lose straygen_ai.*,llm.*andtraceloop.*keys. The collector's GenAI routing drops content on the ops branch unconditionally.observability-builder: generated correlation ids are URL-safe base64, not hex (hex can look like a card number); a Python metrics server that can't bind without reuse retries with reuse when no listener answers (restarts after a scrape); Python allocation profiling (pyroscope-io1.3+) is allowed after the overhead test; AI content is passed to the content helper as a function, so nothing is converted while capture is off.observability-builder:detect.pyreports a project that is only pages as workloadspa, setsfrontendfor fullstack apps, and lists browser RUM SDKs (Faro, Datadog, New Relic, Elastic, Dynatrace, Application Insights, Splunk, OpenTelemetry Web) as existing telemetry.platform-builder: the example product in docs and tests isstorefrontinstead ofoneuptime.
Removed
observability-builder: OneUptime as a backend:backends/oneuptime.md, its collector exporter and identity stamping, and the OneUptime examples in the other pages. The Grafana stack, Dynatrace, New Relic, AWS and Langfuse remain (see MIGRATION.md).observability-builder: the OneUptime browser RUM (OpenTelemetry Web SDK) and session replay, with itsrumkey shape, the receiver's session-replay stand-in and--replay-assets, and therum-propagate-allandrum-input-captureaudit rules. Browser RUM is now Grafana Faro (see Added).observability-builder:genai.content.destinationand the receiver's--genai-content-destination. AI content goes to Langfuse only, never to the ops backend.
3.0.0
Required
- Resource attributes. Remove
k8s.pod.uid,service.instance.id,process.pid,process.command_line,process.executable.*,vcs.*,host.idanddeployment.idfromobservability.ymland from code; the schema now rejects them. Stop droppingtelemetry.sdk.language/version,process.runtime.*andos.type. Remove the pod UID from the chart's downward API, and update collectors to the pod association inobservability-builder/references/collector/README.md(namespace and pod name, then IP; workload names extracted). - Identity. Services that set
user.idortenant.idfrom client headers must take them from the authentication or tenant middleware instead. Removerequest.id; usecorrelation.id. - Metrics. Rename the GenAI
token_typevaluecache_writetocache_creationin code and dashboards. - GenAI content size.
genai.content.max_charsnow defaults to 8000. A yml that sets a larger value must also set the span attribute value limit to the same number, with a comment namingmax_chars, or lower it to 8192 or less. - Collectors with GenAI routing: add the
langfuse_request_contexttransform fromobservability-builder/references/collector/README.mdbefore the Langfuse batch. - OneUptime is no longer a supported backend. Point each service's telemetry at a remaining
backend (the Grafana stack, Dynatrace, New Relic or AWS) by changing onlyobservability.yml
and the collector configuration (backends):- remove the OneUptime exporter, its
x-oneuptime-tokenauth and the
deployment.environmenttransform from collectors; - remove
x-oneuptime-tokenheaders andoneuptime.label.*resource attributes from
observability.ymland from the chart'sresource.opentelemetry.io/*annotations; - remove
oneuptime.database.server.idstamping from database-engine pipelines.
- remove the OneUptime exporter, its
- Delete
genai.content.destinationfrom everyobservability.yml. The schema rejects it
now, andvalidate.pyreports it as an error. - Browser RUM moves to Grafana Faro. The old
rum:keys (app_identifier,sample_ratio,
session_replay,exporter) are rejected; the newrumblock has a different shape
(config). In each frontend:- remove the OpenTelemetry Web SDK module (
src/observability/client/,rum.ts), the
OneUptime replay recorder and itsoneuptime-*masking classes, and the OneUptime
/telemetry/session-replay/proxy routes or nginx locations; - run
observability instrumenton the frontend: it adds the Faro pieces from
observability-builder/assets/js/faro, asks for the transport and whether a same-origin
proxy is needed, and rewrites the runtimeobservability.json(or the fullstack config
script) to the newrumblock; - hand the platform team the
faro.receiverset-up from
observability-builder/references/collector/README.md(exact CORS origins, rate limiting,
payload cap, source maps from files with downloads off, a TLS ingress for/collectonly
when the transport isfaro), and the source-map delivery per release; - drop the OneUptime RUM secrets from the chart; keep a Faro key Secret only for transport
proxy.
- remove the OpenTelemetry Web SDK module (
- Collectors with GenAI routing: update the ops branch from
observability-builder/references/collector/README.md. It no longer copiesuser.idand
team.idonto AI spans, keeps no content keys, and has no list ofopsservices; the Langfuse
branch goes straight to its batch processor (thelangfuse_contenttransform is gone). - Scripts: drop
--replay-assetsand--genai-content-destinationfrom any
scripts/receive.pyinvocation.
Recommended
- Services that set
genai.content.destination: opsnow send no content anywhere unless
Langfuse is on: turn Langfuse on where prompts and answers must be read. - Dashboards or queries that read
user.idorteam.idfrom AI spans read them from the
entry span (ops backend) or from Langfuse's trace context instead. - Frontends with another RUM SDK (Datadog, New Relic, Elastic, Application Insights) move to
Faro withobservability instrument;detect.pylists them under existing telemetry. Import
observability-builder/assets/grafana/rum-dashboard.jsonas the RUM dashboard. - Run
node observability-builder/scripts/rum_overhead.mjsfor each frontend with RUM. - Run
observability instrumenton each service to apply the classified data model: platform rows outside Kubernetes, the masking helper for personal data, the audit log, A2A spans where agents call agents, and the new conditional metrics. - Run
scripts/overhead.pyagainst each instrumented service and fix anything over the budget (observability-builder/references/overhead-and-security.md). - Bind
/metricsto loopback where the scraper runs in the same pod, task or instance, and comment every plaintext exporter endpoint with its private hop. - Rename a Python module's
logging.pytologs.py. - Generate correlation ids as URL-safe base64 (
secrets.token_urlsafe(16),randomBytes(16).toString('base64url')) instead of hex UUIDs: hex ids can hold digit runs that secret scanners read as card numbers. - Python metrics servers: on "address in use", check for a real listener and retry with reuse when none answers, so a restart right after a scrape doesn't lose
/metrics(metrics). - AI content: pass it to the content helper as a function, so nothing is converted while capture is off (genai).
- nginx in front of pages: log the request path from
$request_uriwith the query cut off, not$uri(which is/index.htmlaftertry_files) or$request(spa).
Verify
python3 observability-builder/scripts/audit.py <project> --module <module path>reports noresource-excluded-key,identity-from-headerortelemetry-credential-literalfindings.- The RUM dashboard shows data in every panel after a frontend session (Loki and Tempo data sources selected on import).
scripts/receive.py --platform <platform>writes aconformance.jsonwith no P0 or P1 findings for a test run of the service.python3 -m unittest discover -s observability-builder/scripts/testspasses, and the receiver
tests pass withuv run --no-project --with grpcio --with opentelemetry-proto --with protobuf --with pyyaml python -m unittest discover -s scripts/tests -p "test_receive.py"from
observability-builder/.python3 observability-builder/scripts/validate.py <project>/observability.ymlreports no
genai.content.destinationkey and norumfinding.- For frontends,
audit.pyreports nofaro-*P0 or P1 finding, and a browser run against
scripts/receive.py(Faro on its/collectendpoint) shows every record kind with no Faro
data-model findings.