Skip to content

Release 3.0.0

Choose a tag to compare

@github-actions github-actions released this 05 Nov 05:33
Immutable release. Only release title and notes can be modified.
854133c

Added

  • observability-builder: every attribute, field and label now has a class per location — mandatory, conditional, recommended, optional or prohibited — with who sets it and when (references/data-model.md). The tables cover the resource per platform, trace context, request and identity context, operation spans, logs, metrics, profiles, correlation and AI (GenAI, agents, RAG, MCP, A2A). Maintainer notes on the design are in docs/attribute-standard-design.md.
  • observability-builder: platforms outside Kubernetes: ECS and Fargate, Lambda, Cloud Run services and jobs, VMs and plain Docker, each with its own required resource rows and instance identity, set by platform detectors with a short start-up timeout.
  • observability-builder: A2A (agent-to-agent) calls: workloads/a2a.md, the invoke_agent client span and server entry span, app.a2a.* attributes, traceparent on the JSON-RPC call, and the a2a_client_request_duration_seconds / a2a_server_request_duration_seconds metrics.
  • observability-builder: a shared masking helper for personal data in logs, span attributes and exception messages (emails, phones, names, addresses, government and bank IDs, dates of birth, IPs outside the access and audit logs); it fails closed. A new audit log kind (audit.<action>).
  • observability-builder: references/overhead-and-security.md and scripts/overhead.py: instrumentation must stay within an overhead budget (latency, CPU, memory, errors, shutdown), measured with telemetry on and off, and must not weaken security (TLS off private hops, credentials from files, metrics and profiling ports bound no wider than the scraper needs, packaged config, pinned dependencies). Overhead testing is now part of acceptance.
  • observability-builder: scripts/receiver/conformance.py checks captured telemetry against the data model (receive.py --platform, conformance.json). New audit rules: identity-from-header, request-id-attribute, code-namespace, pii-unmasked, profile-label-mismatch, baggage-over-limit, resource-excluded-key, sync-exporter, exporter-no-timeout, plaintext-remote-endpoint, telemetry-credential-literal, metrics-public-bind, attribute-limit-raised, log-extras-passthrough, eager-debug-format.
  • observability-builder: browser RUM with Grafana Faro for single-page apps (spa workload) and the page side of fullstack apps (references/signals/rum.md, workloads/spa.md). Every self-hosted Faro feature is on: errors, console, web vitals with attribution, performance and resource timing, sessions, views, soft navigation, user actions, frustration, CSP violations, custom signals, browser-to-API tracing and React. Three transports: direct to a public Alloy faro.receiver (default), a same-origin proxy (instrument always asks whether it is needed), or the experimental OTLP transport. Faro starts from runtime config without blocking the render, and every item passes a fail-closed beforeSend scrubber. The rum key returns to observability.yml with this new shape, and the release adds tested templates in assets/js/faro, a Grafana dashboard covering every field Faro sends (68 panels, each query checked against live data), the faro.receiver collector block, RUM queries for Loki and Tempo, and a browser overhead budget with scripts/rum_overhead.mjs. The receiver takes Faro batches on /collect and checks them against the data model. Twelve faro-* audit rules and rum-access-log-query were added. Everything was tested end to end on Alloy 1.21, Loki 3.7 and Tempo 3.1.
  • observability-builder: new conditional metrics for RPC (rpc_server/client_duration_seconds), messaging (messaging_process_duration_seconds, messaging_messages_total) and authentication (auth_attempts_total).

Changed

  • observability-builder: the resource keeps the SDK, runtime and OS rows (telemetry.sdk.language/version, process.runtime.*, os.type) and no longer carries k8s.pod.uid; service.instance.id, process IDs and command lines, vcs.*, host.id and deployment.id are never added. The collector matches pods by namespace and pod name, then IP, and adds the owning workload's name.
  • observability-builder: identity comes only from trusted middleware after authentication, never from raw client headers; request.id is not used. Identity rows beyond the core set are added only when the service's code has the value.
  • observability-builder: baggage is limited to an allow-list of at most 8 keys and 512 bytes; exception span events are not added (the error log record carries the exception); gen_ai.usage.cost is no longer allowed on the ops path; the GenAI token type cache_write is now cache_creation.
  • observability-builder: the Langfuse exporter templates (Go, Java, TypeScript) cap each request at 3 MiB by default and cap content in linear time; hold_port.py now holds a port reliably on macOS; audit.py --module accepts absolute paths and no longer flags the skill's own scripts or filtered detectors.
  • observability-builder: the Python module's logging file is logs.py, since logging.py shadows the standard library.
  • observability-builder: genai.content.max_chars defaults to 8000 and stays at or below the span attribute value limit (8192 by default), so the content cap runs before the SDK cuts anything; longer content raises both together as a recorded exception.
  • observability-builder: request context (client address, user agent, headers, cookies, query strings, peer addresses) never goes to Langfuse: the exporter templates and the collector's Langfuse branch drop it in every content mode. The TypeScript Langfuse transport sends OTLP/protobuf, and Langfuse export failures are logged as rate-limited WARN.
  • observability-builder: the overhead budget also accepts up to 1 ms of added CPU per request (cpu_ms_per_request); a function span for an exception that maps to a 4xx response stays UNSET; tracers, meters and loggers are obtained once; Python metrics servers don't reuse a taken port.
  • observability-builder: AI spans on the ops path no longer get user.id and team.id from the feature stamps; request identity stays on the entry span, and Langfuse keeps the user and team in its own trace context. AI spans keep gen_ai.conversation.id, and spans that aren't AI calls still lose stray gen_ai.*, llm.* and traceloop.* keys. The collector's GenAI routing drops content on the ops branch unconditionally.
  • observability-builder: generated correlation ids are URL-safe base64, not hex (hex can look like a card number); a Python metrics server that can't bind without reuse retries with reuse when no listener answers (restarts after a scrape); Python allocation profiling (pyroscope-io 1.3+) is allowed after the overhead test; AI content is passed to the content helper as a function, so nothing is converted while capture is off.
  • observability-builder: detect.py reports a project that is only pages as workload spa, sets frontend for fullstack apps, and lists browser RUM SDKs (Faro, Datadog, New Relic, Elastic, Dynatrace, Application Insights, Splunk, OpenTelemetry Web) as existing telemetry.
  • platform-builder: the example product in docs and tests is storefront instead of oneuptime.

Removed

  • observability-builder: OneUptime as a backend: backends/oneuptime.md, its collector exporter and identity stamping, and the OneUptime examples in the other pages. The Grafana stack, Dynatrace, New Relic, AWS and Langfuse remain (see MIGRATION.md).
  • observability-builder: the OneUptime browser RUM (OpenTelemetry Web SDK) and session replay, with its rum key shape, the receiver's session-replay stand-in and --replay-assets, and the rum-propagate-all and rum-input-capture audit rules. Browser RUM is now Grafana Faro (see Added).
  • observability-builder: genai.content.destination and the receiver's --genai-content-destination. AI content goes to Langfuse only, never to the ops backend.

3.0.0

Required

  • Resource attributes. Remove k8s.pod.uid, service.instance.id, process.pid, process.command_line, process.executable.*, vcs.*, host.id and deployment.id from observability.yml and from code; the schema now rejects them. Stop dropping telemetry.sdk.language/version, process.runtime.* and os.type. Remove the pod UID from the chart's downward API, and update collectors to the pod association in observability-builder/references/collector/README.md (namespace and pod name, then IP; workload names extracted).
  • Identity. Services that set user.id or tenant.id from client headers must take them from the authentication or tenant middleware instead. Remove request.id; use correlation.id.
  • Metrics. Rename the GenAI token_type value cache_write to cache_creation in code and dashboards.
  • GenAI content size. genai.content.max_chars now defaults to 8000. A yml that sets a larger value must also set the span attribute value limit to the same number, with a comment naming max_chars, or lower it to 8192 or less.
  • Collectors with GenAI routing: add the langfuse_request_context transform from observability-builder/references/collector/README.md before the Langfuse batch.
  • OneUptime is no longer a supported backend. Point each service's telemetry at a remaining
    backend (the Grafana stack, Dynatrace, New Relic or AWS) by changing only observability.yml
    and the collector configuration (backends):
    • remove the OneUptime exporter, its x-oneuptime-token auth and the
      deployment.environment transform from collectors;
    • remove x-oneuptime-token headers and oneuptime.label.* resource attributes from
      observability.yml and from the chart's resource.opentelemetry.io/* annotations;
    • remove oneuptime.database.server.id stamping from database-engine pipelines.
  • Delete genai.content.destination from every observability.yml. The schema rejects it
    now, and validate.py reports it as an error.
  • Browser RUM moves to Grafana Faro. The old rum: keys (app_identifier, sample_ratio,
    session_replay, exporter) are rejected; the new rum block has a different shape
    (config). In each frontend:
    • remove the OpenTelemetry Web SDK module (src/observability/client/, rum.ts), the
      OneUptime replay recorder and its oneuptime-* masking classes, and the OneUptime
      /telemetry/session-replay/ proxy routes or nginx locations;
    • run observability instrument on the frontend: it adds the Faro pieces from
      observability-builder/assets/js/faro, asks for the transport and whether a same-origin
      proxy is needed, and rewrites the runtime observability.json (or the fullstack config
      script) to the new rum block;
    • hand the platform team the faro.receiver set-up from
      observability-builder/references/collector/README.md (exact CORS origins, rate limiting,
      payload cap, source maps from files with downloads off, a TLS ingress for /collect only
      when the transport is faro), and the source-map delivery per release;
    • drop the OneUptime RUM secrets from the chart; keep a Faro key Secret only for transport
      proxy.
  • Collectors with GenAI routing: update the ops branch from
    observability-builder/references/collector/README.md. It no longer copies user.id and
    team.id onto AI spans, keeps no content keys, and has no list of ops services; the Langfuse
    branch goes straight to its batch processor (the langfuse_content transform is gone).
  • Scripts: drop --replay-assets and --genai-content-destination from any
    scripts/receive.py invocation.

Recommended

  • Services that set genai.content.destination: ops now send no content anywhere unless
    Langfuse is on: turn Langfuse on where prompts and answers must be read.
  • Dashboards or queries that read user.id or team.id from AI spans read them from the
    entry span (ops backend) or from Langfuse's trace context instead.
  • Frontends with another RUM SDK (Datadog, New Relic, Elastic, Application Insights) move to
    Faro with observability instrument; detect.py lists them under existing telemetry. Import
    observability-builder/assets/grafana/rum-dashboard.json as the RUM dashboard.
  • Run node observability-builder/scripts/rum_overhead.mjs for each frontend with RUM.
  • Run observability instrument on each service to apply the classified data model: platform rows outside Kubernetes, the masking helper for personal data, the audit log, A2A spans where agents call agents, and the new conditional metrics.
  • Run scripts/overhead.py against each instrumented service and fix anything over the budget (observability-builder/references/overhead-and-security.md).
  • Bind /metrics to loopback where the scraper runs in the same pod, task or instance, and comment every plaintext exporter endpoint with its private hop.
  • Rename a Python module's logging.py to logs.py.
  • Generate correlation ids as URL-safe base64 (secrets.token_urlsafe(16), randomBytes(16).toString('base64url')) instead of hex UUIDs: hex ids can hold digit runs that secret scanners read as card numbers.
  • Python metrics servers: on "address in use", check for a real listener and retry with reuse when none answers, so a restart right after a scrape doesn't lose /metrics (metrics).
  • AI content: pass it to the content helper as a function, so nothing is converted while capture is off (genai).
  • nginx in front of pages: log the request path from $request_uri with the query cut off, not $uri (which is /index.html after try_files) or $request (spa).

Verify

  • python3 observability-builder/scripts/audit.py <project> --module <module path> reports no resource-excluded-key, identity-from-header or telemetry-credential-literal findings.
  • The RUM dashboard shows data in every panel after a frontend session (Loki and Tempo data sources selected on import).
  • scripts/receive.py --platform <platform> writes a conformance.json with no P0 or P1 findings for a test run of the service.
  • python3 -m unittest discover -s observability-builder/scripts/tests passes, and the receiver
    tests pass with uv run --no-project --with grpcio --with opentelemetry-proto --with protobuf --with pyyaml python -m unittest discover -s scripts/tests -p "test_receive.py" from
    observability-builder/.
  • python3 observability-builder/scripts/validate.py <project>/observability.yml reports no
    genai.content.destination key and no rum finding.
  • For frontends, audit.py reports no faro-* P0 or P1 finding, and a browser run against
    scripts/receive.py (Faro on its /collect endpoint) shows every record kind with no Faro
    data-model findings.