Skip to content

v0.5.0

Choose a tag to compare

@deavgprime deavgprime released this 15 Nov 17:24
· 34 commits to main since this release

Fixes from the first end-to-end test of the released modules: three services under load, with Redis, NATS, OpenTelemetry export and kill -9 (see docs/e2e-report.md). Three changes alter behaviour, listed under Changed.

Modules released: core v0.5.0, otelflightrec/v0.3.0 and grpcflightrec/v0.3.1 (documentation only). notifier/redisnotifier and notifier/natsnotifier are unchanged at v0.3.0.

Also new: an animated walkthrough and architecture diagrams in the README, and benchmarks of otelflightrec against a real OTLP export path (#30).

Changed

  • A local failure is announced to the notifier even when the trace ID was generated by flightrec. Services forward generated IDs to the services they call (in X-Request-Id, for example), so those services know them; before, the failure of a request that entered without a trace header never reached them. Services whose IDs are mostly generated (edge services with IgnoreRequestTraceIDs) now publish a notice for each local failure; the publish rate limit still applies (#83).
  • HTTPMiddleware and the gRPC interceptors take the trace ID from the trace already in the request context (an OpenTelemetry span, as reported by Options.TraceExtractor or the registered bridge) before request headers. Before, a header such as the X-Request-Id a proxy adds overrode the span, so logs carried a different trace ID than the spans and cross-service flush matched the wrong trace. A custom GRPCMetadataExtractor is no longer consulted when the context carries a trace, and Options.TraceExtractor now ranks above headers instead of below them. Put OpenTelemetry middleware outside flightrec's for the two to agree (#84).
  • traceparent values are accepted only when valid as the W3C Trace Context specification defines them: lowercase hex, non-zero trace and parent IDs, no version ff, and exactly four fields for version 00. Other values are skipped like an invalid X-Request-Id (#84).
  • otelflightrec.SpanProcessor no longer triggers on an error status or exception event of a client or producer span that records the remote's answer as a client error: rpc.grpc.status_code other than the server-fault codes, or http.response.status_code below 500. Instrumentation such as otelgrpc marks client spans as errors for answers like NotFound, so a 404 flushed every service in the trace, against the documented gRPC rules. Outbound errors without such an answer (connection failures, timeouts, database and third-party calls) still trigger (#85).

Fixed

  • A failed request whose buffered records were all dropped (evicted by MaxTotal, or a single record larger than MaxPerScope) writes one buffered records dropped line with flightrec.truncated and flightrec.dropped_records. Before, it wrote nothing, and the loss showed only in Stats (#86).
  • A record too large for a journal entry (about 64 KB encoded) now leaves a marker, so its request is recovered marked flightrec.truncated, with flightrec.dropped_records on the summary line. Before, the record was left out silently (#87).