Repository navigation
v0.5.0
Fixes from the first end-to-end test of the released modules: three services under load, with Redis, NATS, OpenTelemetry export and kill -9 (see docs/e2e-report.md). Three changes alter behaviour, listed under Changed.
Modules released: core v0.5.0, otelflightrec/v0.3.0 and grpcflightrec/v0.3.1 (documentation only). notifier/redisnotifier and notifier/natsnotifier are unchanged at v0.3.0.
Also new: an animated walkthrough and architecture diagrams in the README, and benchmarks of otelflightrec against a real OTLP export path (#30).
Changed
- A local failure is announced to the notifier even when the trace ID was generated by flightrec. Services forward generated IDs to the services they call (in
X-Request-Id, for example), so those services know them; before, the failure of a request that entered without a trace header never reached them. Services whose IDs are mostly generated (edge services withIgnoreRequestTraceIDs) now publish a notice for each local failure; the publish rate limit still applies (#83). HTTPMiddlewareand the gRPC interceptors take the trace ID from the trace already in the request context (an OpenTelemetry span, as reported byOptions.TraceExtractoror the registered bridge) before request headers. Before, a header such as theX-Request-Ida proxy adds overrode the span, so logs carried a different trace ID than the spans and cross-service flush matched the wrong trace. A customGRPCMetadataExtractoris no longer consulted when the context carries a trace, andOptions.TraceExtractornow ranks above headers instead of below them. Put OpenTelemetry middleware outside flightrec's for the two to agree (#84).traceparentvalues are accepted only when valid as the W3C Trace Context specification defines them: lowercase hex, non-zero trace and parent IDs, no versionff, and exactly four fields for version00. Other values are skipped like an invalidX-Request-Id(#84).otelflightrec.SpanProcessorno longer triggers on an error status or exception event of a client or producer span that records the remote's answer as a client error:rpc.grpc.status_codeother than the server-fault codes, orhttp.response.status_codebelow 500. Instrumentation such as otelgrpc marks client spans as errors for answers likeNotFound, so a 404 flushed every service in the trace, against the documented gRPC rules. Outbound errors without such an answer (connection failures, timeouts, database and third-party calls) still trigger (#85).
Fixed
- A failed request whose buffered records were all dropped (evicted by
MaxTotal, or a single record larger thanMaxPerScope) writes onebuffered records droppedline withflightrec.truncatedandflightrec.dropped_records. Before, it wrote nothing, and the loss showed only inStats(#86). - A record too large for a journal entry (about 64 KB encoded) now leaves a marker, so its request is recovered marked
flightrec.truncated, withflightrec.dropped_recordson the summary line. Before, the record was left out silently (#87).