Skip to content

Latest commit

 

History

History
104 lines (68 loc) · 4.23 KB

tracing.mdx

File metadata and controls

104 lines (68 loc) · 4.23 KB
Raw
sidebar_position description
4
Find out how to export traces of your invocations.

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

Tracing

Restate supports the following tracing features:

  • Runtime execution tracing per invocation
  • Exporting traces to OTLP-compatible systems (e.g. Jaeger)
  • Correlating parent traces of incoming gRPC/Connect HTTP requests, using the W3C TraceContext specification.

Setting up OTLP exporter

Set up the OTLP exporter by pointing the configuration entry observability.tracing.endpoint to your trace collector. The exporter sends OTLP trace data via gRPC.

Exporting OTLP traces to Jaeger

Jaeger accepts OTLP trace data via gRPC on port 4317.

:::note Start Jaeger with the environment variable COLLECTOR_OTLP_ENABLED=true to enable OTLP. :::

Refer to the Jaeger documentation for more details on how to deploy Jaeger.

:::note Configure the tracing endpoint in Restate as a fully specified URL: http://<jaeger-collector>:4317. :::

You can configure a span/event filter in a similar fashion to the log filter setting the observability.tracing.filter configuration entry.

Try it out locally

To experiment with this feature locally, run the Jaeger container via:

docker run -d --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-p 4317:4317 \
-p 16686:16686 \
jaegertracing/all-in-one:1.46

Then launch Restate with the tracing endpoint defined as an environment variable:

docker run --name restate_dev --rm -d -e RESTATE_OBSERVABILITY__TRACING__ENDPOINT=http://localhost:4317 --network=host docker.io/restatedev/restate:VAR::RESTATE_VERSION
docker run --name restate_dev --rm -d -e RESTATE_OBSERVABILITY__TRACING__ENDPOINT=http://host.docker.internal:4317 -p 8080:8080 -p 9070:9070 -p 9071:9071 docker.io/restatedev/restate:VAR::RESTATE_VERSION

Go to the Jaeger UI at http://localhost:16686.

If you now spin up your Restate services and send requests to them, you will see the traces appear in the Jaeger UI.

An example from the shopping cart example: Observability

Setting up Jaeger file exporter

If you can't configure a Jaeger agent, you can export traces by writing them to files, using the Jaeger JSON format. To do so, set up the configuration entry observability.tracing.json_file_export_path pointing towards the path where trace files should be written.

You can import the trace files using the Jaeger UI:

Jaeger UI File import

Understanding traces

The traces contain detailed information about the context calls that were done during the invocation (e.g. sleep, one-way calls, interaction with state):

Understanding traces

The initial ingress_invoke spans show when the gRPC/Connect HTTP request was received by Restate. The invoke span beneath it shows when Restate invoked the service deployment to process the request.

The tags of the spans contain the metadata of the context calls (e.g. call arguments, invocation id).

When a service invokes another service, the child invocation is linked automatically to the parent invocation, as you can see in the image. Note that the spans of one-way calls are shown as separate traces. The parent invocation only shows that the one-way call was scheduled, not its entire tracing span. To see this information, search for the trace of the one-way call by filtering on the invocation id tag:

restate.invocation.id="T4pIkIJIGAsBiiGDV2dxK7PkkKnWyWHE"

Searching traces

Traces export attributes and tags that correlate the trace with the service and/or invocation. For example, in the Jaeger UI, you can filter on the invocation id (restate.invocation.id) or any other tag:

Jaeger invocation id search