diff --git a/docs/src/main/asciidoc/images/guides/tracing_se_expanded_trace.png b/docs/src/main/asciidoc/images/guides/tracing_se_expanded_trace.png new file mode 100644 index 00000000000..2b6660ef046 Binary files /dev/null and b/docs/src/main/asciidoc/images/guides/tracing_se_expanded_trace.png differ diff --git a/docs/src/main/asciidoc/images/guides/tracing_se_first_span.png b/docs/src/main/asciidoc/images/guides/tracing_se_first_span.png new file mode 100644 index 00000000000..92eb59e30ba Binary files /dev/null and b/docs/src/main/asciidoc/images/guides/tracing_se_first_span.png differ diff --git a/docs/src/main/asciidoc/images/guides/tracing_se_first_trace.png b/docs/src/main/asciidoc/images/guides/tracing_se_first_trace.png new file mode 100644 index 00000000000..502533a1735 Binary files /dev/null and b/docs/src/main/asciidoc/images/guides/tracing_se_first_trace.png differ diff --git a/docs/src/main/asciidoc/images/guides/tracing_se_second_expanded_trace.png b/docs/src/main/asciidoc/images/guides/tracing_se_second_expanded_trace.png new file mode 100644 index 00000000000..3b563f643c5 Binary files /dev/null and b/docs/src/main/asciidoc/images/guides/tracing_se_second_expanded_trace.png differ diff --git a/docs/src/main/asciidoc/images/guides/tracing_se_second_trace_list.png b/docs/src/main/asciidoc/images/guides/tracing_se_second_trace_list.png new file mode 100644 index 00000000000..39c8067012f Binary files /dev/null and b/docs/src/main/asciidoc/images/guides/tracing_se_second_trace_list.png differ diff --git a/docs/src/main/asciidoc/images/guides/tracing_se_span_detail.png b/docs/src/main/asciidoc/images/guides/tracing_se_span_detail.png new file mode 100644 index 00000000000..3d67a4e854a Binary files /dev/null and b/docs/src/main/asciidoc/images/guides/tracing_se_span_detail.png differ diff --git a/docs/src/main/asciidoc/images/guides/tracing_se_trace_list.png b/docs/src/main/asciidoc/images/guides/tracing_se_trace_list.png new file mode 100644 index 00000000000..167264a9b68 Binary files /dev/null and b/docs/src/main/asciidoc/images/guides/tracing_se_trace_list.png differ diff --git a/docs/src/main/asciidoc/se/guides/tracing.adoc b/docs/src/main/asciidoc/se/guides/tracing.adoc index 45a58b98f93..9e84025e241 100644 --- a/docs/src/main/asciidoc/se/guides/tracing.adoc +++ b/docs/src/main/asciidoc/se/guides/tracing.adoc @@ -1,6 +1,6 @@ /////////////////////////////////////////////////////////////////////////////// - Copyright (c) 2019, 2023 Oracle and/or its affiliates. + Copyright (c) 2019, 2024 Oracle and/or its affiliates. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -58,7 +58,7 @@ Spans are automatically created by Helidon as needed during execution of the RES == Getting Started with Tracing The examples in this guide demonstrate how to integrate tracing with Helidon, how to view traces, how to trace -across multiple services, and how to integrate with tracing with Kubernetes. All examples use Jaeger and traces +across multiple services, and how to integrate tracing with Kubernetes. All examples use Jaeger and traces will be viewed using the Jaeger UI. === Create a Sample Helidon SE Project @@ -109,7 +109,7 @@ docker run -d --name jaeger \ <1> === Enable Tracing in the Helidon Application -Update the pom.xml file and add the following Jaeger dependency to the `` +Update the `pom.xml` file and add the following Jaeger dependency to the `` section (*not* ``). This will enable Helidon to use Jaeger at the default host and port, `localhost:14250`. @@ -118,25 +118,27 @@ default host and port, `localhost:14250`. ---- io.helidon.tracing - helidon-tracing <1> + helidon-tracing io.helidon.webserver.observe - helidon-webserver-observe-tracing <2> + helidon-webserver-observe-tracing + runtime io.helidon.tracing.providers - helidon-tracing-providers-jaeger <3> + helidon-tracing-providers-jaeger + runtime ---- <1> Helidon Tracing dependencies. <2> Observability features for tracing. <3> Jaeger tracing provider. -All spans sent by Helidon to Jaeger need to be associated with a service. Specify the service name below. +All spans sent by Helidon to Jaeger need to be associated with a service, assigned by the `tracing.service` setting in the example below. [source,bash] -.Add the following lines to `resources/application.yaml`: +.Add the following lines to `src/main/resources/application.yaml`: ---- tracing: service: helidon-se-1 @@ -152,59 +154,87 @@ tracing: propagation: b3 ---- -[source,java] -.Update the `Main` class. Add Tracer to the WebServer builder ----- -import io.helidon.tracing.TracerBuilder; -... +=== View Automatic Tracing of REST Endpoints +Tracing is part of Helidon's observability support. By default, Helidon discovers any observability feature on the classpath and activates it automatically. In particular for tracing, Helidon adds a trace each time a client accesses a service endpoint. +You can see these traces using the Jaeger UI once you build, run, and access your application without changing your application's Java code. -Tracer tracer = TracerBuilder.create("helidon") <1> - .build(); +==== Build and Access QuickStart -WebServer server = WebServer.builder(createRouting(config)) - .config(config.get("server")) - .addFeature(ObserveFeature.builder() - .addObserver(TracingObserver.create(tracer)) <2> - .build()) - .addMediaSupport(JsonpSupport.create()) - .build(); +[source,bash] +.Build and run the application +---- +mvn clean package +java -jar target/helidon-quickstart-se.jar +---- + +[source,bash] +.Access the application +---- +curl http://localhost:8080/greet ---- -<1> Create the `Tracer` object. -<2> Add an observability feature using the created `Tracer`. + +=== Viewing Traces Using the Jaeger UI + +Jaeger provides a web-based UI at http://localhost:16686 where you can see a visual +representation of the traces and spans within them. + +. From the `Service` drop list select `helidon-se-1`. This name corresponds to the `tracing.service` setting you assigned in the `application.yaml` config file. +. Click on the UI Find Traces button. Notice that you can change the look-back time to restrict the trace list. +You will see a trace for each `curl` command you ran to access the application. + +.List of traces +image::guides/tracing_se_trace_list.png[Trace List] + +Click on a trace to see the trace detail page (shown below) which shows the spans within the trace. You can clearly +see the root span (`HTTP Request`) and the single child span (`content-write`) along with the time over which each span was active. + +.Trace detail page +image::guides/tracing_se_first_trace.png[Trace Detail] + +You can examine span details by clicking on the span row. Refer to the image below which shows the span details including timing information. +You can see times for each space relative to the root span. + +.Span detail page +image::guides/tracing_se_span_detail.png[Span Details] + +=== Adding a Custom Span +Your application can use the Helidon tracing API to create custom spans. +The following code replaces the generated `getDefaultMessageHandler` method to add a custom span around the code which prepares the default greeting response. The new custom span's parent span is set to the one which Helidon automatically creates for the REST endpoint. [source,java] -.Update the `GreetService` class: replace the `getDefaultMessageHandler` method: +.Update the `GreetService` class, replacing the `getDefaultMessageHandler` method: ---- private void getDefaultMessageHandler(ServerRequest request, ServerResponse response) { - var spanBuilder = Tracer.global().spanBuilder("mychildSpan"); <1> - request.context().get(SpanContext.class).ifPresent(sc -> sc.asParent(spanBuilder)); <2> - var span = spanBuilder.start(); <3> + var spanBuilder = Tracer.global().spanBuilder("secondchildSpan"); // <1> + request.context().get(SpanContext.class).ifPresent(sc -> sc.asParent(spanBuilder)); // <2> + var span = spanBuilder.start(); // <3> - try { + try (Scope scope = span.activate()) { // <4> sendResponse(response, "World"); - span.end(); <4> + span.end(); // <5> } catch (Throwable t) { - span.end(t); <5> + span.end(t); // <6> } } ---- <1> Create a new `Span` using the global tracer. <2> Set the parent of the new span to the span from the `Request` if available. <3> Start the span. -<4> End the span normally after the response is sent. -<5> End the span with an exception if one was thrown. +<4> Make the new span the current span, returning a `Scope` which is autoclosed. +<5> End the span normally after the response is sent. +<6> End the span with an exception if one was thrown. [source,bash] -.Build the application, skipping unit tests, then run it: +.Build the application and run it: ---- -mvn package -DskipTests=true +mvn package java -jar target/helidon-quickstart-se.jar ---- [source,bash] -.Run the curl command in a new terminal window and check the response: +.Run the `curl` command in a new terminal window and check the response: ---- curl http://localhost:8080/greet ... @@ -213,50 +243,27 @@ curl http://localhost:8080/greet } ---- +Return to the main Jaeger UI screen and click Find Traces again. The new display contains an additional trace, displayed first, for the most recent `curl` you ran. -=== Viewing Tracing Using Jaeger UI - -The tracing output data is verbose and can be challenging to interpret using the REST API, especially since it represents -a structure of spans. Jaeger provides a web-based UI at http://localhost:16686/search, where you can see a visual -representation of the same data and the relationship between spans within a trace. - -Click on the UI Find traces button (the search icon) as shown in the image below. Notice that you can change the look-back time to restrict the trace list. - -.Jaeger UI -image::guides/12_tracing_se_refresh.png[Trace Refresh] - -The image below shows the trace summary, including start time and duration of each trace. There are two traces, -each one generated in response to a `curl http://localhost:8080/greet` invocation. The oldest trace will have a much -longer duration since there is one-time initialization that occurs. - -.Tracing list view -image::guides/12_tracing_se_top.png[Traces] - -Click on a trace, and you will see the trace detail page where the spans are listed. You can clearly -see the root span and the relationship among all the spans in the trace, along with timing information. - -.Trace detail page -image::guides/12_tracing_se_detail.png[Trace Detail] - -NOTE: For OpenTracing, a parent span might not depend on the result of the child. This is called a `FollowsFrom` reference, see -https://github.com/opentracing/specification/blob/master/specification.md[Open Tracing Semantic Spec]. +.Expanded trace list +image::guides/tracing_se_second_trace_list.png[Expanded trace list] -You can examine span details by clicking on the span row. Refer to the image below, which shows the span details, including timing information. -You can see times for each space relative to the root span. These rows are annotated with `Server Start` and `Server Finish`, as shown in the third column. +Notice that the top trace has three spans, not two as with the earlier trace. Click on the trace to see the trace details. -.Span detail page -image::guides/12_tracing_span_detail.png[Span Details] +.Trace details with custom span +image::guides/tracing_se_expanded_trace.png[Trace details with custom span] +Note the row for `mychildSpan`--the custom span created by the added code. -=== Tracing Across Services +=== Using Tracing Across Services Helidon automatically traces across services if the services use the same tracer, for example, the same instance of Jaeger. This means a single trace can include spans from multiple services and hosts. Helidon uses a `SpanContext` to propagate tracing information across process boundaries. When you make client API calls, Helidon will internally call OpenTelemetry APIs or OpenTracing APIs to propagate the `SpanContext`. There is nothing you need to do in your application to make this work. -To demonstrate distributed tracing, you will need to create a second project, where the server listens to on port 8081. -Create a new root directory to hold this new project, then do the following steps, similar to +To demonstrate distributed tracing, create a second project where the server listens to on port 8081. +Create a new directory to hold this new project, then do the following steps, similar to what you did at the start of this guide: === Create the Second Service @@ -274,7 +281,7 @@ mvn -U archetype:generate -DinteractiveMode=false \ ---- [source,bash] -.The project will be built and run from the `helidon-quickstart-se` directory: +.The project is in the `helidon-quickstart-se-2` directory: ---- cd helidon-quickstart-se-2 ---- @@ -284,15 +291,17 @@ cd helidon-quickstart-se-2 ---- io.helidon.tracing - helidon-tracing <1> + helidon-tracing io.helidon.webserver.observe - helidon-webserver-observe-tracing <2> + helidon-webserver-observe-tracing + runtime io.helidon.tracing.providers - helidon-tracing-providers-jaeger <3> + helidon-tracing-providers-jaeger + runtime ---- <1> Helidon Tracing API. @@ -300,7 +309,7 @@ cd helidon-quickstart-se-2 <3> Jaeger tracing provider. [source,bash] -.Replace `resources/application.yaml` with the following: +.Replace `src/main/resources/application.yaml` with the following: ---- app: greeting: "Hello From SE-2" @@ -323,46 +332,30 @@ server: host: 0.0.0.0 ---- -NOTE: The settings above are for development and experimental purposes only. For production environment, please see the link:../tracing.adoc[Tracing documentation]. - -[source,java] -.Update the `Main` class; Add Tracer to the WebServer builder ----- -Tracer tracer = TracerBuilder.create("helidon") <1> - .build(); - -WebServer server = WebServer.builder(createRouting(config)) - .config(config.get("server")) - .addFeature(ObserveFeature.builder() - .addObserver(TracingObserver.create(tracer)) <2> - .build()) - .addMediaSupport(JsonpSupport.create()) - .build(); ----- -<1> Create the `Tracer` object. -<2> Add an observability feature using the created `Tracer`. +NOTE: The settings above are for development and experimental purposes only. For a production environment please see the xref:../tracing.adoc[Tracing documentation]. [source,java] .Update the `GreetService` class. Replace the `getDefaultMessageHandler` method: ---- private void getDefaultMessageHandler(ServerRequest request, - ServerResponse response) { + ServerResponse response) { - var spanBuilder = request.tracer() - .buildSpan("getDefaultMessageHandler"); - request.spanContext().ifPresent(spanBuilder::asChildOf); + var spanBuilder = Tracer.global().spanBuilder("getDefaultMessageHandler"); + request.context().get(SpanContext.class).ifPresent(spanBuilder::parent); Span span = spanBuilder.start(); - try { + try (Scope scope = span.activate()) { sendResponse(response, "World"); + span.end(); } catch (Throwable t) { span.end(t); } } ---- +Build the application, skipping unit tests; the unit tests check for the default greeting response which is now different in the updated config. Then run the application. [source,bash] -.Build the application, skipping unit tests, then run it: +.Build and run: ---- mvn package -DskipTests=true java -jar target/helidon-quickstart-se-2.jar @@ -384,133 +377,109 @@ Once you have validated that the second service is running correctly, you need t call it. [source,xml] -.Add the following dependency to `pom.xml`: +.Add the following dependencies to `pom.xml`: ---- - io.helidon.security.integration - helidon-security-integration-jersey + io.helidon.webclient + helidon-webclient - io.helidon.tracing - helidon-tracing-jersey-client + io.helidon.webclient + helidon-webclient-api - org.glassfish.jersey.core - jersey-client + io.helidon.webclient + helidon-webclient-tracing - org.glassfish.jersey.inject - jersey-hk2 + io.helidon.webclient + helidon-webclient-http1 + runtime ---- +Make the following changes to the `GreetService` class. - +1. Add a `WebClient` field. ++ [source,java] -.Replace the `GreetService` class with the following code: +.Add a private instance field (before the constructors) +---- +private WebClient webClient; ---- -public class GreetService implements HttpService { - - private final AtomicReference greeting = new AtomicReference<>(); - private WebTarget webTarget; - private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap()); - - GreetService(Config config) { - greeting.set(config.get("app.greeting").asString().orElse("Ciao")); - - Client jaxRsClient = ClientBuilder.newBuilder().build(); - - webTarget = jaxRsClient.target("http://localhost:8081/greet"); - } - - @Override - public void routing(HttpRules rules) { - rules - .get("/", this::getDefaultMessageHandler) - .get("/outbound", this::outboundMessageHandler) // <1> - .put("/greeting", this::updateGreetingHandler); - } - - private void getDefaultMessageHandler(ServerRequest request, ServerResponse response) { - var spanBuilder = Tracer.global().spanBuilder("getDefaultMessageHandler"); - request.context().get(SpanContext.class).ifPresent(sc -> sc.asParent(spanBuilder)); - var span = spanBuilder.start(); - - try { - sendResponse(response, "World"); - span.end(); - } catch (Throwable t) { - span.end(t); - } - } - - private void sendResponse(ServerResponse response, String name) { - String msg = String.format("%s %s!", greeting.get(), name); - - JsonObject returnObject = JSON.createObjectBuilder().add("message", msg).build(); - response.send(returnObject); - } - - private void updateGreetingFromJson(JsonObject jo, ServerResponse response) { - - if (!jo.containsKey("greeting")) { - JsonObject jsonErrorObject = - JSON.createObjectBuilder().add("error", "No greeting provided").build(); - response.status(Http.Status.BAD_REQUEST_400).send(jsonErrorObject); - return; - } - - greeting.set(jo.getString("greeting")); - response.status(Http.Status.NO_CONTENT_204).send(); - } - private void outboundMessageHandler(ServerRequest request, ServerResponse response) { - Invocation.Builder requestBuilder = webTarget.request(); +2. Add code to initialize the `WebClient` field. ++ +[source,java] +.Add the following code to the `GreetService(Config)` constructor +---- +webClient = WebClient.builder() + .baseUri("http://localhost:8081") + .addService(WebClientTracing.create()) + .build(); +---- +3. Add a routing rule for the new endpoint `/outbound`. ++ +[source,java] +.Add the following line in the `routing` method as the first `.get` invocation in the method +---- +.get("/outbound", this::outboundMessageHandler) +---- +4. Add a method to handle requests to `/outbound`. ++ +[source,java] +.Add the following method +---- +private void outboundMessageHandler(ServerRequest request, + ServerResponse response) { var spanBuilder = Tracer.global().spanBuilder("outboundMessageHandler"); - request.context().get(SpanContext.class).ifPresent(sc -> sc.asParent(spanBuilder)); - var span = spanBuilder.start(); // <2> + request.context().get(SpanContext.class).ifPresent(spanBuilder::parent); + var span = spanBuilder.start(); - try { - requestBuilder.property( - ClientTracingFilter.CURRENT_SPAN_CONTEXT_PROPERTY_NAME, span.context()); // <3> + try (Scope scope = span.activate()) { + ClientResponseTyped remoteResult = webClient.get() + .path("/greet") + .accept(MediaTypes.APPLICATION_JSON) + .request(JsonObject.class); - String result = requestBuilder // <4> - .get(String.class); - response.send(result); - span.end(); - } catch (Throwable t) { - span.end(t); // <5> + response.status(remoteResult.status()).send(remoteResult.entity()); + span.end(); + } catch (Exception e) { + response.status(Status.INTERNAL_SERVER_ERROR_500).send(); + span.end(e); } - } - } + ---- -<1> Add `outboundMessageHandler` to the routing rules. -<2> Create and start a span that is a child of the current span. -<3> Set a property with the `SpanContext`. -<4> Invoke the second service. -<5> Stop the span. +Stop the application if it is still running, rebuild and run it, then invoke the endpoint and check the response. [source,bash] -.Build and run the application, then invoke the endpoint and check the response: +.Build, run, and access the application ---- +mvn clean package +java -jar target/helidon-quickstart-se.jar curl -i http://localhost:8080/greet/outbound // <1> ... { "message": "Hello From SE-2 World!" // <2> } ---- -<1> The request went to the service on `8080`, which then invoked the service at `8081` to get the greeting. +<1> The request goes to the service on `8080`, which then invokes the service at `8081` to get the greeting. <2> Notice the greeting came from the second service. -Refresh the Jaeger UI trace listing page and notice that there is a trace across two services. +Refresh the Jaeger UI trace listing page and notice that there is a trace across two services. Click on that trace to see its details. .Tracing across multiple services detail view -image::guides/12_tracing_se_detail_2_services.png[Traces] +image::guides/tracing_se_second_expanded_trace.png[Traces] + +Note several things about the display: + +1. The top-level span `helidon-se-1 HTTP Request` includes all the work across _both_ services. +2. `helidon-se-1 outboundMessageHandler` is the custom span you added to the first service `/outbound` endpoint code. +3. `helidon-se-1 GET-http://localhost:8080/greet` captures the work the `WebClient` is doing in sending a request to the second service. Helidon adds these spans automatically to each outbound `WebClient` request. +4. `helidon-se-2 HTTP Request` represents the arrival of the request sent by the first service's `WebClient` at the second service's `/greet` endpoint. +5. `helidon-se-2 getDefaultMessageHandler` is the custom span you added to the second service `/greet` endpoint code. -In the image above, you can see that the trace includes spans from two services. You will notice there is a gap before the sixth span, -which is a `get` operation. This is a one-time client initialization delay. Run the `/outbound` curl command again and look at the new trace to -see that the delay no longer exists. You can now stop your second service, it is no longer used in this guide.