openshift · kalexand-rh · Feb 20, 2025 · Dec 19, 2024
diff --git a/modules/otel-autoinstrumentation.adoc b/modules/otel-autoinstrumentation.adoc
@@ -0,0 +1,21 @@
+// Module included in the following assemblies:
+//
+// * observability/otel/otel-configuration-of-instrumentation.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="otel-autoinstrumentation_{context}"]
+= Auto-instrumentation in the {OTELOperator}
+
+Auto-instrumentation in the {OTELOperator} can automatically instrument an application without manual code changes. Developers and administrators can monitor applications with minimal effort and changes to the existing codebase.
+
+Auto-instrumentation runs as follows:
+
+. The {OTELOperator} injects an init-container, or a sidecar container for Go, to add the instrumentation libraries for the programming language of the instrumented application.
+
+. The {OTELOperator} sets the required environment variables in the application's runtime environment. These variables configure the auto-instrumentation libraries to collect traces, metrics, and logs and send them to the appropriate OpenTelemetry Collector or another telemetry backend.
+
+. The injected libraries automatically instrument your application by connecting to known frameworks and libraries, such as web servers or database clients, to collect telemetry data. The source code of the instrumented application is not modified.
+
+. Once the application is running with the injected instrumentation, the application automatically generates telemetry data, which is sent to a designated OpenTelemetry Collector or an external OTLP endpoint for further processing.
+
+Auto-instrumentation enables you to start collecting telemetry data quickly without having to manually integrate the OpenTelemetry SDK into your application code. However, some applications might require specific configurations or custom manual instrumentation.
diff --git a/modules/otel-config-instrumentation.adoc b/modules/otel-config-instrumentation.adoc
@@ -8,11 +8,9 @@
 
 The {OTELName} can inject and configure the OpenTelemetry auto-instrumentation libraries into your workloads. Currently, the project supports injection of the instrumentation libraries from Go, Java, Node.js, Python, .NET, and the Apache HTTP Server (`httpd`).
 
-Auto-instrumentation in OpenTelemetry refers to the capability where the framework automatically instruments an application without manual code changes. This enables developers and administrators to get observability into their applications with minimal effort and changes to the existing codebase.
-
 [IMPORTANT]
 ====
-The {OTELName} Operator only supports the injection mechanism of the instrumentation libraries but does not support instrumentation libraries or upstream images. Customers can build their own instrumentation images or use community images.
+The {OTELOperator} only supports the injection mechanism of the instrumentation libraries but does not support instrumentation libraries or upstream images. Customers can build their own instrumentation images or use community images.
 ====
 
 [id="otel-instrumentation-options_{context}"]

diff --git a/...troubleshoot-logging-exporter-stdout.adoc → ...l-troubleshoot-debug-exporter-stdout.adoc b/...troubleshoot-logging-exporter-stdout.adoc → ...l-troubleshoot-debug-exporter-stdout.adoc
@@ -4,9 +4,9 @@
 
 :_mod-docs-content-type: PROCEDURE
 [id="debug-exporter-to-stdout_{context}"]
-= Debug exporter
+= Debug Exporter
 
-You can configure the debug exporter to export the collected data to the standard output.
+You can configure the Debug Exporter to export the collected data to the standard output.
 
 .Procedure
 

diff --git a/modules/otel-troubleshooting-instrumentation-injection-into-your-workload.adoc b/modules/otel-troubleshooting-instrumentation-injection-into-your-workload.adoc
@@ -0,0 +1,93 @@
+// Module included in the following assemblies:
+//
+// * observability/otel/otel-troubleshooting.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="otel-troubleshooting-instrumentation-injection-into-your-workload_{context}"]
+= Troubleshooting instrumentation injection into your workload
+
+To troubleshoot instrumentation injection, you can perform the following activities:
+
+* Checking if the `Instrumentation` object was created
+* Checking if the init-container started
+* Checking if the resources were deployed in the correct order
+* Searching for errors in the Operator logs
+* Double-checking the pod annotations
+
+.Procedure
+
+. Run the following command to verify that the `Instrumentation` object was successfully created:
++
+[source,terminal]
+----
+$ oc get instrumentation -n <workload_project> # <1>
+----
+<1> The namespace where the instrumentation was created.
+
+. Run the following command to verify that the `opentelemetry-auto-instrumentation` init-container successfully started, which is a prerequisite for instrumentation injection into workloads:
++
+[source,terminal]
+----
+$ oc get events -n <workload_project> # <1>
+----
+<1> The namespace where the instrumentation is injected for workloads.
++
+.Example output
+[source,terminal]
+----
+... Created container opentelemetry-auto-instrumentation
+... Started container opentelemetry-auto-instrumentation
+----
+
+. Verify that the resources were deployed in the correct order for the auto-instrumentation to work correctly. The correct order is to deploy the `Instrumentation` custom resource (CR) before the application. For information about the `Instrumentation` CR, see the section "Configuring the instrumentation".
++
+[NOTE]
+====
+When the pod starts, the {OTELOperator} checks the `Instrumentation` CR for annotations containing instructions for injecting auto-instrumentation. Generally, the Operator then adds an init-container to the application’s pod that injects the auto-instrumentation and environment variables into the application's container. If the `Instrumentation` CR is not available to the Operator when the application is deployed, the Operator is unable to inject the auto-instrumentation.
+====
++
+Fixing the order of deployment requires the following steps:
+
+.. Update the instrumentation settings.
+.. Delete the instrumentation object.
+.. Redeploy the application.
+
+. Run the following command to inspect the Operator logs for instrumentation errors:
++
+[source,terminal]
+----
+$ oc logs -l app.kubernetes.io/name=opentelemetry-operator --container manager -n openshift-opentelemetry-operator --follow
+----
+
+. Troubleshoot pod annotations for the instrumentations for a specific programming language. See the required annotation fields and values in "Configuring the instrumentation".
+
+.. Verify that the application pods that you are instrumenting are labeled with correct annotations and the appropriate auto-instrumentation settings have been applied.
++
+.Example
+----
+instrumentation.opentelemetry.io/inject-python="true"
+----
++
+.Example command to get pod annotations for an instrumented Python application
+[source,terminal]
+----
+$ oc get pods -n <workload_project> -o jsonpath='{range .items[?(@.metadata.annotations["instrumentation.opentelemetry.io/inject-python"]=="true")]}{.metadata.name}{"\n"}{end}'
+----
+
+.. Verify that the annotation applied to the instrumentation object is correct for the programming language that you are instrumenting.
+
+.. If there are multiple instrumentations in the same namespace, specify the name of the `Instrumentation` object in their annotations.
++
+.Example
+----
+instrumentation.opentelemetry.io/inject-nodejs: "<instrumentation_object>"
+----
+
+.. If the `Instrumentation` object is in a different namespace, specify the namespace in the annotation.
++
+.Example
+----
+instrumentation.opentelemetry.io/inject-nodejs: "<other_namespace>/<instrumentation_object>"
+----
+
+.. Verify that the `OpenTelemetryCollector` custom resource specifies the auto-instrumentation annotations under `spec.template.metadata.annotations`. If the auto-instrumentation annotations are in `spec.metadata.annotations` instead, move them into `spec.template.metadata.annotations`.
diff --git a/...troubleshooting-telemetry-data-generation-by-the-instrumentation-libraries.adoc b/...troubleshooting-telemetry-data-generation-by-the-instrumentation-libraries.adoc
@@ -0,0 +1,31 @@
+// Module included in the following assemblies:
+//
+// * observability/otel/otel-troubleshooting.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="otel-troubleshooting-telemetry-data-generation-by-the-instrumentation-libraries_{context}"]
+= Troubleshooting telemetry data generation by the instrumentation libraries
+
+You can troubleshoot telemetry data generation by the instrumentation libraries by checking the endpoint, looking for errors in your application logs, and verifying that the Collector is receiving the telemetry data.
+
+.Procedure
+
+. Verify that the instrumentation is transmitting data to the correct endpoint:
++
+[source,terminal]
+----
+$ oc get instrumentation <instrumentation_name> -n <workload_project> -o jsonpath='{.spec.endpoint}'
+----
++
+The default endpoint `+http://localhost:4317+` for the `Instrumentation` object is only applicable to a Collector instance that is deployed as a sidecar in your application pod. If you are using an incorrect endpoint, correct it by editing the `Instrumentation` object and redeploying your application.
+
+. Inspect your application logs for error messages that might indicate that the instrumentation is malfunctioning:
++
+[source,terminal]
+----
+$ oc logs <application_pod> -n <workload_project>
+----
+
+. If the application logs contain error messages that indicate that the instrumentation might be malfunctioning, install the OpenTelemetry SDK and libraries locally. Then run your application locally and troubleshoot for issues between the instrumentation libraries and your application without {product-title}.
+
+. Use the Debug Exporter to verify that the telemetry data is reaching the destination OpenTelemetry Collector instance. For more information, see "Debug Exporter".
diff --git a/observability/otel/otel-configuration-of-instrumentation.adoc b/observability/otel/otel-configuration-of-instrumentation.adoc
@@ -7,6 +7,7 @@ include::_attributes/common-attributes.adoc[]
 toc::[]
 
 
-The {OTELName} Operator uses a custom resource definition (CRD) file that defines the configuration of the instrumentation.
+The {OTELName} Operator uses an `Instrumentation` custom resource that defines the configuration of the instrumentation.
 
+include::modules/otel-autoinstrumentation.adoc[leveloffset=+1]
 include::modules/otel-config-instrumentation.adoc[leveloffset=+1]
diff --git a/observability/otel/otel-troubleshooting.adoc b/observability/otel/otel-troubleshooting.adoc
@@ -11,7 +11,7 @@ The OpenTelemetry Collector offers multiple ways to measure its health as well a
 include::modules/otel-troubleshoot-collecting-diagnostic-data-from-command-line.adoc[leveloffset=+1]
 include::modules/otel-troubleshoot-collector-logs.adoc[leveloffset=+1]
 include::modules/otel-troubleshoot-metrics.adoc[leveloffset=+1]
-include::modules/otel-troubleshoot-logging-exporter-stdout.adoc[leveloffset=+1]
+include::modules/otel-troubleshoot-debug-exporter-stdout.adoc[leveloffset=+1]
 include::modules/otel-troubleshoot-network-traffic.adoc[leveloffset=+1]
 
 [role="_additional-resources"]
@@ -20,3 +20,14 @@ include::modules/otel-troubleshoot-network-traffic.adoc[leveloffset=+1]
 * xref:../../observability/network_observability/installing-operators.adoc#installing-network-observability-operators[Installing the Network Observability Operator]
 
 * xref:../../observability/network_observability/observing-network-traffic.adoc#nw-observe-network-traffic[Observing the network traffic from the Topology view]
+
+[id="troubleshooting_instrumentation_{context}"]
+== Troubleshooting the instrumentation
+
+To troubleshoot the instrumentation, look for any of the following issues:
+
+* Issues with instrumentation injection into your workload
+* Issues with data generation by the instrumentation libraries
+
+include::modules/otel-troubleshooting-instrumentation-injection-into-your-workload.adoc[leveloffset=+2]
+include::modules/otel-troubleshooting-telemetry-data-generation-by-the-instrumentation-libraries.adoc[leveloffset=+2]