Rework OTLP endpoint headers for improved in page navigation, add Gen…

…eral Triaging section to OTLP Troubleshooting
newrelic · May 2, 2024 · fcda2a1 · fcda2a1
1 parent 1aa714c
commit fcda2a1
Show file tree

Hide file tree

Showing 2 changed files with 27 additions and 13 deletions.
diff --git a/...ntegrations/opentelemetry/best-practices/opentelemetry-otlp-troubleshooting.mdx b/...ntegrations/opentelemetry/best-practices/opentelemetry-otlp-troubleshooting.mdx
@@ -16,9 +16,25 @@ Filing and working through a support case can be time consuming and at times fru
 
 First, please review the New Relic [OTLP configuration requirements / recommendations](/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#configuration). It contains essential advice and context that anyone looking to use OTLP with New Relic should be aware of.
 
+The [General Triaging](#general-triaging) section describes basic troubleshooting steps you should follow when you encounter some issue with OTLP.
+
 The [Issues Catalog](#issue-catalog) lists a variety of different errors we've seen customers experience, with mitigation steps which often reference items from [OTLP configuration requirements / recommendations](/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#configuration).
 
-# Issue Catalog [#issue-catalog]
+## General Triaging
+
+When you encounter an issue with the New Relic OTLP endpoint, first follow these basic troubleshooting steps. If you end up opening a support ticket, these are the first things we ask:
+
+1. **Enable diagnostic logs.** Ensure that your OTLP client has logging enabled such that you can see details about any errors which may be occruing. The mechanism for enabling logs varies by implementation. Please consult relevant docs.
+2. **Test that New Relic's OTLP endpoint is reachable.** A simple shell command `curl http://otlp.nr-data.net` from the machine in question can help determine if some local network configuration issue (i.e. firewall restriction) is preventing connectivity to New Relic.
+3. **Check for [`NrIntegrationErrors`](/docs/telemetry-data-platform/manage-data/nrintegrationerror/).** New Relic OTLP ingest performs minimal validation synchronously before returning a success status code. If you don't see indications of export errors in your application logs, but don't see data in New Relic, try querying for `NrIntegrationErrors`. There may be issues with your data which were detected during asynchronous validation.
+4. **Determine if the problem is localized.** Often errors are localized to a specific application or environment. In these cases, it's useful to evaluate the differences between the areas which are problematic and properly functioning.
+5. **Look for signs of invalid API key.** The New Relic OTLP endoint [requires setting an `api-key` header](/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#api-key). Invalid or missing API keys are a common issue which present with HTTP 403 or 401 status codes, or gRPC Unauthenticated or PermissionDenied status codes. If you see these, check that your API key is valid and is being properly set.
+6. **Check if the export succeeds after retry.** We [recommend that retry is enabled](/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#retry) is enabled, and that occasionally export attempts initially fail with transient errors but succeed after retrying. This is expected. However, we do have an [SLA](/docs/licenses/license-information/referenced-policies/service-level-availability-commitment/) - if you suspect the problem is frequent enough that it exceeds our SLA, please open a support case.
+7. **Check for signs that transient errors are not being retried.** Despite our best efforts, there may be corner cases where the New Relic OTLP endpoint returns non-retriable status codes for transient errors. If you believe you've encountered this scenario, please open a support case.
+
+## Issue Catalog [#issue-catalog]
+
+The table below catalogs issues we've seen customers encounter with the New Relic OTLP endpoint. Many are straight forward to resolve with proper configuration. The "Fingerprint" column shows a typical log when an application encounters the particular class of issue. See the "Known Resolution" and "Notes" columns for mitigation steps.
 
 | OTLP Protocol Version | Type | Language / Ecosystem | Fingerprint | Known Resolution | Notes |
 |---|---|---|---|---|---|

diff --git a/...urce-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp.mdx b/...urce-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp.mdx
@@ -1,5 +1,5 @@
 ---
-title: New Relic OTLP Endpoint Details
+title: New Relic OTLP Endpoint
 tags:
   - Integrations
   - Open source telemetry integrations
@@ -13,14 +13,12 @@ freshnessValidatedDate: never
 
 New Relic supports native OTLP ingest, and recommends OTLP as the means for sending OpenTelemetry data to New Relic. This document describes New Relic OTLP support, including configuration requirements and recommendations.
 
-## Configuration
-
-### Before you start [#prereqs]
+## Config: Before you start [#prereqs]
 
 * If you haven't already done so, sign up for a free [New Relic account](https://newrelic.com/signup).
 * Get the [license key](https://one.newrelic.com/launcher/api-keys-ui.launcher) for the New Relic account to which you want to report data. This license key will be used to [configure `api-key` header](#api-key)
 
-### Endpoint, Port and Protocol [#endpoint-port-protocol]
+## Config: Endpoint, Port and Protocol [#endpoint-port-protocol]
 
 Requirement level: **Required**
 
@@ -211,23 +209,23 @@ If using the collector, prefer the [otlphttpexporter](https://github.com/open-te
   </Collapser>
 </CollapserGroup>
 
-### TLS [#tls]
+## Config: TLS [#tls]
 
 Requirement level: **Required**
 
 In order to send OTLP data to New Relic, you must configure your OTLP exporter to use TLS 1.2 (see [TLS encryption](docs/new-relic-solutions/get-started/networks/#tls) for more information). Generally, SDK and collector exporters meet this requirement by default.
 
 While many OTLP exporters infer TLS settings from the `https` endpoint scheme, some gRPC exporters may require you to explicitly enable TLS. The mechanism to configure gRPC TLS will vary, but OpenTelemetry language SDKs generally support setting the `OTEL_EXPORTER_OTLP_INSECURE=false` environment variable (see [opentelemetry docs](https://opentelemetry.io/docs/specs/otel/protocol/exporter/) for more info).
 
-### Setting the API Key [#api-key]
+### Config: Setting the API Key [#api-key]
 
 Requirement level: **Required**
 
 In order to send OTLP data to New Relic, you must configure your OTLP exporter to include a header named `api-key` with the value set to your [license key](#prereqs). Failure to do so will result in authentication errors.
 
 The mechanism to configure headers will vary, but OpenTelemetry language SDKs generally support setting the `OTEL_EXPORTER_OTLP_HEADERS=api-key=<INSERT_LICENSE_KEY>` environment variable (see [opentelemetry docs](https://opentelemetry.io/docs/specs/otel/protocol/exporter/) for more info).
 
-### Attribute Limits [#attribute-limits]
+## Config: Attribute Limits [#attribute-limits]
 
 Requirement level: **Required**
 
@@ -250,7 +248,7 @@ Notes:
 - Resource attributes are subject to attribute limits, but there are no standard environment variables to limit them. If a resource attribute is over the allowed limit, consider truncating using the collector [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor), or overwriting the resource attribute to another value.
 - There is no standard mechanism to limit attribute names. However, instrumentation generally does not produce attribute names which exceed New Relic limits. If you encounter name length limits, remove the attribute using the collector.
 
-### Payloads Batching, Compression and Rate Limits [#payload]
+## Config: Payloads Batching, Compression and Rate Limits [#payload]
 
 Requirement level: **Required**
 
@@ -274,7 +272,7 @@ The mechanism to configure the endpoint will vary, but OpenTelemetry language SD
 
 If using the collector, `gzip` is the default compression, but `zstd` can be optionally configured.
 
-### Retry [#retry]
+## Config: Retry [#retry]
 
 Requirement level: **Recommended**
 
@@ -284,7 +282,7 @@ The mechanism to configure retry will vary. Some OpenTelemetry SDKs may have lan
 
 If using the collector, the `otlphttpexporter` and `otlpexporter` retry by default. See [exporterhelper](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md) for more details.
 
-### Metric Aggregation Temporality
+## Config: Metric Aggregation Temporality
 
 Requirement level: **Recommended**
 
@@ -297,7 +295,7 @@ The mechanism to configure the endpoint will vary, but OpenTelemetry language SD
 
 Cumulative temporality is used for instruments which map to [New Relic gauge types](https://docs.newrelic.com/docs/data-apis/understand-data/metric-data/metric-data-type/), and which are generally analyzed using the cumulative value.
 
-### Metric Histogram Aggregation
+## Config: Metric Histogram Aggregation
 
 Requirement level: **Recommended**