Skip to content

Commit

Permalink
Merge pull request #6715 from brnhensley/patch-35
Browse files Browse the repository at this point in the history
docs: add limit attr key/value length
  • Loading branch information
zuluecho9 committed Mar 23, 2022
2 parents 54dea8a + e825af8 commit f1971ce
Show file tree
Hide file tree
Showing 5 changed files with 113 additions and 47 deletions.
57 changes: 29 additions & 28 deletions src/content/docs/data-apis/manage-data/view-system-limits.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -151,86 +151,77 @@ The following table includes general max limits that apply across all New Relic
1KB (10^3 bytes)
</td>
</tr>

<tr>
<td>
Max payload size
Max attribute name length
</td>

<td>
1MB (10^6 bytes)
255 characters
</td>
</tr>

<tr>
<td>
Max total attributes per data type (including default attributes)
Max attribute value length in characters
</td>

<td>
254 (less for some tools; for example, 64 for agents)
Metric, Trace, and Event API: 4096
Log API: 4094
</td>
</tr>

<tr>
<td>
Number of unique custom [data types](/docs/telemetry-data-platform/ingest-manage-data/understand-data/new-relic-data-types)
Max payload size
</td>

<td>
250 per account per day (applies to [custom events](/docs/insights/insights-data-sources/custom-data/report-custom-event-data) because that's source of new data types)
1MB (10^6 bytes)
</td>
</tr>

<tr>
<td>
APM limits
Max total attributes per data type (including default attributes)
</td>

<td>
* Agent instances: 50K per account
* Agent instances per app: 10K
* APM apps/services: 10K per second
254 (less for some tools; for example, 64 for agents)
</td>
</tr>

<tr>
<td>
Browser: number of page views
Number of unique custom [data types](/docs/telemetry-data-platform/ingest-manage-data/understand-data/new-relic-data-types) per account per [namespace](/docs/data-apis/manage-data/manage-data-retention) per day
</td>

<td>
1M per minute per app
250 (primarily a limit for [custom events](/docs/insights/insights-data-sources/custom-data/report-custom-event-data))
</td>
</tr>

<tr>
<td>
Distributed tracing: Max age of span timestamp values
APM limits
</td>

<td>
20 minutes. Timestamp must be within 20 minutes of current time at ingest or within 20 minutes from the time the last span with the same `trace.id` was received by New Relic.
* Agent instances: 50K per account
* Agent instances per app: 10K
* APM apps/services: 10K per second
</td>
</tr>

<tr>
<td>
Distributed tracing: Max spans per trace
</td>

<td>
50K
</td>
</tr>

<tr>
<td>
Distributed tracing: Max attributes per span
Browser: number of page views
</td>

<td>
200
1M per minute per app
</td>
</tr>

Expand Down Expand Up @@ -268,6 +259,16 @@ The following table includes general max limits that apply across all New Relic
</td>
</tr>

<tr>
<td>
Distributed tracing
</td>

<td>
See [Trace limits](/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works).
</td>
</tr>

<tr>
<td>
Query limits
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: How New Relic distributed tracing works
title: Technical distributed tracing details
tags:
- Understand dependencies
- Distributed tracing
- Get started
metaDescription: 'How New Relic''s distributed tracing feature works, including explanation of sampling, trace data structure, and trace storage.'
metaDescription: "Technical details of New Relic's distributed tracing, including limits, explanation of sampling, trace data structure, and trace storage."
redirects:
- /docs/explanation-how-sampling-works-distributed-tracing-data
- /docs/apm/distributed-tracing/ui-data/explanation-how-sampling-works-distributed-tracing-data
Expand All @@ -14,16 +14,88 @@ redirects:
- /docs/distributed-tracing/get-started/how-new-relic-distributed-tracing-works
---

Here are some technical details about how New Relic [distributed tracing](/docs/understand-dependencies/distributed-tracing/get-started/introduction-distributed-tracing) works:
Here are some technical details about how New Relic distributed tracing works:

* [Trace limits](#limits)
* [How trace sampling works](#sampling)
* [How we structure trace data](#trace-structure)
* [How we store trace data](#trace-storage)
* [How trace context is passed between applications](#headers)

<Callout variant="tip">
For instructions about setting up distributed tracing, see [Overview: Enable distributed tracing](/docs/understand-dependencies/distributed-tracing/enable-configure/overview-enable-distributed-tracing).
</Callout>
## Trace limits [#limits]

Here are some trace-related limits:

<table>
<thead>
<tr>
<th style={{ width: "180px" }}>
Limited aspect
</th>

<th>
Limit
</th>
</tr>
</thead>
<tbody>
<tr>
<td>
Max age of span timestamp values
</td>

<td>
20 minutes. Timestamp must be within 20 minutes of current time at ingest or within 20 minutes from the time the last span with the same `trace.id` was received by New Relic.
</td>
</tr>

<tr>
<td>
Max spans per trace
</td>

<td>
50K
</td>
</tr>

<tr>
<td>
Max attributes per span
</td>

<td>
200
</td>
</tr>

<tr>
<td>
Max spans per minute per APM agent instance
</td>

<td>
Go, PHP, Python: 1000
Other agents: 2000
</td>
</tr>
</tbody>
</table>

For more rules related to using the Trace API, see [Trace API requirements and limits](/docs/distributed-tracing/trace-api/trace-api-general-requirements-limits).

For other New Relic limits, see [Limits](/docs/data-apis/manage-data/view-system-limits#limits-ui).

### Exceeding trace limits [#exceed-limits]

When you exceed your span rate limit, an [`NrIntegrationError` event](/docs/telemetry-data-platform/manage-data/nrintegrationerror) is generated. You can get rate limit messages with this NRQL query:

```
SELECT * FROM NrIntegrationError WHERE newRelicFeature = 'Distributed Tracing' AND category = 'RateLimit' AND rateLimitType = 'SpansPerMinute'
```

To get a notification when you exceed the limit, you can set up a [NRQL alert](/docs/alerts/new-relic-alerts/defining-conditions/create-alert-conditions-nrql-queries).


## Trace sampling [#sampling]

Expand Down Expand Up @@ -52,7 +124,7 @@ Here are some details about how head-based sampling is implemented in our standa

The first service we monitor in a distributed trace is called the trace origin. The trace origin chooses requests at random to be traced. That decision propagates to the downstream services touched by that request. When the request has completed, all of the spans touched by that request that we've detected are made available in the UI as a complete trace (though [agent limits](#agent-limits-sampling) may result in fragmented traces).

APM agents have a limit on the number of transactions collected per minute (this can vary, depending on agent) and a limit on the number of spans collected per minute (1000 per agent instance). To adhere to these limits, the default number of traces at the trace origin is 10 traces per minute.
APM agents have a limit on the number of transactions collected per minute (this can vary, depending on agent) and a limit on the [number of spans collected per minute per agent instance](/docs/data-apis/manage-data/view-system-limits/#all_products). To adhere to these limits, the default number of traces at the trace origin is 10 traces per minute.

An APM agent spreads out the collection of these 10 traces over a minute in order to get a representative sample over time. The exact sampling rate depends on the number of transactions in the previous minute. The rate responds to changes in transaction throughput, going up or down.

Expand All @@ -63,7 +135,7 @@ Here are some details about how head-based sampling is implemented in our standa
id="agent-limits-sampling"
title="Language agents: limits and sampling"
>
An APM language agent instance using head-based sampling has a limit of 1000 spans per minute. The agent attempts to keep all spans that are marked to be sampled as part of a distributed trace.
An APM agent instance using head-based sampling has a maximum limit on [the number of spans per minute per agent instance](/docs/data-apis/manage-data/view-system-limits/#all_products). The agent attempts to keep all spans that are marked to be sampled as part of a distributed trace.

In many distributed systems, the average microservice may generate 10 to 20 spans per request. In those cases, the agent span limit can accommodate all spans chosen, and that service will have full detail in a trace.

Expand Down Expand Up @@ -207,7 +279,7 @@ Some of our tools don't use sampling. Sampling details for these tools:
id="browser-spans"
title="Browser and mobile trace reporting"
>
[Browser monitoring distributed tracing](/docs/browser/new-relic-browser/browser-pro-features/browser-data-distributed-tracing) and [mobile monitoring](/docs/mobile-monitoring/new-relic-mobile-android/get-started/new-relic-mobile-and-dt/) report all spans.
[Browser monitoring distributed tracing](/docs/browser/new-relic-browser/browser-pro-features/browser-data-distributed-tracing) and [mobile monitoring](/docs/mobile-monitoring/new-relic-mobile-android/get-started/new-relic-mobile-and-dt) report all spans.

Our APM language agents are often used in conjunction with browser and mobile monitoring, and our language agents [use sampling](#trace-origin-sampling). This means that there will likely be many more browser and mobile spans than back-end spans, which can result in browser and mobile app spans disconnected from back-end spans. For tips on querying for traces that contain front and back-end spans, see [Find browser span data](/docs/browser/new-relic-browser/browser-pro-features/browser-data-distributed-tracing#find-data).
</Collapser>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -448,12 +448,5 @@ traceId = <var>TRACE_ID_SENT</var>

## Data limits [#data-limits]

Distributed tracing rate limits are set per account and data type. For details about data limits, see [New Relic data usage limits and policies](/docs/licenses/license-information/general-usage-licenses/new-relic-data-usage-limits-policies/#all_products).
For trace-related limits, see [How distributed tracing works](/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works).

When you exceed your span rate limit, an [`NrIntegrationError` event](/docs/telemetry-data-platform/manage-data/nrintegrationerror) is generated. You can query rate limit messages with this NRQL:

```
SELECT * FROM NrIntegrationError WHERE newRelicFeature = 'Distributed Tracing' AND category = 'RateLimit' AND rateLimitType = 'SpansPerMinute'
```

To get a notification when you exceed the limit, you can set up a [NRQL alert](/docs/alerts/new-relic-alerts/defining-conditions/create-alert-conditions-nrql-queries).
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Not all log data in a message or for a specific attribute is being displayed. Th

## Solution

This occurs because the logs datastore in New Relic limits the field length to 4,096 characters. Any data longer than that is truncated during ingestion.
This occurs because the logs datastore in New Relic limits the field length to 4,094 characters. Any data longer than that is truncated during ingestion.

If you have values exceeding the character limit, here are some options to try:

Expand Down
2 changes: 1 addition & 1 deletion src/nav/distributed-tracing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ pages:
path: /docs/distributed-tracing/concepts/distributed-tracing-planning-guide
- title: Begin the setup
path: /docs/distributed-tracing/concepts/quick-start
- title: How distributed tracing works
- title: Technical details
path: /docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works
- title: Infinite Tracing
pages:
Expand Down

0 comments on commit f1971ce

Please sign in to comment.