docs: add distributed tracing section to production deployment guide

[ankurs]

Summary

The production deployment guide had no dedicated tracing section — just a one-line mention in the checklist. Add comprehensive coverage:

• OTLP backend setup (Jaeger, Tempo, Honeycomb, etc.) with env var examples
• New Relic OpenTelemetry configuration
• Table of what gets traced automatically (gRPC RPCs, HTTP, custom spans)
• Sampling ratio recommendations by QPS tier
• Parent-based sampling note
• OTLP auth headers in secrets example

Test plan

• Verify page renders correctly on Jekyll site

Summary by CodeRabbit

Documentation

• Production Deployment Guide: Added comprehensive guide covering Kubernetes manifests, environment configuration, health probes, and graceful shutdown procedures.
• Distributed Tracing: Added setup documentation for OpenTelemetry/OTLP and New Relic with configuration examples.
• Sampling Guidance: Added production sampling recommendations with performance metrics.
• Navigation Updates: Updated site links to reference the new deployment guide.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b5f06948-6d80-4557-912c-b5dd53b6c66c

📥 Commits

Reviewing files that changed from the base of the PR and between 3cb4226 and 2bffed9.

📒 Files selected for processing (3)

• Index.md
• howto/production.md
• quickstart.md

---

📝 Walkthrough

Walkthrough

Documentation extended with a new production deployment guide covering Kubernetes setup, health probes, distributed tracing via OpenTelemetry/OTLP and New Relic, and graceful shutdown. Navigation links to this guide were added to index and quickstart pages.

Changes

Cohort / File(s)	Summary
Navigation Updates Index.md, quickstart.md	Added links to new Production Deployment how-to page in the "Next Steps" sections.
Production Deployment Guide howto/production.md	Added comprehensive distributed tracing configuration documentation covering OTLP (with auth headers), New Relic integration, tracing spans, and production sampling guidance with QPS-to-ratio table. Extended Kubernetes Secret example with OTLP_HEADERS entry.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

• docs: update for OTEL migration — remove OpenTracing references #44: Covers OTEL/OTLP configuration and OpenTracing migration, directly related to the distributed tracing configuration documentation added here.

Poem

🐰 Hops through the clouds of traces so fine,
OTLP and New Relic in perfect align,
Production thrives with Kubernetes grace,
Sampling wisely keeps pace,
Documentation now shines! 🌟

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch docs/production-tracing

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

Adds a dedicated distributed tracing section to the Production Deployment guide so operators have concrete OTLP/New Relic setup guidance alongside existing production Kubernetes recommendations.

Changes:

• Updates page metadata/description to include distributed tracing.
• Extends the example Kubernetes Secret with OTLP_HEADERS (for OTLP backend auth headers).
• Adds a new “Distributed tracing” section covering OTLP + New Relic configuration, traced span sources, and production sampling guidance.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The “Recommended ratio” cell uses inline-code to show a range (0.001–0.01). Since OTLP_SAMPLING_RATIO is a single float, readers may copy/paste the range literally and end up with an invalid value. Consider expressing this as plain text (e.g., “0.001 to 0.01”) or splitting into min/max guidance.

Suggested change

	| 70,000+ | `0.001–0.01` | 70–700 |
	| 70,000+ | 0.001 to 0.01 | 70–700 |

[Copilot]

This example includes an API key inline in the manifest (OTLP_HEADERS: "x-honeycomb-team=your-api-key"). Elsewhere on this page you recommend putting secrets in Kubernetes Secrets; consider showing valueFrom.secretKeyRef here (or explicitly noting the header value should come from a Secret) to avoid encouraging in-repo/plaintext configs.

Suggested change

	value: "x-honeycomb-team=your-api-key"
	valueFrom:
	secretKeyRef:
	name: myservice-secrets
	key: OTLP_HEADERS

[Copilot]

The text says “New Relic tracing … can run alongside OTLP”, but the config reference states that when OTLP_ENDPOINT is set it takes precedence over New Relic OpenTelemetry configuration. This is internally inconsistent and could confuse users about whether traces are dual-exported. Please clarify the precedence/interaction here (or link to the canonical behavior) so the guide matches the config reference.

Suggested change

	New Relic tracing is configured separately and can run alongside OTLP:
	New Relic tracing is configured separately. It can be enabled in the same service as OTLP, but when `OTLP_ENDPOINT` is set it takes precedence over the New Relic OpenTelemetry exporter, so traces are not dual-exported. See the [config reference](/config-reference#distributed-tracing) for details: