Skip to content

APM onboarding: Set Up APM + K8s SSI rewrite w/ Agent Execution Context#36370

Draft
brett0000FF wants to merge 10 commits into
masterfrom
brett.blue/apm-onboarding-ia-restructure
Draft

APM onboarding: Set Up APM + K8s SSI rewrite w/ Agent Execution Context#36370
brett0000FF wants to merge 10 commits into
masterfrom
brett.blue/apm-onboarding-ia-restructure

Conversation

@brett0000FF

@brett0000FF brett0000FF commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

What does this PR do? What is the motivation?

Summary: This PR is an experiment showing how docs and agent skills can work together, with docs as the single source of truth for domain knowledge. It restructures the APM onboarding flow, rewrites the K8s SSI page, and incorporates the most valuable content from the dd-apm skills directly into our docs. The agent-only shortcode handles content that helps agents but would clutter the page for humans.

The ICP squad's skills contain great domain knowledge (silent failure modes, verification steps, diagnostic commands, troubleshooting flows) that fills real gaps in our docs. This PR brings that knowledge into the docs pages where it can be reviewed, versioned, and kept current alongside the product content.

Changes follow the Writing Guidelines for AI Readability.

Changes

1. Nav rename: "Application Instrumentation" → "Set Up APM" (config/_default/menus/main.en.yaml)

  • Task-oriented name that matches what users are trying to do

2. Entry page rework (content/en/tracing/trace_collection/_index.md)

  • Transformed from a method-comparison page with feature tables to a task-oriented entry point
  • SSI is the primary path; manual SDKs and custom instrumentation are secondary
  • Added agent-only shortcode with routing context (platform detection, language compatibility)

3. Kubernetes SSI page rewrite (content/en/tracing/trace_collection/single-step-apm/kubernetes.md)

Structural improvements:

  • Three flat tabs for Enable SSI (In-app wizard / Helm / Datadog Operator) instead of h3s with nested tabs
  • Prerequisites split into clean requirements, tracer dependency check, and compatibility notes
  • UST section tightened: recommended method inline, alternatives collapsed
  • "Enable additional products" moved into Advanced SSI configuration
  • Added description to frontmatter for AI retrieval (guideline 1.4)
  • Headings include feature name for chunk context: "Verify SSI is working", "Advanced SSI configuration", "SSI best practices" (guideline 1.1)
  • Removed directional language and ambiguous pronouns (guidelines 1.2, 2.4)
  • PodSecurity callout includes exact error string (guideline 2.8)
  • Java -javaagent added to tracer dependency grep (also checks Dockerfiles, JAVA_TOOL_OPTIONS)

Content incorporated from skills:

Skills source What was incorporated Where it appears
k8s-ssi/enable-ssi Pre-flight grep for existing tracer dependencies that silently disable SSI Prerequisites (human-visible)
k8s-ssi/enable-ssi Node.js ESM restriction, Ruby glibc requirement Compatibility notes (human-visible)
k8s-ssi/troubleshoot-ssi Alpine/musl support on K8s (uses LD_PRELOAD env var, not /etc/ld.so.preload) Compatibility notes (human-visible)
k8s-ssi/troubleshoot-ssi PodSecurity restricted policy blocking with exact error string Enable SSI callout (human-visible)
k8s-ssi/verify-ssi Three-step verification flow (init containers, service visible, traces arriving) Verify SSI is working (human-visible)
k8s-ssi/verify-ssi Send test traffic via port-forward if no automatic traffic Verify SSI is working (human-visible)
k8s-ssi/troubleshoot-ssi Variable resolution table, per-language grep patterns, configuration guardrails agent-only in Prerequisites
k8s-ssi/enable-ssi Recommended default SDK versions agent-only in Enable SSI
k8s-ssi/troubleshoot-ssi Injection chain explanation, pup CLI setup, diagnostic commands, "pup vs kubectl" reasoning agent-only in Verify

Not incorporated because it's agent behavior instructions (how to interact with users, when to execute commands, how to format output), not Datadog product documentation:

  • Interaction choreography ("state your hypotheses", "confirm with user before restarting")
  • Execution markers (### Claude runs)
  • Onboarding summary report template
  • Trigger/routing logic ("invoke this skill when...")

4. Troubleshooting triage section (content/en/tracing/trace_collection/single-step-apm/troubleshooting.md)

  • Added symptom-first "Triage: no traces after enabling SSI?" section before existing content
  • Platform-specific checks: shared (restart, existing deps, runtime version), Kubernetes (namespace, pod selector), Linux (ld.so.preload, musl/glibc, SELinux/AppArmor)
  • Added agent-only shortcode with platform-labeled diagnostics: all-platforms pup commands, Kubernetes kubectl triage with signal table, Linux host checks
  • Updated link references from old alias paths to canonical paths

5. Page descriptions and link cleanup

  • Added description frontmatter to all pages in the onboarding path
  • Updated footnote and further_reading links to canonical paths (preserving aliases for redirects)

How agent-only shortcodes are used

The agent-only shortcode renders content hidden from human readers (display: none) but available to AI agents parsing the page source. Each block has a clear reason for being agent-only rather than human-visible:

Block location What it contains Why it's agent-only
Prerequisites Variable resolution table, pre-flight commands, per-language grep patterns, "never kubectl patch" guardrail Agents need to gather context variables before acting and need more detailed grep patterns than humans need in a prerequisites section. Configuration guardrails prevent agents from making destructive changes.
Enable SSI Tab navigation guidance, recommended SDK version defaults Agents need explicit instructions on which tab to use. SDK version defaults are useful when writing a config file but would clutter the human-visible setup flow.
Verify SSI Injection chain explanation (how SSI works step by step), pup CLI setup and commands, kubectl diagnostic sequences, "pup vs kubectl" reasoning The injection chain helps agents reason about why things fail. For example, understanding that injection happens at pod startup explains why a restart is required. pup is an internal CLI not available to external users. Diagnostic command sequences are procedural detail an agent can execute directly but would overwhelm the human verification steps.
Entry page Platform routing logic, language compatibility check Agents need explicit decision logic ("if Go/C++/Rust, use manual SDKs"). Humans scan the page and click the right link.
Troubleshooting Platform-labeled triage commands, diagnosis-by-signal table, silent failure modes Structured diagnostic tables help agents triage systematically. The signal-to-root-cause mapping is reasoning context that reads as overly technical for most human troubleshooters.

Why a single source of truth matters

While reviewing the skills content, we found that the skills repo recommended python: "2" as the default SDK version, while the live docs and best-practice examples use python: "3". Maintaining domain knowledge in two places creates drift that undermines the agent's accuracy. By incorporating the domain knowledge directly into docs pages, it stays current through the normal review process, and every agent consuming the docs benefits automatically.

Merge instructions

Merge readiness:

  • Ready for merge

Additional notes

This is an experiment for the ICP squad meeting.

Rename Application Instrumentation to Set Up APM in the nav. Rework the
entry page as a task-oriented routing page. Rewrite the Kubernetes SSI
page with a clear onboarding flow, verification steps, agent-only
diagnostic content, and tighter advanced configuration sections. Add a
symptom-first triage section to SSI troubleshooting.
@github-actions github-actions Bot added the Architecture Everything related to the Doc backend label Apr 29, 2026
Per AI readability guidelines, page descriptions in frontmatter are
critical for retrieval. Adds specific, action-oriented descriptions
to the Set Up APM entry page, SSI index page, and SSI troubleshooting
page.
Fix broken link to dynamic instrumentation (use canonical path).
Fix label vs annotation mismatch in troubleshooting triage.
Add build.gradle to troubleshooting grep command.
Fix agent-only tab reference to name all three tabs.
Add opt-out label check to K8s agent-only diagnostic block.
Add shell language tags to grep code blocks.
Expand collapse titles to use full Unified Service Tags name.
Fix Datadog app vale warning.
Add full install-from-scratch steps (Helm repo, namespace, secret,
site/clusterName) to the Helm and Operator tabs so new users don't
need to cross-reference other pages. Add outage warnings before pod
restart commands. Clarify Alpine/musl support nuance (Ruby requires
glibc). Improve agent-only guidance to check existing Agent state
before modifying config.
Ask for language before platform in the APM setup routing block so
unsupported SSI languages (Go, C++, Rust) bail early. Add a node
discovery step before the troubleshooting triage commands so
NODE_HOSTNAME is resolved before it is used.
@brett0000FF brett0000FF changed the title APM onboarding IA restructure: Set Up APM + K8s SSI rewrite APM onboarding: Set Up APM + K8s SSI rewrite w/ Agent Execution Context Apr 30, 2026
Rewrite Alpine/musl bullet as a constraint ("Ruby: glibc only") to
match the pattern of neighboring bullets. Move "already installed"
shortcut before the full YAML blocks in Helm and Operator tabs so
existing users can skip early. Standardize restart warning text
across all three tabs.
1. Add platform-specific triage sections (Kubernetes, Linux) to
   troubleshooting — was K8s-only despite page covering all platforms.
   Added Linux checks: /etc/ld.so.preload, musl/glibc, SELinux/AppArmor.

2. Add dd-java-agent and javaagent to tracer grep commands on K8s and
   troubleshooting pages. Add note about checking Dockerfiles and
   JAVA_TOOL_OPTIONS for Java.

3. Fix stale python: "2" in agent-only SDK version defaults — should be
   python: "3" to match current docs and best-practice examples.

4. Update footnote and further_reading links from old alias paths
   (/automatic_instrumentation/single-step-apm/) to canonical paths
   (/single-step-apm/) on SSI index and troubleshooting pages.
   Preserves aliases in frontmatter for redirects.
Split prerequisites into three clear sections: requirements (bullet
list), tracer dependency check (own heading with code block), and
compatibility notes (clean bullet list). Removes inconsistent bold-
header-paragraph formatting mixed with list items.
The agent-only triage block assumed Kubernetes only. Split into
platform-labeled sections: all-platforms pup commands, Kubernetes
kubectl diagnostics with signal table, and Linux host diagnostics
(ld.so.preload, /proc/maps, libc check, SELinux/AppArmor). Silent
failure modes now labeled by platform.
…arding-ia-restructure

# Conflicts:
#	content/en/tracing/trace_collection/single-step-apm/kubernetes.md
#	content/en/tracing/trace_collection/single-step-apm/troubleshooting.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Architecture Everything related to the Doc backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant