chore: enhance smoke test workflow to include datadog validation

[mnkiefer]

chore: enhance smoke test workflow to include datadog validation

Summary

Extends the OTEL smoke test infrastructure to validate Datadog as a
third observability backend alongside Sentry and Grafana. Adds a
dedicated Datadog MCP query step to the smoke workflow, wires in the
official Datadog MCP server, and introduces a new shared
otlp-backends.md import that consolidates OTLP export configuration
for all three backends.

---

What changed and why

.github/workflows/smoke-otel-backends.md (modified — high impact)

• Imports the Datadog MCP server config alongside the existing Sentry
  and Grafana configs.
• Switches from shared/otlp.md to the new shared/otlp-backends.md
  import, which fans out OTLP export to all three backends in one step.
• Adds Step 4 — a Datadog span/trace query step — that produces a
  datadog_status result.
• Folds datadog_status into the final pass/fail verdict, so a
  Datadog failure now gates the smoke run.
• Updates the workflow description to reflect the three-backend scope.

.github/workflows/shared/otlp-backends.md (added — high impact)

• New shared workflow import that configures OTLP export to Sentry,
  Grafana, and Datadog in a single, reusable block.
• Each backend is wired with its own endpoint secret and authorization
  secret, replacing the narrower single-backend shared/otlp.md import.

.github/workflows/shared/mcp/datadog.md (modified — medium impact)

• Switches the Datadog MCP server to the official endpoint with the
  core toolset pinned.
• Replaces the application key secret reference with an
  || secrets.DD_APP_KEY fallback to align with the DD_APP_KEY /
  DD_APPLICATION_KEY dual-naming convention used across the repo.
• Adds search_datadog_spans and get_datadog_trace to the allowed
  tool list.
• Refreshes all inline documentation to match the official server's
  tool surface.

---

Key commits

SHA	Description
bcd3fc993	Initial smoke workflow Datadog enhancement
83f6c43f0	Align Datadog MCP header with DD_APPLICATION_KEY
5ae9c2647	Address remaining workflow review comments
56d148e63	Switch smoke Datadog secret to DD_APP_KEY; recompile
6fc195679	Fix Datadog auth and OTLP fallbacks in smoke workflow

---

Risk & compatibility

Area	Assessment
Breaking change	No
Secrets required	DD_APP_KEY / DD_APPLICATION_KEY must be present for the Datadog step to pass
Existing backends	Sentry and Grafana paths are unchanged; only additive changes
Pass/fail gate	datadog_status is now part of the verdict — workflows without the Datadog secret configured will fail the smoke run

---

Reviewer focus areas

1. shared/otlp-backends.md — verify all three backend endpoint/auth
   secret names are correct for the target environment.
2. smoke-otel-backends.md Step 4 — confirm the Datadog query logic
   and the way datadog_status is folded into the verdict matches the
   pattern used for sentry_status / grafana_status.
3. mcp/datadog.md fallback — confirm the || secrets.DD_APP_KEY
   precedence order is intentional (prefer DD_APPLICATION_KEY, fall
   back to DD_APP_KEY).

Generated by PR Description Updater for issue #33023 · ● 3.2M · ◷

[Copilot]

Pull request overview

Enhances the Smoke OTEL workflow to validate Datadog alongside Sentry and Grafana.

Changes:

• Adds Datadog OTLP and MCP configuration to the smoke workflow.
• Expands smoke-test instructions, status logic, and reporting table to include Datadog.
• Updates generated lock workflows and shared Datadog/OTLP imports.

Show a summary per file

File	Description
.github/workflows/smoke-otel-backends.md	Adds Datadog validation steps and reporting guidance.
.github/workflows/smoke-otel-backends.lock.yml	Regenerates the runnable workflow with Datadog secrets, MCP server config, and allowed domains.
.github/workflows/shared/otlp-sentry-grafana-datadog.md	Adds shared OTLP configuration for Sentry, Grafana, and Datadog.
.github/workflows/shared/mcp/datadog.md	Updates Datadog MCP server configuration and allowed tools.
.github/workflows/mcp-inspector.lock.yml	Regenerates inspector lock output for updated Datadog MCP configuration.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 5/5 changed files
• Comments generated: 3

[github-actions]

Skills-Based Review 🧠

Applied /zoom-out — this PR extends an existing two-backend pattern (Sentry + Grafana) to a third backend (Datadog), so the key question is whether the new pieces fit consistently with what's already there.

Key Themes

• Docs/config mismatch in datadog.md: The prose section says "Allowed tools in this shared import" and lists 6 tools, but the allowed YAML config only contains 2. The documentation overstates the permitted tool surface.
• Dead secret entry: DD_SITE (optional) appears in the secrets list but is never interpolated into any config, so its purpose is unclear.
• Underspecified verdict for inconclusive: Datadog introduces a third status value (inconclusive) that has no explicit mapping in the Step 5 final verdict rule.
• Backend-encoded filename: otlp-sentry-grafana-datadog.md will need renaming if a fourth backend is added; a generic name would be more durable.

Positive Highlights

• ✅ Clean, consistent extension of the Sentry/Grafana pattern — the Step 4 structure mirrors the existing steps well.
• ✅ The inconclusive status concept is well-reasoned (ingestion delay vs broken path) and the fallback 24-hour query is a practical heuristic.
• ✅ Good use of the ?toolsets=core URL parameter to keep the Datadog tool surface narrow.
• ✅ The updated datadog.md switches to the official Datadog MCP endpoint and corrects the secret name (DD_APP_KEY vs the old DD_APPLICATION_KEY).

Verdict

No merge-blocking issues — the three inline comments are worth addressing before the workflow runs in production, especially the inconclusive verdict gap and the doc/config mismatch in datadog.md.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · ● 15.6M

[mnkiefer]

@copilot Address all remaining review comments by github-actions. Then recompile workflow.

[github-actions]

✅ smoke-ci: safeoutputs CLI comment + comment-memory run (26033889688)

Generated by 🧪 Smoke CI for issue #33023 · ◷

[Copilot]

@copilot Address all remaining review comments by github-actions. Then recompile workflow.

Done in 5ae9c26 and workflow recompiled. I addressed the remaining github-actions feedback by switching to shared/otlp-backends.md, updating imports/lock output, and removing DD_SITE from the smoke workflow’s secrets list.

[mnkiefer]

@copilot DD_APPLICATION_KEY should be DD_APP_KEY. Then recompile workflow

[Copilot]

@copilot DD_APPLICATION_KEY should be DD_APP_KEY. Then recompile workflow

Updated in 56d148e and recompiled. The Datadog key is now DD_APP_KEY in the shared MCP import, the smoke workflow required secrets list, and the regenerated lock file.

[mnkiefer]

@copilot Address datadog issues found in run https://github.com/github/gh-aw/actions/runs/26035293666

[Copilot]

@copilot Address datadog issues found in run https://github.com/github/gh-aw/actions/runs/26035293666

Addressed in 6fc1956. I investigated run 26035293666 logs and fixed the Datadog failures by restoring DD_APPLICATION_KEY auth handling (with DD_APP_KEY fallback) and adding Datadog OTLP fallbacks in shared/otlp-backends.md, then recompiled the smoke workflow.

[github-actions]

@copilot confirm the latest smoke workflow is green after the Datadog validation fix.

Generated by 👨‍🍳 PR Sous Chef · ◷