Skip to content

feat(datadog): Use Pup CLI for telemetry workflows #316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dcramer merged 1 commit into from
May 9, 2026
Merged

feat(datadog): Use Pup CLI for telemetry workflows #316

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ Start here:

## Packages

| Package | Purpose |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `@sentry/junior` | Core Slack bot runtime |
| `@sentry/junior-agent-browser` | Agent Browser plugin package for browser automation |
| `@sentry/junior-datadog` | Datadog plugin package for observability workflows (**non-functional**: Datadog has DCR locked down, see the [package README](./packages/junior-datadog/README.md)) |
| `@sentry/junior-github` | GitHub plugin package for issue workflows |
| `@sentry/junior-hex` | Hex plugin package for data warehouse query workflows |
| `@sentry/junior-linear` | Linear plugin package for issue workflows |
| `@sentry/junior-notion` | Notion plugin package for page search workflows |
| `@sentry/junior-sentry` | Sentry plugin package for issue workflows |
| Package | Purpose |
| ------------------------------ | ---------------------------------------------------------------------------- |
| `@sentry/junior` | Core Slack bot runtime |
| `@sentry/junior-agent-browser` | Agent Browser plugin package for browser automation |
| `@sentry/junior-datadog` | Datadog plugin package for observability workflows through Datadog's Pup CLI |
| `@sentry/junior-github` | GitHub plugin package for issue workflows |
| `@sentry/junior-hex` | Hex plugin package for data warehouse query workflows |
| `@sentry/junior-linear` | Linear plugin package for issue workflows |
| `@sentry/junior-notion` | Notion plugin package for page search workflows |
| `@sentry/junior-sentry` | Sentry plugin package for issue workflows |
79 changes: 43 additions & 36 deletions packages/docs/src/content/docs/extend/datadog-plugin.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Datadog Plugin
description: Configure the hosted Datadog MCP server for read-only observability workflows (logs, metrics, traces, monitors, incidents, dashboards).
description: Configure Datadog's Pup CLI for read-only observability workflows (logs, metrics, traces, monitors, incidents, dashboards).
type: tutorial
prerequisites:
- /extend/
Expand All @@ -9,17 +9,11 @@ related:
- /operate/security-hardening/
---

:::danger[This plugin does not currently work]
Datadog's hosted MCP server requires OAuth Dynamic Client Registration ([DCR, RFC 7591](https://www.rfc-editor.org/rfc/rfc7591)) for third-party clients like Junior, and **DCR is locked down on Datadog's side**. Until Datadog exposes DCR (or an equivalent registration path) on `mcp.datadoghq.com`, Junior cannot complete the OAuth handshake and every Datadog tool call will fail at connect time.
The Datadog plugin installs Datadog's Pup CLI so Slack users can query Datadog telemetry from Junior: logs, metrics, APM traces/spans, monitors, incidents, dashboards, hosts, services, and RUM.

The `@sentry/junior-datadog` package is kept in-tree so the integration is ready to ship the moment Datadog unblocks DCR. **Do not add it to a production deployment in the meantime.** The rest of this page documents how the plugin will behave once Datadog unblocks DCR.
:::
Junior intentionally keeps this plugin read-only. The packaged manifest sets Pup read-only mode and the bundled skill instructs Junior to run `pup --read-only --agent` commands. It is for search, fetch, and analytics workflows, not Datadog mutations.

The Datadog plugin uses Datadog's hosted MCP server so Slack users can query their own Datadog account context — logs, metrics, APM traces, monitors, incidents, dashboards, and RUM — without sharing a workspace API key.

Junior intentionally keeps this plugin read-only. The packaged manifest exposes only search-, fetch-, and analytics-oriented Datadog MCP tools. It does not expose notebook, monitor, SLO, or incident mutations, even though Datadog's MCP server supports some of them.

The packaged plugin defaults to Datadog's US1 endpoint and enables the `core`, `apm`, and `error-tracking` toolsets. A Junior deployment points at exactly one Datadog org (which is pinned to exactly one region), so the site is a deployment-level setting, not a per-user or per-channel one. Operators on other sites select their site with the `DATADOG_SITE` env var — see [Non-US1 sites](#non-us1-sites) below.
The packaged plugin defaults to Datadog's US1 endpoint. A Junior deployment points at one Datadog org/site, so the site is a deployment-level setting, not a per-user or per-channel one. Operators on other sites select their site with the `DATADOG_SITE` env var; see [Non-US1 sites](#non-us1-sites).

## Install

Expand All @@ -39,6 +33,18 @@ juniorNitro({
});
```

Set Datadog credentials in your Junior deployment environment:

```bash
DATADOG_API_KEY=...
DATADOG_APP_KEY=...
DATADOG_SITE=datadoghq.com # optional; defaults to US1
```

Use `DATADOG_API_KEY`, `DATADOG_APP_KEY`, and `DATADOG_SITE` in the Junior deployment environment. The plugin maps those host-side `DATADOG_*` values to Datadog API headers and Pup's sandbox `DD_*` env values.

Use a Datadog application key with the smallest read scopes/role that covers the telemetry users need.

## Optional channel defaults

If a Slack channel usually investigates the same Datadog environment or service, store that as a conversation-scoped default:
Expand All @@ -52,35 +58,37 @@ These defaults are optional fallbacks. If a user names a different env or servic

## Auth model

- No `DD_API_KEY`, `DD_APP_KEY`, or shared workspace integration secret is required.
- Each user completes OAuth the first time Junior calls a Datadog MCP tool on their behalf.
- Junior sends the authorization link privately, then resumes the same thread automatically after the user authorizes.
- Datadog MCP requires user-based OAuth (OAuth 2.1 + PKCE) and does not accept shared bearer tokens here, so this plugin is not suitable for fully headless automation.
- The plugin uses deployment-level Datadog API and application keys, not per-user OAuth.
- Junior keeps the real `DATADOG_API_KEY` and `DATADOG_APP_KEY` values host-side.
- Matching Datadog API requests from Pup receive host-managed `DD-API-KEY` and `DD-APPLICATION-KEY` headers.
- The sandbox receives only non-secret placeholder env values so Pup can perform its normal credential checks before making requests.
- Users do not connect or disconnect individual Datadog accounts from Junior App Home for this plugin.

## What users can do

- Search logs, events, RUM sessions, spans, and hosts scoped by env/service/time window.
- Run SQL-style log analytics (counts, top-N, group-bys) with `analyze_datadog_logs`.
- Search raw logs and aggregate log counts/top-N buckets.
- Search spans and aggregate latency/error buckets.
- Query metrics, find metric names, and inspect metric metadata/tag dimensions.
- Inspect monitors and incidents to answer "is this alerting?" and "what is INC-123?".
- Fetch a trace or a notebook by ID.
- List services and their upstream/downstream dependencies from the Software Catalog.
- Query a metric by name and inspect its available tag dimensions before querying.
- Disconnect their account later from Junior App Home with `Unlink`.
- List APM services and service dependencies.
- List hosts and inspect host details.
- Fetch dashboards and notebooks by ID.
- Query RUM events, sessions, and frontend aggregates.

## Non-US1 sites

Datadog customers are region-pinned. The packaged manifest declares `DATADOG_SITE` in its `env-vars` block with a default of `datadoghq.com` (US1) and references it from `mcp.url`:
Datadog customers are region-pinned. The packaged manifest declares `DATADOG_SITE` with a default of `datadoghq.com` (US1), then exposes it to Pup as `DD_SITE`:

```yaml
env-vars:
DATADOG_SITE:
default: datadoghq.com

mcp:
url: https://mcp.${DATADOG_SITE}/api/unstable/mcp-server/mcp?toolsets=core,apm,error-tracking
command-env:
DD_SITE: ${DATADOG_SITE}
```

Set `DATADOG_SITE` in your Junior deployment env (e.g. Vercel project settings) to the hostname portion of your Datadog site:
Set `DATADOG_SITE` in your Junior deployment env (for example Vercel project settings) to the hostname portion of your Datadog site:

| Datadog site | `DATADOG_SITE` value |
| ------------ | ------------------------------------ |
Expand All @@ -92,27 +100,26 @@ Set `DATADOG_SITE` in your Junior deployment env (e.g. Vercel project settings)
| AP2 | `ap2.datadoghq.com` |
| GovCloud | `ddog-gov.com` |

No code changes, no app-local plugin copy, no rebuild. Junior reads the variable at plugin-discovery time and hits the correct regional MCP endpoint.
The packaged API allowlist covers those standard Datadog sites. Custom or staging Datadog domains require a manifest change so Junior is allowed to inject headers for that host.

## Verify

Confirm a real user can connect and query successfully:
Confirm Junior can query Datadog successfully:

1. Ask Junior a Datadog question in a channel, for example: `What monitors are alerting for service checkout in prod right now?`
2. Complete the private OAuth flow when Junior prompts for it.
3. Confirm the thread resumes automatically with the monitor state (or incident / log / trace detail) and a Datadog deep link.
4. Open Junior App Home and confirm Datadog appears under `Connected accounts`.
2. Confirm the thread returns monitor state, incident/log/trace detail, or a clear "no results" answer.
3. Confirm the answer includes the query/time window used and a Datadog deep link when one is available.

## Failure modes

- No auth prompt or no resume: the user still needs to complete the OAuth flow. Retry the request and finish the private authorization flow when prompted.
- `401` mid-session: the Datadog OAuth token expired or was revoked; the runtime will resurface the authorization flow. Finish it and retry.
- `403 Forbidden` or `permission denied`: the user's Datadog role cannot read the requested resource. Verify their Datadog team/role assignments.
- `429 Too Many Requests`: the Datadog MCP endpoint is throttling. Junior retries once. If it still fails, the user should retry again shortly.
- `DATADOG_API_KEY` or `DATADOG_APP_KEY` missing: add both env vars to the Junior deployment and redeploy.
- `401 Unauthorized`: the API key or application key is invalid, revoked, or not being injected for the selected Datadog site.
- `403 Forbidden` or `permission denied`: the Datadog application key cannot read the requested resource. Verify its scopes/role.
- `429 Too Many Requests`: Datadog is throttling. Retry the request later or narrow the query.
- Empty query results: env/service tag values are case-sensitive. Confirm the tag values exist and try a wider time window before widening the filter.
- Truncated trace response: very large traces are reported as truncated; the displayed spans are not the full trace.
- Mutation requests (create notebook, edit monitor, resolve incident): the plugin intentionally does not expose write tools. The skill will decline these.
- Wrong Datadog site: the packaged manifest defaults to US1. Operators on other sites must set `DATADOG_SITE` in the deployment env (see [Non-US1 sites](#non-us1-sites)).
- Partial span/trace output: Pup exposes span search; a trace ID search may not prove that every span in the trace was returned.
- Mutation requests (create notebook, edit monitor, submit metric, resolve incident): the plugin is read-only and the skill will decline these.
- Wrong Datadog site: set `DATADOG_SITE` in the deployment env (see [Non-US1 sites](#non-us1-sites)).

## Next step

Expand Down
35 changes: 30 additions & 5 deletions packages/docs/src/content/docs/extend/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,27 +159,28 @@ runtime-postinstall:
- `capabilities`: actions the plugin’s skills may request, qualified as `<plugin>.<capability>`
- `config-keys`: provider-specific configuration keys, qualified as `<plugin>.<key>`
- `api-domains` and `api-headers`: optional host-managed HTTP headers injected for matching sandbox requests
- `command-env`: optional non-secret sandbox env vars injected when provider credentials or API headers are enabled; use it for CLI placeholders and deployment defaults
- `credentials`: how token auth is delivered to tools; current types are `oauth-bearer` and `github-app`
- `oauth`: user OAuth setup; use it with `credentials.type: oauth-bearer`
- `target`: optional credential target scope tied to a declared config key
- `runtime-dependencies`: sandbox dependencies required by the plugin’s tools
- `runtime-postinstall`: commands that run after dependency install and before snapshot capture
- `mcp`: optional MCP server configuration for provider-scoped tool sources; `mcp.url` implies hosted HTTP transport, so `mcp.transport: http` is optional
- `env-vars`: optional map of deployment env vars the manifest may reference from `mcp.url` or `api-headers`. Each key names an env var (uppercase, `[A-Z_][A-Z0-9_]*`) and may declare a `default` for `mcp.url`; API header references cannot use defaults.
- `env-vars`: optional map of deployment env vars the manifest may reference from `mcp.url`, `api-headers`, or `command-env`. Each key names an env var (uppercase, `[A-Z_][A-Z0-9_]*`) and may declare a `default` for `mcp.url` and `command-env`; API header references cannot use defaults.
- `mcp.url`: supports `${VAR}` placeholders that must be declared in `env-vars`. This lets region-pinned providers pick the right host at deploy time without a manifest fork.
- `mcp.allowed-tools`: optional raw MCP tool-name allowlist when a plugin should expose only part of a provider's tool surface

### Env-var expansion in `mcp.url`

Some providers (Datadog, Sentry self-hosted, GitHub Enterprise, Linear EU, ...) have different hostnames per region or deployment. The packaged plugin manifest keeps a single `mcp.url` and declares the deployment-level env vars it may read in an `env-vars` block. Defaults live in the declaration, not inline in the URL:
Some providers (Sentry self-hosted, GitHub Enterprise, Linear EU, ...) have different hostnames per region or deployment. The packaged plugin manifest keeps a single `mcp.url` and declares the deployment-level env vars it may read in an `env-vars` block. Defaults live in the declaration, not inline in the URL:

```yaml
env-vars:
DATADOG_SITE:
default: datadoghq.com
EXAMPLE_SITE:
default: example.com

mcp:
url: https://mcp.${DATADOG_SITE}/api/unstable/mcp-server/mcp?toolsets=core,apm,error-tracking
url: https://mcp.${EXAMPLE_SITE}/mcp
```

The only supported placeholder form is `${NAME}` — replaced with `process.env[NAME]`, falling back to the declared `default`. Plugin discovery fails loudly at load time if `NAME` is not listed in `env-vars`, or if it is listed without a default and the env var is unset.
Expand Down Expand Up @@ -212,6 +213,30 @@ api-headers:
X-Api-Version: "2026-01-01"
```

### Command env

Use top-level `command-env` when a sandbox CLI needs non-secret env vars. This is commonly used for placeholder auth env vars so the CLI proceeds to make HTTP requests while Junior injects the real credentials as host-managed headers.

`command-env` values may be literals or `${NAME}` placeholders declared in `env-vars`. Referenced env vars must declare defaults, because command env values are visible inside the sandbox and must not depend on secret deployment env vars.

Manifests with `command-env` must also declare `credentials` or `api-headers`, since command env is delivered with the provider credential lease.

```yaml
env-vars:
EXAMPLE_AUTH_HEADER:
EXAMPLE_SITE:
default: example.com

api-domains:
- api.example.com
api-headers:
Authorization: ${EXAMPLE_AUTH_HEADER}

command-env:
EXAMPLE_API_KEY: host_managed_credential
EXAMPLE_SITE: ${EXAMPLE_SITE}
```

### Add skills to the plugin

Put at least one skill under `skills/<skill-name>/SKILL.md`. Provider config keys belong in `plugin.yaml`, not in skill frontmatter.
Expand Down
Loading
Loading