v1.22.0

Buildkite MCP server v1.22.0 adds tools for listing, inspecting, and creating cluster secrets, along with improved observability for MCP eval runs.

What's Changed

Features

• Add MCP tools for listing and inspecting cluster secrets by @halogenandtoast in #388
• Add an MCP tool for creating cluster secrets by @halogenandtoast in #395

Development and Evals

• Publish per-entry/scenario eval run metrics to Datadog by @nethsix in #386
• docs: add release notes skill by @wolfeidau in #394

Tool Changes

New cluster_secrets toolset:

• list_cluster_secrets lists cluster secret metadata with page-based pagination.
• get_cluster_secret retrieves metadata for a named cluster secret without exposing its write-only value.
• Both read tools require the read_secrets_details scope and remain available in read-only mode.
• create_cluster_secret creates a secret with optional description and access policy fields.
• The create tool requires the write_secrets scope and is excluded from read-only mode.

Eval Changes

• Eval runs now publish per-entry and per-scenario metrics to Datadog, including goal completion, duration, tool calls, wait time, and token usage.
• Metric collection remains separate from publication so LLM-driven eval steps cannot access the Datadog API key.
• Publishing is best-effort and does not fail an eval entry when Datadog is unavailable.

New Contributors

• @halogenandtoast made their first contribution in #388

Full Changelog: v1.21.0...v1.22.0