feat: add agent discovery features

[JakeSCahill]

Summary

Implements agent discovery features to improve our score on https://isitagentready.com/docs.redpanda.com

Changes

• Link Headers: Added RFC 8288 Link headers on homepage pointing to API catalog, MCP server card, and agent skills
• Content-Signal: Added Content-Signal directive to robots.txt declaring AI usage preferences (ai-train, search, ai-input)
• API Catalog: Created RFC 9727 compliant API catalog at /.well-known/api-catalog listing all 5 Redpanda APIs
• MCP Server Card: Added MCP server card at /.well-known/mcp/server-card.json for agent discovery
• Agent Skills: Created agent skills index with SHA256 digests and skill definitions
• Bug Fix: Fixed URLSearchParams bug in feedback API

Testing

All endpoints tested locally with Netlify CLI:

• ✅ Link headers on homepage
• ✅ API catalog endpoint
• ✅ MCP server card endpoint
• ✅ Agent skills index with dynamic SHA256 digests
• ✅ Skill markdown files
• ✅ Content-Signal in robots.txt configuration

Impact

These changes improve agent readiness by implementing RFC-compliant discovery mechanisms for AI agents, search engines, and automated tools.

[netlify]

✅ Deploy Preview for redpanda-documentation ready!

Name	Link
🔨 Latest commit	60621c5
🔍 Latest deploy log	https://app.netlify.com/projects/redpanda-documentation/deploys/69e736917ef6710008f59828
😎 Deploy Preview	https://deploy-preview-166--redpanda-documentation.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 93 (🟢 up 15 from production) Accessibility: 96 (no change from production) Best Practices: 100 (no change from production) SEO: 83 (🔴 down 9 from production) PWA: - View the detailed breakdown and full score reports

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[micheleRP]

PR Review

Files reviewed: 0 .adoc files, 10 infrastructure/code files. Docs-team-standards review doesn't apply directly, but there are real technical issues in the edge functions plus embedded customer-facing markdown worth surfacing. URLs verified against the deploy preview.

Critical issues

1. netlify/edge-functions/api-catalog.ts — 4 of 5 service-meta URLs return 404 on the deploy preview

Verified against https://deploy-preview-166--redpanda-documentation.netlify.app:

service-meta URL	Status
/current/api/admin-api/ (Admin API)	404
/redpanda-cloud/api/cloud-controlplane-api/ (Cloud Control Plane)	404
/redpanda-cloud/api/cloud-dataplane-api/ (Cloud Data Plane)	404
/current/develop/schema-registry/ (Schema Registry)	404
/current/develop/http-proxy/ (HTTP Proxy)	200

Agents following these catalog entries will hit dead links. RFC 9727 makes service-meta optional — either fix each path or drop the service-meta entries that don't resolve. The service-doc paths under /api/doc/* all work, so those are fine.

2. netlify/edge-functions/feedback-api.ts — onRequestOptions is a Cloudflare Pages convention, not Netlify Edge Functions

Netlify Edge Functions don't recognize a separate onRequestOptions export — only the default export runs. CORS preflight (OPTIONS) requests will hit the default handler's method check and return 405, breaking any cross-origin browser caller. Fix by handling request.method === 'OPTIONS' inside the default handler (return 204 + CORS headers) and removing the onRequestOptions export.

Suggestions

3. PR description wording — "Fixed URLSearchParams bug in feedback API" is misleading; feedback-api.ts is a new file in this PR, not a fix. Consider "Add feedback API edge function" for release-notes accuracy.

4. local-antora-playbook.yml drift — the full AI-bot allowlist and Content-Signal are now mirrored into the local playbook, but local builds don't serve robots.txt meaningfully. Dropping it keeps the local playbook minimal and avoids config drift from production.

5. Content-Signal policy — ai-train=yes, search=yes, ai-input=yes is a policy decision with business implications. Flagging in case this hasn't already been signed off with marketing/DevRel — not blocking, just worth a paper trail.

6. netlify/edge-functions/feedback-api.ts error handling — error.message assumes error is an Error instance. Low risk in practice, but error instanceof Error ? error.message : String(error) is safer.

7. netlify/edge-functions/mcp-server-card.ts repository field — points to redpanda-data/docs-site, but the doc-tools-assistant MCP server may actually live in docs-extensions-and-macros. Worth double-checking so agents following the card land in the right repo.

Impact on other files

• llms.txt — referenced by the Link header, skill files, and embedded agent skills. Confirm it's still generated and current.
• nav.adoc / whats-new — no user-visible pages added; no updates required.
• home/modules/ROOT/attachments/api-feedback-registration.html — has <meta name="robots" content="noindex"> (good). Antora publishes it at /home/_attachments/api-feedback-registration.html, matching the edge function's POST target. Consistent.

What works well

• /.well-known/api-catalog, /.well-known/mcp/server-card.json, /.well-known/agent-skills/index.json, /mcp, and /search?q= all verified working on the deploy preview.
• Agent skills index returns real SHA256 digests, not the 0000… placeholder — fallback path isn't being hit.
• RFC references are correct (RFC 8288, RFC 9727, Content-Signal draft).
• Explicit edge-function path for /api/feedback prevents collision with the /api/* proxy.
• Preview playbook ends in its original robots: disallow state after the revert — previews won't leak the AI-bot allowlist.
• The HTML form registration uses hidden + noindex correctly.

[micheleRP]

Follow-up on critical #1 — verified the same paths against production (docs.redpanda.com), not just the deploy preview. All four also 404 in prod, so it's not a preview-build artifact.

However, the working paths do exist under a bare /api/ prefix. Suggested corrections for netlify/edge-functions/api-catalog.ts:

Current (404)	Working
/current/api/admin-api/	/api/admin-api/
/redpanda-cloud/api/cloud-controlplane-api/	/api/cloud-controlplane-api/
/redpanda-cloud/api/cloud-dataplane-api/	/api/cloud-dataplane-api/
/current/develop/schema-registry/	/api/schema-registry/
/current/develop/http-proxy/ (currently works)	/api/http-proxy/ (also works, and matches the others)

Using /api/* consistently across all five would also align the service-meta and service-doc URLs to the same path space.

[micheleRP]

lgtm!