feat: add docs robots.txt generation and doctor checks

[Kinfe123]

Summary by cubic

Adds docs robots generate to create an agent-friendly robots.txt and extends the doctor with local/hosted robots checks to prevent blocking agent routes and common AI crawlers. Improves docs and exports robots utilities so teams can configure, generate, and validate policies easily.

• New Features

  • CLI: docs robots generate writes or appends robots.txt (supports --append, --force, --path, --check, --config); defaults to the framework’s public dir.
  • Doctor: new robots checks for local files and hosted /robots.txt; flags blocked/missing routes and AI crawler policies; max score increases to 145 with robots and hosted-robots checks.
  • Config: robots in docs.config.ts (baseUrl, path, ai policy, extra rules). Robots helpers exported from @farming-labs/docs (render, analyze, upsert).

• Migration

  • Optionally add robots to docs.config.ts, then run pnpm exec docs robots generate. Use --append to keep existing content or --force to replace. Ensure /robots.txt is deployed and allows docs, markdown, llms.txt, sitemap, skill, MCP, and well-known agent routes.

^{Written for commit 75c5e54. Summary will update on new commits.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs-website	Ready	Preview, Comment	May 9, 2026 10:42pm

[greptile-apps]

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

[pkg-pr-new]

Open in StackBlitz

@farming-labs/astro

pnpm add https://pkg.pr.new/farming-labs/docs/@farming-labs/astro@173

@farming-labs/astro-theme

pnpm add https://pkg.pr.new/farming-labs/docs/@farming-labs/astro-theme@173

@farming-labs/docs

pnpm add https://pkg.pr.new/farming-labs/docs/@farming-labs/docs@173

@farming-labs/theme

pnpm add https://pkg.pr.new/farming-labs/docs/@farming-labs/theme@173

@farming-labs/next

pnpm add https://pkg.pr.new/farming-labs/docs/@farming-labs/next@173

@farming-labs/nuxt

pnpm add https://pkg.pr.new/farming-labs/docs/@farming-labs/nuxt@173

@farming-labs/nuxt-theme

pnpm add https://pkg.pr.new/farming-labs/docs/@farming-labs/nuxt-theme@173

@farming-labs/svelte

pnpm add https://pkg.pr.new/farming-labs/docs/@farming-labs/svelte@173

@farming-labs/svelte-theme

pnpm add https://pkg.pr.new/farming-labs/docs/@farming-labs/svelte-theme@173

@farming-labs/tanstack-start

pnpm add https://pkg.pr.new/farming-labs/docs/@farming-labs/tanstack-start@173

commit: 75c5e54

[cubic-dev-ai]

1 issue found across 14 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="website/app/docs/cli/page.mdx">

<violation number="1" location="website/app/docs/cli/page.mdx:454">
P3: The Doctor examples are internally inconsistent: the updated text sample uses a 105-point agent score, but the JSON sample still documents `maxScore: 100`.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

P3: The Doctor examples are internally inconsistent: the updated text sample uses a 105-point agent score, but the JSON sample still documents maxScore: 100.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At website/app/docs/cli/page.mdx, line 454:

<comment>The Doctor examples are internally inconsistent: the updated text sample uses a 105-point agent score, but the JSON sample still documents `maxScore: 100`.</comment>

<file context>
@@ -412,14 +451,16 @@ Expected output:
 @farming-labs/docs doctor — agent
 
-Score: 87/100 (Agent-ready)
+Score: 92/105 (Agent-ready)
 Framework: nextjs • Entry: docs • Content: app/docs
 Explicit agent-friendly pages: 10/41 pages (24%)
</file context>