docs(#844): allow agent crawlers to read documentation paths

[fullsend-ai-coder]

Summary

Update robots.txt to allow training/agent crawlers (ClaudeBot, GPTBot, CCBot, Google-Extended, Bytespider, meta-externalagent, Applebot-Extended) to access documentation paths (/docs/ and /llms.txt) while keeping the rest of the site blocked. This enables agents to discover and read documentation for context during autonomous operations.

Related Issue

Closes #844

Changes

• Added Allow: /docs/ and Allow: /llms.txt directives before Disallow: / for each of the 7 blocked training crawlers
• Updated the section comment to reflect the documentation exception
• Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot) and default User-agent: * remain unchanged
• Applebot (non-Extended) remains fully allowed as before

Testing

• Verified the robots.txt follows the correct format: more specific Allow directives before the blanket Disallow: /
• Secret scan passed
• make lint passes (pre-commit could not run in sandbox due to network restrictions)

Checklist

• PR title follows Conventional Commits (correct type, ! for breaking changes)
• Commits are signed off (DCO) — human and human-directed agent sessions only
• I wrote this contribution myself and can explain all changes in it

---

Closes #844

Post-script verification

• Branch is not main/master (agent/844-robots-txt-allow-docs)
• Secret scan passed (gitleaks — c887fc47c59d8b87b5282ca44044d2933d30df41..HEAD)
• PR body secret scan passed (gitleaks — no-git)
• Pre-commit hooks passed (authoritative run on runner)
• Tests ran inside sandbox

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 9:42 AM UTC · Completed 9:53 AM UTC
Commit: 06f3bd8 · View workflow run →

[fullsend-ai-review]

Review

Findings

Low

• [scope-incomplete] cloudflare_site/public/robots.txt — Issue Update robots.txt to enable agent discovery of documentation and examples #844's acceptance criteria include "full reads of docs and sales/example materials," but this PR only allows /docs/ and /llms.txt. If sales or example content lives under a different path (e.g., /examples/), those paths are still blocked. Verify whether "sales/example materials" are located under /docs/ or require additional Allow rules before closing Update robots.txt to enable agent discovery of documentation and examples #844.

---

Labels: PR modifies website configuration (robots.txt) for documentation accessibility

[guyoron1]

/fs-fix

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 10:32 AM UTC · Completed 10:37 AM UTC
Commit: 06f3bd8 · View workflow run →

[fullsend-ai-coder]

🔧 Fix agent — iteration 1 (human-triggered)

Investigated the review finding about potentially missing Allow rules for 'sales/example materials'. After verifying the VitePress config (base: '/docs/'), Cloudflare wrangler.toml, and site directory structure, confirmed that ALL documentation and example content is served under /docs/. No additional paths exist on the deployed site for sales or example materials. The existing Allow: /docs/ directive fully satisfies Issue #844's acceptance criteria. No code changes needed.

Disagreed (1):

1. scope-incomplete: PR only allows /docs/ and /llms.txt but Issue Update robots.txt to enable agent discovery of documentation and examples #844 mentions sales/example materials: Verified against the site configuration. The VitePress documentation site (website/.vitepress/config.ts) uses base: '/docs/', meaning ALL documentation content — including guides, agents, examples, experiments, normative specs, and any 'sales/example materials' — is served exclusively under the /docs/ path. There is no separate /examples/, /sales/, or other top-level content path on the deployed Cloudflare site. The only non-docs paths are /api/* (OAuth worker) and root (admin SPA). The llms.txt file's links all point to fullsend.sh/docs/... paths, confirming this. Therefore, Allow: /docs/ already covers all documentation and example content referenced in Issue Update robots.txt to enable agent discovery of documentation and examples #844's acceptance criteria. No additional Allow rules are needed.

Tests: passed

Next steps:

• /fs-review — request a re-review of the changes
• /fs-fix <your instruction> — run another fix pass with specific guidance
• Push commits directly — review re-runs automatically on push
  _{Updated by fullsend fix agent}