[AI Audit] AI Audit update docs

[Oxyjun]

Summary

1. Updating main AI Audit landing page to match docs restructure
2. Updating Get started to match the new design
3. Updating Analyze AI crawlers to match new design
4. Adding new chapter Manage AI crawlers to better explain the new AI Crawlers tab
5. Adding glossary
6. Using Tabs / TabItem components to delineate features that are only available on paid plans from those available on free plans
7. Removing Enforce robots.txt chapter, as this feature is no longer being accessed through AI Audit

Screenshots (optional)

Documentation checklist

• The documentation style guide has been adhered to.
• If a larger change - such as adding a new page- an issue has been opened in relation to any incorrect or out of date information that this PR fixes.
• Files which have changed name or location have been allocated redirects.

[github-actions]

This PR changes current filenames or deletes current files. Make sure you have redirects set up to cover the following paths:

• /ai-audit/features/detect-ai-crawlers/
• /ai-audit/features/enforce-robots-txt/

[hyperlint-ai]

Howdy and thanks for contributing to our repo. The Cloudflare team reviews new, external PRs within two (2) weeks. If it's been two weeks or longer without any movement, please tag the PR Assignees in a comment.

We review internal PRs within 1 week. If it's something urgent or has been sitting without a comment, start a thread in the Developer Docs space internally.

---

PR Change Summary

Updated AI Audit documentation to reflect new design and features, including a glossary for better understanding.

• Updated 'Get started' and 'Analyze AI crawlers' sections to match new design.
• Added a new chapter 'Manage AI crawlers' to explain the AI Crawlers tab.
• Introduced a glossary for key terms used in AI Audit documentation.
• Implemented Tabs/TabItem components to differentiate features available on paid plans.

Modified Files

• src/content/docs/ai-audit/get-started.mdx

Added Files

• src/content/docs/ai-audit/features/analyze-ai-crawlers.mdx
• src/content/docs/ai-audit/features/manage-ai-crawlers.mdx
• src/content/docs/ai-audit/reference/glossary.mdx
• src/content/docs/ai-audit/reference/index.mdx

---

How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

[github-actions]

This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:

Pattern	Owners
/public/__redirects	@GregBrimble, @KianNH, @pedrosousa, @WalshyDev, @cloudflare/pcx-technical-writing
/src/assets/images/	@cloudflare/pm-changelogs, @cloudflare/pcx-technical-writing
/src/content/changelog/	@cloudflare/pm-changelogs, @cloudflare/pcx-cloudflare-one
/src/content/docs/ai-audit/	@oxyjun, @kodster28, @cloudflare/pcx-technical-writing
*	@cloudflare/pcx-technical-writing

[github-actions]

Preview URL: https://654d6e9d.preview.developers.cloudflare.com
Preview Branch URL: https://jun-ai-audit-update.preview.developers.cloudflare.com

Files with changes (up to 15)

Original Link	Updated Link
https://developers.cloudflare.com/ai-audit/get-started/	https://jun-ai-audit-update.preview.developers.cloudflare.com/ai-audit/get-started/
https://developers.cloudflare.com/ai-audit/features/manage-ai-crawlers/	https://jun-ai-audit-update.preview.developers.cloudflare.com/ai-audit/features/manage-ai-crawlers/
https://developers.cloudflare.com/ai-audit/features/analyze-ai-crawlers/	https://jun-ai-audit-update.preview.developers.cloudflare.com/ai-audit/features/analyze-ai-crawlers/
https://developers.cloudflare.com/ai-audit/features/detect-ai-crawlers/	https://jun-ai-audit-update.preview.developers.cloudflare.com/ai-audit/features/detect-ai-crawlers/
https://developers.cloudflare.com/ai-audit/features/enforce-robots-txt/	https://jun-ai-audit-update.preview.developers.cloudflare.com/ai-audit/features/enforce-robots-txt/
https://developers.cloudflare.com/changelog/2025-06-30-refresh/	https://jun-ai-audit-update.preview.developers.cloudflare.com/changelog/2025-06-30-refresh/
https://developers.cloudflare.com/ai-audit/	https://jun-ai-audit-update.preview.developers.cloudflare.com/ai-audit/
https://developers.cloudflare.com/ai-audit/reference/glossary/	https://jun-ai-audit-update.preview.developers.cloudflare.com/ai-audit/reference/glossary/
https://developers.cloudflare.com/ai-audit/reference/	https://jun-ai-audit-update.preview.developers.cloudflare.com/ai-audit/reference/
https://developers.cloudflare.com/changelog/2024-09-23-ai-audit-launch/	https://jun-ai-audit-update.preview.developers.cloudflare.com/changelog/2024-09-23-ai-audit-launch/

[CameronWhiteside]

✅ (I've only got comment permissions, looks super)

[kodster28]

I'd also make sure that @CameronWhiteside has signed off on the changelog content.