From 95faf8feeb6a405904b8b19830aa06522597f684 Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Wed, 22 Oct 2025 20:13:03 +0000 Subject: [PATCH 01/12] Update llms.txt documentation with analytics and technical details Co-Authored-By: Devin Logan --- fern/products/docs/pages/seo/llms-txt.mdx | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index 0d3444645..63019f254 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -19,11 +19,23 @@ See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt) ### llms-full.txt - **Comprehensive**: Includes the full content of your documentation. -- **API-Ready**: Automatically incorporates your full API reference and SDK snippets (if generated by Fern). +- **API-Ready**: Automatically incorporates your full API reference with OpenAPI specifications and SDK code examples (if generated by Fern). - **Token efficient**: Removes unnecessary formatting to be as token-efficient as possible. See an example: [docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt) +## Analytics and Monitoring + +Fern tracks comprehensive analytics for llms.txt usage to help you understand how AI tools are consuming your documentation. The system monitors performance metrics including load times, processing times, and content length. User agent detection automatically identifies bot traffic versus browser requests, and detailed logging captures request headers for deeper insights. + +These analytics are available in the Fern dashboard, where you can view dedicated tables for llms.txt and markdown file access, along with channel attribution and device type information. This visibility helps you understand LLM traffic patterns and optimize your documentation for AI consumption. + +## Technical Details + +The llms.txt generation system handles complex schema types including self-referencing schemas, preventing infinite recursion during content generation. For large documentation sites, the system uses optimized timeouts to ensure reliable content delivery without timing out. + +For REST API endpoints, the markdown generation includes OpenAPI specifications in YAML format, making it easy for LLMs to understand your API structure. The same markdown generation logic is used for both serving llms.txt files and indexing content for search, ensuring consistency across your documentation platform. + ## View in Action Check out the llms.txt files for this site: @@ -36,4 +48,4 @@ Check out the llms.txt files for this site: alt="Example of using llms.txt" className="rounded-lg border border-gray-200 dark:border-gray-700" /> - \ No newline at end of file + From 58fca4866775df8e7a21c29aade084b8411e8c64 Mon Sep 17 00:00:00 2001 From: Devin Logan Date: Wed, 22 Oct 2025 16:49:22 -0400 Subject: [PATCH 02/12] add info on llms.txt improvements --- fern/products/docs/pages/seo/llms-txt.mdx | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index 63019f254..c17bce7be 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -19,24 +19,21 @@ See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt) ### llms-full.txt - **Comprehensive**: Includes the full content of your documentation. -- **API-Ready**: Automatically incorporates your full API reference with OpenAPI specifications and SDK code examples (if generated by Fern). -- **Token efficient**: Removes unnecessary formatting to be as token-efficient as possible. +- **API-ready**: Automatically incorporates your complete API reference with resolved OpenAPI specifications and SDK code examples for all enabled languages. Each API endpoint page includes its full OpenAPI spec. +- **Token efficient**: When Fern detects an LLM bot, it automatically serves markdown content from the llms.txt file, reducing token consumption by 90%+. See an example: [docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt) -## Analytics and Monitoring +## Analytics and monitoring -Fern tracks comprehensive analytics for llms.txt usage to help you understand how AI tools are consuming your documentation. The system monitors performance metrics including load times, processing times, and content length. User agent detection automatically identifies bot traffic versus browser requests, and detailed logging captures request headers for deeper insights. +Fern tracks comprehensive analytics for llms.txt usage in the Fern Dashboard, including: +- LLM traffic by provider (Claude, ChatGPT, Cursor, etc.) +- Traffic breakdown for .md and llms.txt files by agents and humans -These analytics are available in the Fern dashboard, where you can view dedicated tables for llms.txt and markdown file access, along with channel attribution and device type information. This visibility helps you understand LLM traffic patterns and optimize your documentation for AI consumption. +This visibility helps you understand LLM traffic patterns and optimize your documentation for AI consumption. -## Technical Details -The llms.txt generation system handles complex schema types including self-referencing schemas, preventing infinite recursion during content generation. For large documentation sites, the system uses optimized timeouts to ensure reliable content delivery without timing out. - -For REST API endpoints, the markdown generation includes OpenAPI specifications in YAML format, making it easy for LLMs to understand your API structure. The same markdown generation logic is used for both serving llms.txt files and indexing content for search, ensuring consistency across your documentation platform. - -## View in Action +## View in action Check out the llms.txt files for this site: - `https://buildwithfern.com/learn/llms.txt` From 2fa561e396769ddb1fa076f6340e1615d3efec65 Mon Sep 17 00:00:00 2001 From: Devin Logan Date: Wed, 22 Oct 2025 16:50:16 -0400 Subject: [PATCH 03/12] vale fix --- fern/products/docs/pages/seo/llms-txt.mdx | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index c17bce7be..c77739c9c 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -19,7 +19,7 @@ See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt) ### llms-full.txt - **Comprehensive**: Includes the full content of your documentation. -- **API-ready**: Automatically incorporates your complete API reference with resolved OpenAPI specifications and SDK code examples for all enabled languages. Each API endpoint page includes its full OpenAPI spec. +- **API-ready**: Automatically incorporates your complete API Reference with resolved OpenAPI specifications and SDK code examples for all languages you've enabled. Each API endpoint page includes its full OpenAPI spec. - **Token efficient**: When Fern detects an LLM bot, it automatically serves markdown content from the llms.txt file, reducing token consumption by 90%+. See an example: [docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt) @@ -32,7 +32,6 @@ Fern tracks comprehensive analytics for llms.txt usage in the Fern Dashboard, in This visibility helps you understand LLM traffic patterns and optimize your documentation for AI consumption. - ## View in action Check out the llms.txt files for this site: From da4bb1b9c35781af56f7aa9a794a30c46664ebdf Mon Sep 17 00:00:00 2001 From: Devin Logan Date: Wed, 22 Oct 2025 17:20:31 -0400 Subject: [PATCH 04/12] small fix --- fern/products/docs/pages/seo/llms-txt.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index c77739c9c..c75eea42b 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -20,7 +20,7 @@ See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt) ### llms-full.txt - **Comprehensive**: Includes the full content of your documentation. - **API-ready**: Automatically incorporates your complete API Reference with resolved OpenAPI specifications and SDK code examples for all languages you've enabled. Each API endpoint page includes its full OpenAPI spec. -- **Token efficient**: When Fern detects an LLM bot, it automatically serves markdown content from the llms.txt file, reducing token consumption by 90%+. +- **Token efficient**: When Fern detects an LLM bot, it automatically serves Markdown content from the llms.txt file instead of HTML, reducing token consumption by 90%+. See an example: [docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt) From b083acb0ee2b49594f7e662a84d8c29a77ab6378 Mon Sep 17 00:00:00 2001 From: Devin Logan Date: Wed, 22 Oct 2025 19:03:23 -0400 Subject: [PATCH 05/12] reorganize page for clarity --- .vale/styles/FernStyles/Acronyms.yml | 1 + fern/products/docs/pages/seo/llms-txt.mdx | 44 ++++++++++------------- 2 files changed, 19 insertions(+), 26 deletions(-) diff --git a/.vale/styles/FernStyles/Acronyms.yml b/.vale/styles/FernStyles/Acronyms.yml index 9491d7e68..1b8106618 100644 --- a/.vale/styles/FernStyles/Acronyms.yml +++ b/.vale/styles/FernStyles/Acronyms.yml @@ -68,3 +68,4 @@ exceptions: - MDX # Fern-specific - SEO # Fern-specific - AWS # Fern-specific + - LLM # Fern-specific diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index c75eea42b..233912463 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -1,40 +1,23 @@ --- title: llms.txt -subtitle: Enable tools like Cursor, GitHub Copilot, ChatGPT, and Claude to quickly understand your documentation. -description: Learn how your documentation becomes accessible to AI tools using the llms.txt standard. Make your docs AI-friendly. -layout: guide +description: Enable tools like Cursor, GitHub Copilot, ChatGPT, and Claude to understand your documentation. --- -## What is llms.txt? -[LLMs.txt](https://llmstxt.org/) is an emerging standard so that websites can easily expose information to AI developer tools. -LLMs thrive on concise, structured information, and `llms.txt` help them locate and interpret key information in your developer documentation with ease. -No configuration required--your `llms.txt` files are automatically generated and maintained, just like a `sitemap.xml` or `robots.txt`. +[llms.txt](https://llmstxt.org/) is a standard for exposing website content to AI developer tools. Fern automatically generates and maintains `llms.txt` and `llms-full.txt` files for your documentation site. -### llms.txt -- **Small and fast**: Quick to load and easy to parse. -- **Summary-focused**: Each page distilled into a one-sentence description with its URL. -- **Structured for AI**: Helps tools understand the overall structure of your documentation. +## `llms.txt` -See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt) - -### llms-full.txt -- **Comprehensive**: Includes the full content of your documentation. -- **API-ready**: Automatically incorporates your complete API Reference with resolved OpenAPI specifications and SDK code examples for all languages you've enabled. Each API endpoint page includes its full OpenAPI spec. -- **Token efficient**: When Fern detects an LLM bot, it automatically serves Markdown content from the llms.txt file instead of HTML, reducing token consumption by 90%+. - -See an example: [docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt) +Contains a lightweight summary of your documentation site with each page distilled into a one-sentence description and URL. The format is structured to help AI tools parse and understand your documentation's hierarchy and organization. -## Analytics and monitoring +See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt) -Fern tracks comprehensive analytics for llms.txt usage in the Fern Dashboard, including: -- LLM traffic by provider (Claude, ChatGPT, Cursor, etc.) -- Traffic breakdown for .md and llms.txt files by agents and humans +## `llms-full.txt` -This visibility helps you understand LLM traffic patterns and optimize your documentation for AI consumption. +Contains complete documentation content including the full text of all pages. For API documentation, this includes your complete API Reference with resolved OpenAPI specifications and SDK code examples for enabled languages. Each API endpoint page includes its full OpenAPI spec. -## View in action +When Fern detects an LLM bot accessing your documentation, it automatically serves Markdown instead of HTML, reducing token consumption by 90%+. -Check out the llms.txt files for this site: +View the llms.txt files for this site: - `https://buildwithfern.com/learn/llms.txt` - `https://buildwithfern.com/learn/llms-full.txt` @@ -45,3 +28,12 @@ Check out the llms.txt files for this site: className="rounded-lg border border-gray-200 dark:border-gray-700" /> + +## Analytics and monitoring + +The [Fern Dashboard](https://dashboard.buildwithfern.com/) provides comprehensive analytics for llms.txt usage including: +- Traffic by LLM provider (Claude, ChatGPT, Cursor, etc.) +- Page-level breakdowns of bot vs. human visitors for `.md` and `llms.txt` files + +This visibility helps you understand LLM traffic patterns and optimize your documentation for AI consumption. + From 8a37fcd114d6388250e252ad89513cb860417620 Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Thu, 23 Oct 2025 00:06:48 +0000 Subject: [PATCH 06/12] Clarify llms.txt availability and make example links clickable Co-Authored-By: Devin Logan --- fern/products/docs/pages/seo/llms-txt.mdx | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index 233912463..d50fb5630 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -9,17 +9,25 @@ description: Enable tools like Cursor, GitHub Copilot, ChatGPT, and Claude to un Contains a lightweight summary of your documentation site with each page distilled into a one-sentence description and URL. The format is structured to help AI tools parse and understand your documentation's hierarchy and organization. +`llms.txt` is available at any level of your documentation hierarchy (e.g., `/llms.txt`, `/docs/llms.txt`, `/v1/api-reference/llms.txt`). + See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt) ## `llms-full.txt` Contains complete documentation content including the full text of all pages. For API documentation, this includes your complete API Reference with resolved OpenAPI specifications and SDK code examples for enabled languages. Each API endpoint page includes its full OpenAPI spec. +`llms-full.txt` is available at the root of your documentation site and summarizes all of your documentation. + When Fern detects an LLM bot accessing your documentation, it automatically serves Markdown instead of HTML, reducing token consumption by 90%+. -View the llms.txt files for this site: -- `https://buildwithfern.com/learn/llms.txt` -- `https://buildwithfern.com/learn/llms-full.txt` +See an example: [docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt) + +## View in Action + +Check out the llms.txt files for this site: +- [https://buildwithfern.com/learn/llms.txt](https://buildwithfern.com/learn/llms.txt) +- [https://buildwithfern.com/learn/llms-full.txt](https://buildwithfern.com/learn/llms-full.txt) Date: Wed, 22 Oct 2025 20:40:31 -0400 Subject: [PATCH 07/12] remove technical details and other extra info --- fern/products/docs/pages/seo/llms-txt.mdx | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-) diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index d50fb5630..b8523a4c5 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -17,15 +17,12 @@ See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt) Contains complete documentation content including the full text of all pages. For API documentation, this includes your complete API Reference with resolved OpenAPI specifications and SDK code examples for enabled languages. Each API endpoint page includes its full OpenAPI spec. -`llms-full.txt` is available at the root of your documentation site and summarizes all of your documentation. +`llms-full.txt` is available only at the root of your documentation site. When Fern detects an LLM bot accessing your documentation, it automatically serves Markdown instead of HTML, reducing token consumption by 90%+. -See an example: [docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt) +View the `llms.txt` files for this site: -## View in Action - -Check out the llms.txt files for this site: - [https://buildwithfern.com/learn/llms.txt](https://buildwithfern.com/learn/llms.txt) - [https://buildwithfern.com/learn/llms-full.txt](https://buildwithfern.com/learn/llms-full.txt) @@ -42,13 +39,5 @@ Check out the llms.txt files for this site: The [Fern Dashboard](https://dashboard.buildwithfern.com/) provides comprehensive analytics for llms.txt usage including: - Traffic by LLM provider (Claude, ChatGPT, Cursor, etc.) - Page-level breakdowns of bot vs. human visitors for `.md` and `llms.txt` files -- Performance metrics including load times, processing times, and content length -- User agent detection to identify bot traffic versus browser requests This visibility helps you understand LLM traffic patterns and optimize your documentation for AI consumption. - -## Technical Details - -The llms.txt generation system handles complex schema types including self-referencing schemas, preventing infinite recursion during content generation. For large documentation sites, the system uses optimized timeouts to ensure reliable content delivery without timing out. - -For REST API endpoints, the markdown generation includes OpenAPI specifications in YAML format, making it easy for LLMs to understand your API structure. The same markdown generation logic is used for both serving llms.txt files and indexing content for search, ensuring consistency across your documentation platform. From e9ff9369b70a66487510781716f5c85c7c112c02 Mon Sep 17 00:00:00 2001 From: Devin Logan Date: Wed, 22 Oct 2025 20:43:04 -0400 Subject: [PATCH 08/12] nit --- fern/products/docs/pages/seo/llms-txt.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index b8523a4c5..2c2ee8970 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -21,6 +21,8 @@ Contains complete documentation content including the full text of all pages. Fo When Fern detects an LLM bot accessing your documentation, it automatically serves Markdown instead of HTML, reducing token consumption by 90%+. +### Examples + View the `llms.txt` files for this site: - [https://buildwithfern.com/learn/llms.txt](https://buildwithfern.com/learn/llms.txt) @@ -36,7 +38,7 @@ View the `llms.txt` files for this site: ## Analytics and monitoring -The [Fern Dashboard](https://dashboard.buildwithfern.com/) provides comprehensive analytics for llms.txt usage including: +The [Fern Dashboard](https://dashboard.buildwithfern.com/) provides comprehensive analytics for `llms.txt` usage including: - Traffic by LLM provider (Claude, ChatGPT, Cursor, etc.) - Page-level breakdowns of bot vs. human visitors for `.md` and `llms.txt` files From 28047eb6acc1e2f5d9d3ff02b29ccf2b1d686e73 Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Thu, 23 Oct 2025 13:58:25 +0000 Subject: [PATCH 09/12] Address PR feedback: move markdown serving info to llms.txt section and move image up Co-Authored-By: Devin Logan --- fern/products/docs/pages/seo/llms-txt.mdx | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index 2c2ee8970..3c59c8579 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -11,15 +11,25 @@ Contains a lightweight summary of your documentation site with each page distill `llms.txt` is available at any level of your documentation hierarchy (e.g., `/llms.txt`, `/docs/llms.txt`, `/v1/api-reference/llms.txt`). +When Fern detects an LLM bot accessing your documentation, it automatically serves Markdown instead of HTML, reducing token consumption by 90%+. + See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt) + + Example of using llms.txt + + ## `llms-full.txt` Contains complete documentation content including the full text of all pages. For API documentation, this includes your complete API Reference with resolved OpenAPI specifications and SDK code examples for enabled languages. Each API endpoint page includes its full OpenAPI spec. `llms-full.txt` is available only at the root of your documentation site. -When Fern detects an LLM bot accessing your documentation, it automatically serves Markdown instead of HTML, reducing token consumption by 90%+. +See an example: [docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt) ### Examples @@ -28,14 +38,6 @@ View the `llms.txt` files for this site: - [https://buildwithfern.com/learn/llms.txt](https://buildwithfern.com/learn/llms.txt) - [https://buildwithfern.com/learn/llms-full.txt](https://buildwithfern.com/learn/llms-full.txt) - - Example of using llms.txt - - ## Analytics and monitoring The [Fern Dashboard](https://dashboard.buildwithfern.com/) provides comprehensive analytics for `llms.txt` usage including: From 62b717c38052264648defa4e1a323dedea7d478c Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Thu, 23 Oct 2025 15:14:08 +0000 Subject: [PATCH 10/12] Update llms-full.txt example to use elevenlabs.io instead of cohere Co-Authored-By: Devin Logan --- fern/products/docs/pages/seo/llms-txt.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index 3c59c8579..2f3f063e2 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -29,7 +29,7 @@ Contains complete documentation content including the full text of all pages. Fo `llms-full.txt` is available only at the root of your documentation site. -See an example: [docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt) +See an example: [elevenlabs.io/docs/llms-full.txt](https://elevenlabs.io/docs/llms-full.txt) ### Examples From e753c458b6209e70efee6d1cc2b6daa7a18a5480 Mon Sep 17 00:00:00 2001 From: Devin Logan Date: Thu, 23 Oct 2025 11:42:08 -0400 Subject: [PATCH 11/12] add better examples, update title, small restructures --- fern/products/docs/docs.yml | 1 + fern/products/docs/pages/seo/llms-txt.mdx | 35 +++++++++-------------- 2 files changed, 15 insertions(+), 21 deletions(-) diff --git a/fern/products/docs/docs.yml b/fern/products/docs/docs.yml index 300965eaa..9c7edf7da 100644 --- a/fern/products/docs/docs.yml +++ b/fern/products/docs/docs.yml @@ -194,6 +194,7 @@ navigation: path: ./pages/seo/redirects.mdx - page: llms.txt path: ./pages/seo/llms-txt.mdx + slug: llms-txt - section: Authentication collapsed: true contents: diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index 2f3f063e2..d5a7991f8 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -1,19 +1,9 @@ --- -title: llms.txt +title: llms.txt and llms-full.txt description: Enable tools like Cursor, GitHub Copilot, ChatGPT, and Claude to understand your documentation. --- -[llms.txt](https://llmstxt.org/) is a standard for exposing website content to AI developer tools. Fern automatically generates and maintains `llms.txt` and `llms-full.txt` files for your documentation site. - -## `llms.txt` - -Contains a lightweight summary of your documentation site with each page distilled into a one-sentence description and URL. The format is structured to help AI tools parse and understand your documentation's hierarchy and organization. - -`llms.txt` is available at any level of your documentation hierarchy (e.g., `/llms.txt`, `/docs/llms.txt`, `/v1/api-reference/llms.txt`). - -When Fern detects an LLM bot accessing your documentation, it automatically serves Markdown instead of HTML, reducing token consumption by 90%+. - -See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt) +[llms.txt](https://llmstxt.org/) is a standard for exposing website content to AI developer tools. Fern automatically generates and maintains `llms.txt` and `llms-full.txt` files for your documentation site. When Fern detects an LLM bot accessing your documentation, it automatically serves Markdown instead of HTML, reducing token consumption by 90%+. -## `llms-full.txt` +## `llms.txt` -Contains complete documentation content including the full text of all pages. For API documentation, this includes your complete API Reference with resolved OpenAPI specifications and SDK code examples for enabled languages. Each API endpoint page includes its full OpenAPI spec. +Contains a lightweight summary of your documentation site with each page distilled into a one-sentence description and URL. The format is structured to help AI tools parse and understand your documentation's hierarchy and organization. -`llms-full.txt` is available only at the root of your documentation site. +`llms.txt` is available at any level of your documentation hierarchy (e.g., `/llms.txt`, `/docs/llms.txt`, `/v1/api-reference/llms.txt`). -See an example: [elevenlabs.io/docs/llms-full.txt](https://elevenlabs.io/docs/llms-full.txt) +Examples from Eleven Labs: +- Developer documentation: [elevenlabs.io/docs/llms.txt](https://elevenlabs.io/docs/llms.txt) +- API Reference (Create speech endpoint): [elevenlabs.io/docs/api-reference/text-to-speech/convert/llms.txt](https://elevenlabs.io/docs/api-reference/text-to-speech/convert/llms.txt) -### Examples +## `llms-full.txt` + +Contains complete documentation content including the full text of all pages. For API documentation, this includes your complete API Reference with resolved OpenAPI specifications and SDK code examples for enabled languages. Each API endpoint page includes its full OpenAPI spec. -View the `llms.txt` files for this site: +`llms-full.txt` is available only at the root of your documentation site. -- [https://buildwithfern.com/learn/llms.txt](https://buildwithfern.com/learn/llms.txt) -- [https://buildwithfern.com/learn/llms-full.txt](https://buildwithfern.com/learn/llms-full.txt) +Example from Cash App: [developers.cash.app/llms-full.txt](https://developers.cash.app/llms-full.txt) ## Analytics and monitoring The [Fern Dashboard](https://dashboard.buildwithfern.com/) provides comprehensive analytics for `llms.txt` usage including: - Traffic by LLM provider (Claude, ChatGPT, Cursor, etc.) -- Page-level breakdowns of bot vs. human visitors for `.md` and `llms.txt` files +- Page-level breakdowns of bot vs. human visitors for Markdown and `llms.txt` files This visibility helps you understand LLM traffic patterns and optimize your documentation for AI consumption. From 2a7a1e38d0d7cd99c98ba61a394161c125ba1222 Mon Sep 17 00:00:00 2001 From: Devin Logan Date: Thu, 23 Oct 2025 12:00:50 -0400 Subject: [PATCH 12/12] fix example --- fern/products/docs/pages/seo/llms-txt.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fern/products/docs/pages/seo/llms-txt.mdx b/fern/products/docs/pages/seo/llms-txt.mdx index d5a7991f8..002c9a8f8 100644 --- a/fern/products/docs/pages/seo/llms-txt.mdx +++ b/fern/products/docs/pages/seo/llms-txt.mdx @@ -17,10 +17,10 @@ description: Enable tools like Cursor, GitHub Copilot, ChatGPT, and Claude to un Contains a lightweight summary of your documentation site with each page distilled into a one-sentence description and URL. The format is structured to help AI tools parse and understand your documentation's hierarchy and organization. -`llms.txt` is available at any level of your documentation hierarchy (e.g., `/llms.txt`, `/docs/llms.txt`, `/v1/api-reference/llms.txt`). +`llms.txt` is available at any level of your documentation hierarchy (`/llms.txt`, `/docs/llms.txt`, `/v1/api-reference/llms.txt`, etc.). Examples from Eleven Labs: -- Developer documentation: [elevenlabs.io/docs/llms.txt](https://elevenlabs.io/docs/llms.txt) +- Developer documentation (Prompting directory): [elevenlabs.io/docs/best-practices/prompting/llms.txt](https://elevenlabs.io/docs/best-practices/prompting/llms.txt) - API Reference (Create speech endpoint): [elevenlabs.io/docs/api-reference/text-to-speech/convert/llms.txt](https://elevenlabs.io/docs/api-reference/text-to-speech/convert/llms.txt) ## `llms-full.txt`