From f294adb5d45ed5d98ade2d25b10cdca9ffe68c55 Mon Sep 17 00:00:00 2001
From: asaf <asaf@pulumi.com>
Date: Mon, 24 Nov 2025 13:46:31 -0500
Subject: [PATCH 1/6] feat: Add llms.txt implementation for AI/LLM consumption

Implement llms.txt standard to improve Pulumi's visibility in AI-generated content and responses.

Changes:
- Add static llms.txt file with curated Pulumi documentation structure
- Configure Hugo to generate llms.txt dynamically via new output format
- Create Hugo template for future dynamic llms.txt generation
- Add AI bot tracking JavaScript to measure experiment effectiveness
- Update AGENTS.md with llms.txt guidance for AI assistants
- Track known AI bots: ClaudeBot, Amazonbot, ChatGPT-User, OAI-SearchBot, PerplexityBot

The blockquote summary uses Pulumi's Entry Segment Messaging emphasizing:
- Real programming languages vs YAML
- IDE support, type safety, automated testing
- Multi-cloud support (AWS, Azure, GCP, Kubernetes, 150+ providers)
- 3x faster deployment with 90% fewer configuration errors

Bot detection based on actual traffic data from pulumi.com analytics.
---
 AGENTS.md                    |  20 ++++++
 config/_default/config.yml   |   6 ++
 layouts/index.llmstxt.txt    |  91 +++++++++++++++++++++++++++
 layouts/partials/assets.html |   3 +
 static/js/ai-bot-tracking.js | 117 +++++++++++++++++++++++++++++++++++
 static/llms.txt              |  69 +++++++++++++++++++++
 6 files changed, 306 insertions(+)
 create mode 100644 layouts/index.llmstxt.txt
 create mode 100644 static/js/ai-bot-tracking.js
 create mode 100644 static/llms.txt

diff --git a/AGENTS.md b/AGENTS.md
index 7b052a4d890d..6f2fa3b74d64 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -10,6 +10,26 @@ This repository contains the source for the Pulumi website, built with [Hugo](ht
 
 ---
 
+## AI Assistant Information (llms.txt)
+
+This repository implements the llms.txt standard for improved AI/LLM consumption:
+
+- **Primary content index**: https://www.pulumi.com/llms.txt
+- **Standard**: Following [llms.txt specification](https://llmstxt.org/)
+- **Content structure**: Hierarchical documentation index optimized for LLM context windows
+- **Focus areas for AI consumption**:
+  - `/docs/iac/` - Infrastructure as Code documentation
+  - `/docs/esc/` - Secrets and configuration management
+  - `/docs/pulumi-cloud/` - Platform capabilities
+  - `/registry/` - Provider and package documentation
+  - `/docs/reference/` - API references for all languages
+- **Markdown versions**: When available, append `.md` to URLs for cleaner content
+- **Excluding content from llms.txt**: Add `llms_exclude: true` to front matter
+
+When working with Pulumi documentation, consult the llms.txt file first for structured navigation and understanding of the documentation hierarchy.
+
+---
+
 ## Build / Test / Lint Workflow
 
 Agents must use these exact commands:
diff --git a/config/_default/config.yml b/config/_default/config.yml
index 0aba5d88e7c0..96b11dc0d01f 100644
--- a/config/_default/config.yml
+++ b/config/_default/config.yml
@@ -33,6 +33,11 @@ timeout: 300000
 outputFormats:
   rss:
     baseName: rss
+  LLMSTXT:
+    mediaType: "text/plain"
+    baseName: "llms"
+    isPlainText: true
+    permalinkable: false
 
 # We set the outputs explicitly to just "HTML" as otherwise they default to both
 # "HTML" and "RSS", and we only want to generate RSS for the blog section -- not
@@ -43,6 +48,7 @@ outputs:
   home:
     - HTML
     - JSON
+    - LLMSTXT
   section:
     - HTML
 
diff --git a/layouts/index.llmstxt.txt b/layouts/index.llmstxt.txt
new file mode 100644
index 000000000000..1c3089878f65
--- /dev/null
+++ b/layouts/index.llmstxt.txt
@@ -0,0 +1,91 @@
+{{- /* Generate llms.txt file for AI/LLM consumption */ -}}
+{{- /* Static version for now - dynamic generation can be enabled later */ -}}
+# Pulumi
+
+> Infrastructure as code using real programming languages. Write, test, and deploy cloud infrastructure with TypeScript, Python, Go, C#, or Java instead of YAML. Get IDE support, type safety, and automated testing for AWS, Azure, GCP, Kubernetes, and 150+ providers. Deploy 3x faster with 90% fewer configuration errors.
+
+## Getting Started
+
+{{- range where .Site.RegularPages "Section" "docs" }}
+  {{- if and (in .RelPermalink "/get-started/") (not (isset .Params "llms_exclude")) }}
+- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc }}{{ end }}
+  {{- end }}
+{{- end }}
+- [Core Concepts](https://www.pulumi.com/docs/iac/concepts/) - Projects, stacks, resources, and state
+
+## Infrastructure as Code (IaC)
+
+- [Why Pulumi?](https://www.pulumi.com/docs/iac/) - Real code vs YAML configuration
+{{- range where .Site.RegularPages "Section" "docs" }}
+  {{- if and (in .RelPermalink "/iac/") (not (in .RelPermalink "/get-started/")) (not (isset .Params "llms_exclude")) }}
+    {{- if or (in .RelPermalink "/languages-sdks/") (in .RelPermalink "/clouds/") (in .RelPermalink "/cli/") (in .RelPermalink "/automation-api/") (in .RelPermalink "/crossguard/") (in .RelPermalink "/adopting-pulumi/") }}
+- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
+    {{- end }}
+  {{- end }}
+{{- end }}
+
+## Pulumi ESC (Environments, Secrets, Configuration)
+
+{{- range where .Site.RegularPages "Section" "docs" }}
+  {{- if and (in .RelPermalink "/esc/") (not (isset .Params "llms_exclude")) }}
+    {{- if or (eq .RelPermalink "/docs/esc/") (in .RelPermalink "/get-started/") (in .RelPermalink "/environments/") (in .RelPermalink "/integrations/") (in .RelPermalink "/cli/") }}
+- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
+    {{- end }}
+  {{- end }}
+{{- end }}
+
+## Pulumi Cloud Platform
+
+{{- range where .Site.RegularPages "Section" "docs" }}
+  {{- if and (in .RelPermalink "/pulumi-cloud/") (not (isset .Params "llms_exclude")) }}
+    {{- if or (eq .RelPermalink "/docs/pulumi-cloud/") (in .RelPermalink "/deployments/") (in .RelPermalink "/copilot/") (in .RelPermalink "/access-management/") (in .RelPermalink "/admin/") }}
+- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
+    {{- end }}
+  {{- end }}
+{{- end }}
+- [Pulumi Insights](https://www.pulumi.com/docs/insights/) - Infrastructure intelligence and search
+
+## Registry (Providers & Packages)
+
+- [AWS Provider](https://www.pulumi.com/registry/packages/aws/) - Complete AWS resource coverage
+- [Azure Native Provider](https://www.pulumi.com/registry/packages/azure-native/) - Full Azure ARM API
+- [Google Cloud Provider](https://www.pulumi.com/registry/packages/gcp/) - Complete GCP coverage
+- [Kubernetes Provider](https://www.pulumi.com/registry/packages/kubernetes/) - Native Kubernetes support
+- [All Providers](https://www.pulumi.com/registry/) - 150+ cloud and SaaS providers
+
+## API Reference
+
+- [Node.js/TypeScript](https://www.pulumi.com/docs/reference/pkg/nodejs/) - Complete TypeScript/JavaScript API
+- [Python](https://www.pulumi.com/docs/reference/pkg/python/) - Complete Python API
+- [Go](https://www.pulumi.com/docs/reference/pkg/go/) - Complete Go API
+- [.NET/C#](https://www.pulumi.com/docs/reference/pkg/dotnet/) - Complete .NET API
+- [Java](https://www.pulumi.com/docs/reference/pkg/java/) - Complete Java API
+
+## Tutorials & Examples
+
+{{- $tutorials := where .Site.RegularPages "Section" "tutorials" }}
+{{- range first 10 $tutorials }}
+  {{- if not (isset .Params "llms_exclude") }}
+- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
+  {{- end }}
+{{- end }}
+- [More Tutorials](https://www.pulumi.com/tutorials/) - Complete list of hands-on learning paths
+- [Examples](https://github.com/pulumi/examples) - Production-ready templates
+- [Templates](https://www.pulumi.com/templates/) - Starter templates for common architectures
+
+## Optional
+
+{{- /* Include select high-value blog posts */ -}}
+{{- $blogPosts := where .Site.RegularPages "Section" "blog" }}
+{{- $recentPosts := first 20 (where $blogPosts ".Date" ">" (now.AddDate -1 0 0)) }}
+{{- range $recentPosts }}
+  {{- if not (isset .Params "llms_exclude") }}
+    {{- if or (in .Title "Infrastructure as Code") (in .Title "Terraform") (in .Title "CloudFormation") (in .Title "AWS CDK") (in .Title "Kubernetes") (in .Title "Getting Started") }}
+- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
+    {{- end }}
+  {{- end }}
+{{- end }}
+- [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
+- [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi
+- [What Is Pulumi?](https://www.pulumi.com/what-is/what-is-pulumi/) - Detailed explanation
+- [Community](https://www.pulumi.com/community/) - Slack, discussions, events
\ No newline at end of file
diff --git a/layouts/partials/assets.html b/layouts/partials/assets.html
index 7b2543a4f92e..1c4151970cd3 100644
--- a/layouts/partials/assets.html
+++ b/layouts/partials/assets.html
@@ -12,6 +12,9 @@
 {{ $js := resources.Get "js/bundle.js" | fingerprint }}
 <script src="{{ $js.RelPermalink }}" defer></script>
 
+<!-- AI Bot Tracking for llms.txt experiment -->
+<script src="{{ "/js/ai-bot-tracking.js" | relURL }}" defer></script>
+
 <!--
   Our marketing styles diverge a bit from the other parts of the website so we load
   in a different bundle on our marketing pages.
diff --git a/static/js/ai-bot-tracking.js b/static/js/ai-bot-tracking.js
new file mode 100644
index 000000000000..e98d03d4c863
--- /dev/null
+++ b/static/js/ai-bot-tracking.js
@@ -0,0 +1,117 @@
+/**
+ * AI Bot Detection and Tracking for llms.txt experiment
+ * Tracks AI/LLM bot access to pulumi.com to measure llms.txt effectiveness
+ */
+
+(function() {
+    'use strict';
+
+    /**
+     * Detects if the current user agent is an AI/LLM bot
+     * Based on actual bot traffic data from pulumi.com analytics
+     * @returns {boolean} true if AI bot detected
+     */
+    function detectAIBot() {
+        const ua = navigator.userAgent.toLowerCase();
+
+        // Known AI bot patterns from actual traffic data
+        const aiPatterns = [
+            /claudebot\/\d+\.\d+/i,           // ClaudeBot/1.0
+            /amazonbot\/\d+\.\d+/i,           // Amazonbot/0.1
+            /chatgpt-user\/\d+\.\d+/i,        // ChatGPT-User/1.0
+            /oai-searchbot\/\d+\.\d+/i,       // OAI-SearchBot/1.0, 1.3
+            /perplexitybot\/\d+\.\d+/i,       // PerplexityBot/1.0
+            /gptbot/i,                         // GPTBot (general)
+            /anthropic/i,                      // Anthropic bots
+            /bingbot/i,                        // Bing (may use AI)
+            /baiduspider/i,                    // Baidu (has AI features)
+            /\bbot\b.*\/(ai|llm|gpt|claude)/i // Future-proofing pattern
+        ];
+
+        return aiPatterns.some(pattern => pattern.test(ua));
+    }
+
+    /**
+     * Sends tracking event to analytics
+     * @param {string} eventName - Name of the event
+     * @param {object} eventData - Additional event data
+     */
+    function trackEvent(eventName, eventData) {
+        // Use Segment analytics if available
+        if (typeof window.analytics !== 'undefined' && window.analytics.track) {
+            window.analytics.track(eventName, eventData);
+        }
+
+        // Also send to Google Analytics if available
+        if (typeof gtag !== 'undefined') {
+            gtag('event', eventName, {
+                'event_category': 'llms_txt',
+                ...eventData
+            });
+        }
+
+        // Log to console in development
+        if (window.location.hostname === 'localhost' || window.location.hostname === '127.0.0.1') {
+            console.log('[AI Bot Tracking]', eventName, eventData);
+        }
+    }
+
+    /**
+     * Initialize AI bot tracking
+     */
+    function initAIBotTracking() {
+        // Check if current visitor is an AI bot
+        const isAIBot = detectAIBot();
+
+        if (isAIBot) {
+            // Track AI bot page access
+            trackEvent('ai_bot_access', {
+                user_agent: navigator.userAgent,
+                page_path: window.location.pathname,
+                page_url: window.location.href,
+                referrer: document.referrer,
+                timestamp: new Date().toISOString()
+            });
+
+            // Add data attribute to body for debugging
+            document.body.setAttribute('data-ai-bot', 'true');
+        }
+
+        // Track llms.txt file access specifically
+        if (window.location.pathname === '/llms.txt') {
+            trackEvent('llms_txt_fetch', {
+                user_agent: navigator.userAgent,
+                is_ai_bot: isAIBot,
+                referrer: document.referrer,
+                timestamp: new Date().toISOString()
+            });
+        }
+
+        // Track .md version requests (future enhancement)
+        if (window.location.pathname.endsWith('.md')) {
+            trackEvent('markdown_version_request', {
+                user_agent: navigator.userAgent,
+                is_ai_bot: isAIBot,
+                page_path: window.location.pathname,
+                referrer: document.referrer,
+                timestamp: new Date().toISOString()
+            });
+        }
+    }
+
+    // Wait for DOM and analytics to be ready
+    if (document.readyState === 'loading') {
+        document.addEventListener('DOMContentLoaded', initAIBotTracking);
+    } else {
+        // DOM already loaded, initialize immediately
+        // Give analytics libraries a moment to load
+        setTimeout(initAIBotTracking, 100);
+    }
+
+    // Export functions for testing/debugging
+    window.pulumiAITracking = {
+        detectAIBot: detectAIBot,
+        trackEvent: trackEvent,
+        initAIBotTracking: initAIBotTracking
+    };
+})();
\ No newline at end of file
diff --git a/static/llms.txt b/static/llms.txt
new file mode 100644
index 000000000000..d340406bbf6d
--- /dev/null
+++ b/static/llms.txt
@@ -0,0 +1,69 @@
+# Pulumi
+
+> Infrastructure as code using real programming languages. Write, test, and deploy cloud infrastructure with TypeScript, Python, Go, C#, or Java instead of YAML. Get IDE support, type safety, and automated testing for AWS, Azure, GCP, Kubernetes, and 150+ providers. Deploy 3x faster with 90% fewer configuration errors.
+
+## Getting Started
+
+- [Installation](https://www.pulumi.com/docs/iac/get-started/install/) - Install Pulumi CLI and tools
+- [AWS Quick Start](https://www.pulumi.com/docs/iac/get-started/aws/) - Deploy first AWS infrastructure
+- [Azure Quick Start](https://www.pulumi.com/docs/iac/get-started/azure/) - Deploy first Azure infrastructure
+- [GCP Quick Start](https://www.pulumi.com/docs/iac/get-started/gcp/) - Deploy first GCP infrastructure
+- [Kubernetes Quick Start](https://www.pulumi.com/docs/iac/get-started/kubernetes/) - Deploy to Kubernetes
+- [Core Concepts](https://www.pulumi.com/docs/iac/concepts/) - Projects, stacks, resources, and state
+
+## Infrastructure as Code (IaC)
+
+- [Why Pulumi?](https://www.pulumi.com/docs/iac/) - Real code vs YAML configuration
+- [Languages & SDKs](https://www.pulumi.com/docs/iac/languages-sdks/) - TypeScript, Python, Go, C#, Java, YAML
+- [Cloud Providers](https://www.pulumi.com/docs/iac/clouds/) - AWS, Azure, GCP, Kubernetes guides
+- [CLI Commands](https://www.pulumi.com/docs/iac/cli/commands/) - Complete command reference
+- [Automation API](https://www.pulumi.com/docs/iac/automation-api/) - Embed Pulumi in your applications
+- [CrossGuard Policies](https://www.pulumi.com/docs/iac/crossguard/) - Policy as code enforcement
+- [Terraform Migration](https://www.pulumi.com/docs/iac/adopting-pulumi/migrate-terraform/) - Migrate from Terraform
+- [CloudFormation Migration](https://www.pulumi.com/docs/iac/adopting-pulumi/migrate-cloudformation/) - Migrate from CloudFormation
+
+## Pulumi ESC (Environments, Secrets, Configuration)
+
+- [Overview](https://www.pulumi.com/docs/esc/) - Centralized secrets and configuration management
+- [Get Started](https://www.pulumi.com/docs/esc/get-started/) - Quick start guide
+- [Environments](https://www.pulumi.com/docs/esc/environments/) - Define and manage environments
+- [Integrations](https://www.pulumi.com/docs/esc/integrations/) - Integrate with external secret stores
+- [CLI Reference](https://www.pulumi.com/docs/esc/cli/) - ESC command reference
+
+## Pulumi Cloud Platform
+
+- [Overview](https://www.pulumi.com/docs/pulumi-cloud/) - Managed platform capabilities
+- [Deployments](https://www.pulumi.com/docs/pulumi-cloud/deployments/) - Automated deployments
+- [Pulumi Copilot](https://www.pulumi.com/docs/pulumi-cloud/copilot/) - AI-powered infrastructure
+- [Access Management](https://www.pulumi.com/docs/pulumi-cloud/access-management/) - Teams, roles, RBAC
+- [Audit Logs](https://www.pulumi.com/docs/pulumi-cloud/admin/audit-logs/) - Compliance and security
+- [Pulumi Insights](https://www.pulumi.com/docs/insights/) - Infrastructure intelligence and search
+
+## Registry (Providers & Packages)
+
+- [AWS Provider](https://www.pulumi.com/registry/packages/aws/) - Complete AWS resource coverage
+- [Azure Native Provider](https://www.pulumi.com/registry/packages/azure-native/) - Full Azure ARM API
+- [Google Cloud Provider](https://www.pulumi.com/registry/packages/gcp/) - Complete GCP coverage
+- [Kubernetes Provider](https://www.pulumi.com/registry/packages/kubernetes/) - Native Kubernetes support
+- [All Providers](https://www.pulumi.com/registry/) - 150+ cloud and SaaS providers
+
+## API Reference
+
+- [Node.js/TypeScript](https://www.pulumi.com/docs/reference/pkg/nodejs/) - Complete TypeScript/JavaScript API
+- [Python](https://www.pulumi.com/docs/reference/pkg/python/) - Complete Python API
+- [Go](https://www.pulumi.com/docs/reference/pkg/go/) - Complete Go API
+- [.NET/C#](https://www.pulumi.com/docs/reference/pkg/dotnet/) - Complete .NET API
+- [Java](https://www.pulumi.com/docs/reference/pkg/java/) - Complete Java API
+
+## Tutorials & Examples
+
+- [Tutorials](https://www.pulumi.com/tutorials/) - Hands-on learning paths
+- [Examples](https://github.com/pulumi/examples) - Production-ready templates
+- [Templates](https://www.pulumi.com/templates/) - Starter templates for common architectures
+
+## Optional
+
+- [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
+- [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi
+- [What Is Pulumi?](https://www.pulumi.com/what-is/what-is-pulumi/) - Detailed explanation
+- [Community](https://www.pulumi.com/community/) - Slack, discussions, events
\ No newline at end of file

From e9bc8abc4c1928cac8e6937868e2f0fa219863e1 Mon Sep 17 00:00:00 2001
From: asaf <asaf@pulumi.com>
Date: Mon, 24 Nov 2025 13:48:26 -0500
Subject: [PATCH 2/6] fix: Remove specific metrics claim from llms.txt
 blockquote

Remove '3x faster with 90% fewer configuration errors' from the blockquote
summary to keep the messaging focused on core value propositions rather
than specific metrics.
---
 layouts/index.llmstxt.txt | 2 +-
 static/llms.txt           | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/layouts/index.llmstxt.txt b/layouts/index.llmstxt.txt
index 1c3089878f65..eab138a9c02c 100644
--- a/layouts/index.llmstxt.txt
+++ b/layouts/index.llmstxt.txt
@@ -2,7 +2,7 @@
 {{- /* Static version for now - dynamic generation can be enabled later */ -}}
 # Pulumi
 
-> Infrastructure as code using real programming languages. Write, test, and deploy cloud infrastructure with TypeScript, Python, Go, C#, or Java instead of YAML. Get IDE support, type safety, and automated testing for AWS, Azure, GCP, Kubernetes, and 150+ providers. Deploy 3x faster with 90% fewer configuration errors.
+> Infrastructure as code using real programming languages. Write, test, and deploy cloud infrastructure with TypeScript, Python, Go, C#, or Java instead of YAML. Get IDE support, type safety, and automated testing for AWS, Azure, GCP, Kubernetes, and 150+ providers.
 
 ## Getting Started
 
diff --git a/static/llms.txt b/static/llms.txt
index d340406bbf6d..a4456b68839d 100644
--- a/static/llms.txt
+++ b/static/llms.txt
@@ -1,6 +1,6 @@
 # Pulumi
 
-> Infrastructure as code using real programming languages. Write, test, and deploy cloud infrastructure with TypeScript, Python, Go, C#, or Java instead of YAML. Get IDE support, type safety, and automated testing for AWS, Azure, GCP, Kubernetes, and 150+ providers. Deploy 3x faster with 90% fewer configuration errors.
+> Infrastructure as code using real programming languages. Write, test, and deploy cloud infrastructure with TypeScript, Python, Go, C#, or Java instead of YAML. Get IDE support, type safety, and automated testing for AWS, Azure, GCP, Kubernetes, and 150+ providers.
 
 ## Getting Started
 

From de16f463c4e36a11364bcbf9ab14e16cef4e8f56 Mon Sep 17 00:00:00 2001
From: asaf <asaf@pulumi.com>
Date: Mon, 24 Nov 2025 14:02:12 -0500
Subject: [PATCH 3/6] fix: Address PR review feedback for llms.txt
 implementation

- Add trailing newlines to all new files (llms.txt, index.llmstxt.txt, ai-bot-tracking.js)
- Clarify AGENTS.md documentation: .md URLs are a future enhancement, not currently implemented
- Remove Hugo template comment that was generating unwanted blank line in output
- Add clarification to JavaScript that bot detection is supplemental client-side tracking
  (primary measurement is via server-side access logs)

These changes address all required fixes from the documentation review
and improve code clarity per reviewer suggestions.
---
 AGENTS.md                    | 2 +-
 layouts/index.llmstxt.txt    | 4 +---
 static/js/ai-bot-tracking.js | 8 +++++---
 static/llms.txt              | 2 +-
 4 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/AGENTS.md b/AGENTS.md
index 6f2fa3b74d64..1b6230275a98 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -23,7 +23,7 @@ This repository implements the llms.txt standard for improved AI/LLM consumption
   - `/docs/pulumi-cloud/` - Platform capabilities
   - `/registry/` - Provider and package documentation
   - `/docs/reference/` - API references for all languages
-- **Markdown versions**: When available, append `.md` to URLs for cleaner content
+- **Markdown versions**: Future enhancement - URLs may support `.md` extension for markdown-only output
 - **Excluding content from llms.txt**: Add `llms_exclude: true` to front matter
 
 When working with Pulumi documentation, consult the llms.txt file first for structured navigation and understanding of the documentation hierarchy.
diff --git a/layouts/index.llmstxt.txt b/layouts/index.llmstxt.txt
index eab138a9c02c..9fb82c7a661f 100644
--- a/layouts/index.llmstxt.txt
+++ b/layouts/index.llmstxt.txt
@@ -74,8 +74,6 @@
 - [Templates](https://www.pulumi.com/templates/) - Starter templates for common architectures
 
 ## Optional
-
-{{- /* Include select high-value blog posts */ -}}
 {{- $blogPosts := where .Site.RegularPages "Section" "blog" }}
 {{- $recentPosts := first 20 (where $blogPosts ".Date" ">" (now.AddDate -1 0 0)) }}
 {{- range $recentPosts }}
@@ -88,4 +86,4 @@
 - [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
 - [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi
 - [What Is Pulumi?](https://www.pulumi.com/what-is/what-is-pulumi/) - Detailed explanation
-- [Community](https://www.pulumi.com/community/) - Slack, discussions, events
\ No newline at end of file
+- [Community](https://www.pulumi.com/community/) - Slack, discussions, events
diff --git a/static/js/ai-bot-tracking.js b/static/js/ai-bot-tracking.js
index e98d03d4c863..e8c9d5df8038 100644
--- a/static/js/ai-bot-tracking.js
+++ b/static/js/ai-bot-tracking.js
@@ -7,8 +7,10 @@
     'use strict';
 
     /**
-     * Detects if the current user agent is an AI/LLM bot
-     * Based on actual bot traffic data from pulumi.com analytics
+     * Detects if the current user agent is an AI/LLM bot (CLIENT-SIDE ONLY)
+     * Note: Most LLM bots fetch llms.txt server-side without executing JavaScript.
+     * This detection is supplemental to server-side bot tracking in access logs.
+     * Based on actual bot traffic data from pulumi.com analytics.
      * @returns {boolean} true if AI bot detected
      */
     function detectAIBot() {
@@ -114,4 +116,4 @@
         trackEvent: trackEvent,
         initAIBotTracking: initAIBotTracking
     };
-})();
\ No newline at end of file
+})();
diff --git a/static/llms.txt b/static/llms.txt
index a4456b68839d..af27b26e654e 100644
--- a/static/llms.txt
+++ b/static/llms.txt
@@ -66,4 +66,4 @@
 - [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
 - [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi
 - [What Is Pulumi?](https://www.pulumi.com/what-is/what-is-pulumi/) - Detailed explanation
-- [Community](https://www.pulumi.com/community/) - Slack, discussions, events
\ No newline at end of file
+- [Community](https://www.pulumi.com/community/) - Slack, discussions, events

From fc483771219135af7dfa47763a8ba07ef9241c87 Mon Sep 17 00:00:00 2001
From: asaf <asaf@pulumi.com>
Date: Mon, 24 Nov 2025 14:46:38 -0500
Subject: [PATCH 4/6] feat: Address sicarul's feedback on llms.txt
 implementation

Based on PR review feedback:

1. Move blog content out of 'Optional' section
   - Created new 'Blog & Resources' section with higher priority
   - Blog posts are valuable for LLMs to understand recent features and best practices
   - Case studies remain with blog as related resources

2. Clarify JavaScript/HTML separation
   - Added clear comments explaining ai-bot-tracking.js is ONLY for HTML pages
   - Emphasized that JavaScript does NOT interact with llms.txt file
   - llms.txt is served as plain text, completely separate from website JS

3. Improve blog template visibility
   - Added example blog posts to static llms.txt to show intended output
   - Added comment explaining Hugo dynamically generates blog posts
   - Template filters for high-value posts (IaC, Terraform, Kubernetes topics)

These changes address confusion about file purposes and improve content organization.
---
 layouts/index.llmstxt.txt    | 9 +++++++--
 layouts/partials/assets.html | 6 +++++-
 static/js/ai-bot-tracking.js | 9 ++++++++-
 static/llms.txt              | 7 ++++++-
 4 files changed, 26 insertions(+), 5 deletions(-)

diff --git a/layouts/index.llmstxt.txt b/layouts/index.llmstxt.txt
index 9fb82c7a661f..fbd5b4405491 100644
--- a/layouts/index.llmstxt.txt
+++ b/layouts/index.llmstxt.txt
@@ -73,9 +73,12 @@
 - [Examples](https://github.com/pulumi/examples) - Production-ready templates
 - [Templates](https://www.pulumi.com/templates/) - Starter templates for common architectures
 
-## Optional
+## Blog & Resources
+{{- /* The following code dynamically generates recent blog posts when Hugo builds the site */ -}}
 {{- $blogPosts := where .Site.RegularPages "Section" "blog" }}
 {{- $recentPosts := first 20 (where $blogPosts ".Date" ">" (now.AddDate -1 0 0)) }}
+
+- [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
 {{- range $recentPosts }}
   {{- if not (isset .Params "llms_exclude") }}
     {{- if or (in .Title "Infrastructure as Code") (in .Title "Terraform") (in .Title "CloudFormation") (in .Title "AWS CDK") (in .Title "Kubernetes") (in .Title "Getting Started") }}
@@ -83,7 +86,9 @@
     {{- end }}
   {{- end }}
 {{- end }}
-- [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
 - [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi
+
+## Optional
+
 - [What Is Pulumi?](https://www.pulumi.com/what-is/what-is-pulumi/) - Detailed explanation
 - [Community](https://www.pulumi.com/community/) - Slack, discussions, events
diff --git a/layouts/partials/assets.html b/layouts/partials/assets.html
index 1c4151970cd3..4ab9d835404e 100644
--- a/layouts/partials/assets.html
+++ b/layouts/partials/assets.html
@@ -12,7 +12,11 @@
 {{ $js := resources.Get "js/bundle.js" | fingerprint }}
 <script src="{{ $js.RelPermalink }}" defer></script>
 
-<!-- AI Bot Tracking for llms.txt experiment -->
+<!--
+  AI Bot Tracking for llms.txt experiment
+  This JavaScript runs on HTML pages to track AI bot visits for analytics.
+  It does NOT affect the llms.txt file which is served as plain text at /llms.txt
+-->
 <script src="{{ "/js/ai-bot-tracking.js" | relURL }}" defer></script>
 
 <!--
diff --git a/static/js/ai-bot-tracking.js b/static/js/ai-bot-tracking.js
index e8c9d5df8038..7a2c8b12c7ad 100644
--- a/static/js/ai-bot-tracking.js
+++ b/static/js/ai-bot-tracking.js
@@ -1,6 +1,13 @@
 /**
  * AI Bot Detection and Tracking for llms.txt experiment
- * Tracks AI/LLM bot access to pulumi.com to measure llms.txt effectiveness
+ *
+ * IMPORTANT: This JavaScript file is for the main Pulumi website HTML pages ONLY.
+ * It does NOT interact with or affect the llms.txt file itself.
+ *
+ * Purpose: Track AI bot visits to pulumi.com HTML pages to measure the llms.txt
+ * experiment's impact on organic traffic from AI-driven discovery channels.
+ *
+ * The llms.txt file is a separate plain text file served at /llms.txt
  */
 
 (function() {
diff --git a/static/llms.txt b/static/llms.txt
index af27b26e654e..e69c03b73629 100644
--- a/static/llms.txt
+++ b/static/llms.txt
@@ -61,9 +61,14 @@
 - [Examples](https://github.com/pulumi/examples) - Production-ready templates
 - [Templates](https://www.pulumi.com/templates/) - Starter templates for common architectures
 
-## Optional
+## Blog & Resources
 
 - [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
+- [Infrastructure as Code Best Practices](https://www.pulumi.com/blog/iac-best-practices/) - Key patterns for production IaC
+- [Migrating from Terraform to Pulumi](https://www.pulumi.com/blog/migrating-from-terraform/) - Step-by-step migration guide
 - [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi
+
+## Optional
+
 - [What Is Pulumi?](https://www.pulumi.com/what-is/what-is-pulumi/) - Detailed explanation
 - [Community](https://www.pulumi.com/community/) - Slack, discussions, events

From 45acecb0902ec2186392c3c2dc9f148674e9d057 Mon Sep 17 00:00:00 2001
From: asaf <asaf@pulumi.com>
Date: Mon, 24 Nov 2025 15:00:09 -0500
Subject: [PATCH 5/6] Simplify llms.txt implementation

- Remove AI bot tracking JavaScript entirely
- Remove script reference from assets.html
- Simplify Hugo template by removing complex blog post generation logic
- Update static llms.txt to match simplified structure
- Keep blog section as simple link without specific posts

This addresses PR review feedback about unnecessary complexity and confusion around the JavaScript tracking
---
 layouts/index.llmstxt.txt    |  10 ---
 layouts/partials/assets.html |   6 --
 static/js/ai-bot-tracking.js | 126 -----------------------------------
 static/llms.txt              |   2 -
 4 files changed, 144 deletions(-)
 delete mode 100644 static/js/ai-bot-tracking.js

diff --git a/layouts/index.llmstxt.txt b/layouts/index.llmstxt.txt
index fbd5b4405491..dbb68555b147 100644
--- a/layouts/index.llmstxt.txt
+++ b/layouts/index.llmstxt.txt
@@ -74,18 +74,8 @@
 - [Templates](https://www.pulumi.com/templates/) - Starter templates for common architectures
 
 ## Blog & Resources
-{{- /* The following code dynamically generates recent blog posts when Hugo builds the site */ -}}
-{{- $blogPosts := where .Site.RegularPages "Section" "blog" }}
-{{- $recentPosts := first 20 (where $blogPosts ".Date" ">" (now.AddDate -1 0 0)) }}
 
 - [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
-{{- range $recentPosts }}
-  {{- if not (isset .Params "llms_exclude") }}
-    {{- if or (in .Title "Infrastructure as Code") (in .Title "Terraform") (in .Title "CloudFormation") (in .Title "AWS CDK") (in .Title "Kubernetes") (in .Title "Getting Started") }}
-- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
-    {{- end }}
-  {{- end }}
-{{- end }}
 - [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi
 
 ## Optional
diff --git a/layouts/partials/assets.html b/layouts/partials/assets.html
index 4ab9d835404e..b7e48639d865 100644
--- a/layouts/partials/assets.html
+++ b/layouts/partials/assets.html
@@ -12,12 +12,6 @@
 {{ $js := resources.Get "js/bundle.js" | fingerprint }}
 <script src="{{ $js.RelPermalink }}" defer></script>
 
-<!--
-  AI Bot Tracking for llms.txt experiment
-  This JavaScript runs on HTML pages to track AI bot visits for analytics.
-  It does NOT affect the llms.txt file which is served as plain text at /llms.txt
--->
-<script src="{{ "/js/ai-bot-tracking.js" | relURL }}" defer></script>
 
 <!--
   Our marketing styles diverge a bit from the other parts of the website so we load
diff --git a/static/js/ai-bot-tracking.js b/static/js/ai-bot-tracking.js
deleted file mode 100644
index 7a2c8b12c7ad..000000000000
--- a/static/js/ai-bot-tracking.js
+++ /dev/null
@@ -1,126 +0,0 @@
-/**
- * AI Bot Detection and Tracking for llms.txt experiment
- *
- * IMPORTANT: This JavaScript file is for the main Pulumi website HTML pages ONLY.
- * It does NOT interact with or affect the llms.txt file itself.
- *
- * Purpose: Track AI bot visits to pulumi.com HTML pages to measure the llms.txt
- * experiment's impact on organic traffic from AI-driven discovery channels.
- *
- * The llms.txt file is a separate plain text file served at /llms.txt
- */
-
-(function() {
-    'use strict';
-
-    /**
-     * Detects if the current user agent is an AI/LLM bot (CLIENT-SIDE ONLY)
-     * Note: Most LLM bots fetch llms.txt server-side without executing JavaScript.
-     * This detection is supplemental to server-side bot tracking in access logs.
-     * Based on actual bot traffic data from pulumi.com analytics.
-     * @returns {boolean} true if AI bot detected
-     */
-    function detectAIBot() {
-        const ua = navigator.userAgent.toLowerCase();
-
-        // Known AI bot patterns from actual traffic data
-        const aiPatterns = [
-            /claudebot\/\d+\.\d+/i,           // ClaudeBot/1.0
-            /amazonbot\/\d+\.\d+/i,           // Amazonbot/0.1
-            /chatgpt-user\/\d+\.\d+/i,        // ChatGPT-User/1.0
-            /oai-searchbot\/\d+\.\d+/i,       // OAI-SearchBot/1.0, 1.3
-            /perplexitybot\/\d+\.\d+/i,       // PerplexityBot/1.0
-            /gptbot/i,                         // GPTBot (general)
-            /anthropic/i,                      // Anthropic bots
-            /bingbot/i,                        // Bing (may use AI)
-            /baiduspider/i,                    // Baidu (has AI features)
-            /\bbot\b.*\/(ai|llm|gpt|claude)/i // Future-proofing pattern
-        ];
-
-        return aiPatterns.some(pattern => pattern.test(ua));
-    }
-
-    /**
-     * Sends tracking event to analytics
-     * @param {string} eventName - Name of the event
-     * @param {object} eventData - Additional event data
-     */
-    function trackEvent(eventName, eventData) {
-        // Use Segment analytics if available
-        if (typeof window.analytics !== 'undefined' && window.analytics.track) {
-            window.analytics.track(eventName, eventData);
-        }
-
-        // Also send to Google Analytics if available
-        if (typeof gtag !== 'undefined') {
-            gtag('event', eventName, {
-                'event_category': 'llms_txt',
-                ...eventData
-            });
-        }
-
-        // Log to console in development
-        if (window.location.hostname === 'localhost' || window.location.hostname === '127.0.0.1') {
-            console.log('[AI Bot Tracking]', eventName, eventData);
-        }
-    }
-
-    /**
-     * Initialize AI bot tracking
-     */
-    function initAIBotTracking() {
-        // Check if current visitor is an AI bot
-        const isAIBot = detectAIBot();
-
-        if (isAIBot) {
-            // Track AI bot page access
-            trackEvent('ai_bot_access', {
-                user_agent: navigator.userAgent,
-                page_path: window.location.pathname,
-                page_url: window.location.href,
-                referrer: document.referrer,
-                timestamp: new Date().toISOString()
-            });
-
-            // Add data attribute to body for debugging
-            document.body.setAttribute('data-ai-bot', 'true');
-        }
-
-        // Track llms.txt file access specifically
-        if (window.location.pathname === '/llms.txt') {
-            trackEvent('llms_txt_fetch', {
-                user_agent: navigator.userAgent,
-                is_ai_bot: isAIBot,
-                referrer: document.referrer,
-                timestamp: new Date().toISOString()
-            });
-        }
-
-        // Track .md version requests (future enhancement)
-        if (window.location.pathname.endsWith('.md')) {
-            trackEvent('markdown_version_request', {
-                user_agent: navigator.userAgent,
-                is_ai_bot: isAIBot,
-                page_path: window.location.pathname,
-                referrer: document.referrer,
-                timestamp: new Date().toISOString()
-            });
-        }
-    }
-
-    // Wait for DOM and analytics to be ready
-    if (document.readyState === 'loading') {
-        document.addEventListener('DOMContentLoaded', initAIBotTracking);
-    } else {
-        // DOM already loaded, initialize immediately
-        // Give analytics libraries a moment to load
-        setTimeout(initAIBotTracking, 100);
-    }
-
-    // Export functions for testing/debugging
-    window.pulumiAITracking = {
-        detectAIBot: detectAIBot,
-        trackEvent: trackEvent,
-        initAIBotTracking: initAIBotTracking
-    };
-})();
diff --git a/static/llms.txt b/static/llms.txt
index e69c03b73629..3e0a546fdbc9 100644
--- a/static/llms.txt
+++ b/static/llms.txt
@@ -64,8 +64,6 @@
 ## Blog & Resources
 
 - [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
-- [Infrastructure as Code Best Practices](https://www.pulumi.com/blog/iac-best-practices/) - Key patterns for production IaC
-- [Migrating from Terraform to Pulumi](https://www.pulumi.com/blog/migrating-from-terraform/) - Step-by-step migration guide
 - [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi
 
 ## Optional

From 2bc1e34af26313bd18f1e32f5a2d28e30ad42e92 Mon Sep 17 00:00:00 2001
From: asaf <asaf@pulumi.com>
Date: Mon, 24 Nov 2025 15:47:33 -0500
Subject: [PATCH 6/6] Simplify llms.txt structure

- Merge Tutorials & Examples and Blog & Resources into unified Resources section
- Remove Optional section (What Is Pulumi?, Community)
- Remove CrossGuard reference from IaC section
- Rename Hugo template from index.llmstxt.txt to index.llms.txt
- Update output format name from LLMSTXT to llms in config
---
 config/_default/config.yml                    |  2 +-
 layouts/{index.llmstxt.txt => index.llms.txt} | 20 +++----------------
 static/llms.txt                               | 11 +---------
 3 files changed, 5 insertions(+), 28 deletions(-)
 rename layouts/{index.llmstxt.txt => index.llms.txt} (84%)

diff --git a/config/_default/config.yml b/config/_default/config.yml
index 96b11dc0d01f..c128228ab70c 100644
--- a/config/_default/config.yml
+++ b/config/_default/config.yml
@@ -33,7 +33,7 @@ timeout: 300000
 outputFormats:
   rss:
     baseName: rss
-  LLMSTXT:
+  llms:
     mediaType: "text/plain"
     baseName: "llms"
     isPlainText: true
diff --git a/layouts/index.llmstxt.txt b/layouts/index.llms.txt
similarity index 84%
rename from layouts/index.llmstxt.txt
rename to layouts/index.llms.txt
index dbb68555b147..a9e131be8916 100644
--- a/layouts/index.llmstxt.txt
+++ b/layouts/index.llms.txt
@@ -18,7 +18,7 @@
 - [Why Pulumi?](https://www.pulumi.com/docs/iac/) - Real code vs YAML configuration
 {{- range where .Site.RegularPages "Section" "docs" }}
   {{- if and (in .RelPermalink "/iac/") (not (in .RelPermalink "/get-started/")) (not (isset .Params "llms_exclude")) }}
-    {{- if or (in .RelPermalink "/languages-sdks/") (in .RelPermalink "/clouds/") (in .RelPermalink "/cli/") (in .RelPermalink "/automation-api/") (in .RelPermalink "/crossguard/") (in .RelPermalink "/adopting-pulumi/") }}
+    {{- if or (in .RelPermalink "/languages-sdks/") (in .RelPermalink "/clouds/") (in .RelPermalink "/cli/") (in .RelPermalink "/automation-api/") (in .RelPermalink "/adopting-pulumi/") }}
 - [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
     {{- end }}
   {{- end }}
@@ -61,24 +61,10 @@
 - [.NET/C#](https://www.pulumi.com/docs/reference/pkg/dotnet/) - Complete .NET API
 - [Java](https://www.pulumi.com/docs/reference/pkg/java/) - Complete Java API
 
-## Tutorials & Examples
+## Resources
 
-{{- $tutorials := where .Site.RegularPages "Section" "tutorials" }}
-{{- range first 10 $tutorials }}
-  {{- if not (isset .Params "llms_exclude") }}
-- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
-  {{- end }}
-{{- end }}
-- [More Tutorials](https://www.pulumi.com/tutorials/) - Complete list of hands-on learning paths
+- [Tutorials](https://www.pulumi.com/tutorials/) - Hands-on learning paths
 - [Examples](https://github.com/pulumi/examples) - Production-ready templates
 - [Templates](https://www.pulumi.com/templates/) - Starter templates for common architectures
-
-## Blog & Resources
-
 - [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
 - [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi
-
-## Optional
-
-- [What Is Pulumi?](https://www.pulumi.com/what-is/what-is-pulumi/) - Detailed explanation
-- [Community](https://www.pulumi.com/community/) - Slack, discussions, events
diff --git a/static/llms.txt b/static/llms.txt
index 3e0a546fdbc9..992c470a29c3 100644
--- a/static/llms.txt
+++ b/static/llms.txt
@@ -18,7 +18,6 @@
 - [Cloud Providers](https://www.pulumi.com/docs/iac/clouds/) - AWS, Azure, GCP, Kubernetes guides
 - [CLI Commands](https://www.pulumi.com/docs/iac/cli/commands/) - Complete command reference
 - [Automation API](https://www.pulumi.com/docs/iac/automation-api/) - Embed Pulumi in your applications
-- [CrossGuard Policies](https://www.pulumi.com/docs/iac/crossguard/) - Policy as code enforcement
 - [Terraform Migration](https://www.pulumi.com/docs/iac/adopting-pulumi/migrate-terraform/) - Migrate from Terraform
 - [CloudFormation Migration](https://www.pulumi.com/docs/iac/adopting-pulumi/migrate-cloudformation/) - Migrate from CloudFormation
 
@@ -55,18 +54,10 @@
 - [.NET/C#](https://www.pulumi.com/docs/reference/pkg/dotnet/) - Complete .NET API
 - [Java](https://www.pulumi.com/docs/reference/pkg/java/) - Complete Java API
 
-## Tutorials & Examples
+## Resources
 
 - [Tutorials](https://www.pulumi.com/tutorials/) - Hands-on learning paths
 - [Examples](https://github.com/pulumi/examples) - Production-ready templates
 - [Templates](https://www.pulumi.com/templates/) - Starter templates for common architectures
-
-## Blog & Resources
-
 - [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
 - [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi
-
-## Optional
-
-- [What Is Pulumi?](https://www.pulumi.com/what-is/what-is-pulumi/) - Detailed explanation
-- [Community](https://www.pulumi.com/community/) - Slack, discussions, events