Add llms.txt implementation for improved AI/LLM visibility (#16656)

* feat: Add llms.txt implementation for AI/LLM consumption

Implement llms.txt standard to improve Pulumi's visibility in AI-generated content and responses.

Changes:
- Add static llms.txt file with curated Pulumi documentation structure
- Configure Hugo to generate llms.txt dynamically via new output format
- Create Hugo template for future dynamic llms.txt generation
- Add AI bot tracking JavaScript to measure experiment effectiveness
- Update AGENTS.md with llms.txt guidance for AI assistants
- Track known AI bots: ClaudeBot, Amazonbot, ChatGPT-User, OAI-SearchBot, PerplexityBot

The blockquote summary uses Pulumi's Entry Segment Messaging emphasizing:
- Real programming languages vs YAML
- IDE support, type safety, automated testing
- Multi-cloud support (AWS, Azure, GCP, Kubernetes, 150+ providers)
- 3x faster deployment with 90% fewer configuration errors

Bot detection based on actual traffic data from pulumi.com analytics.

* fix: Remove specific metrics claim from llms.txt blockquote

Remove '3x faster with 90% fewer configuration errors' from the blockquote
summary to keep the messaging focused on core value propositions rather
than specific metrics.

* fix: Address PR review feedback for llms.txt implementation

- Add trailing newlines to all new files (llms.txt, index.llmstxt.txt, ai-bot-tracking.js)
- Clarify AGENTS.md documentation: .md URLs are a future enhancement, not currently implemented
- Remove Hugo template comment that was generating unwanted blank line in output
- Add clarification to JavaScript that bot detection is supplemental client-side tracking
  (primary measurement is via server-side access logs)

These changes address all required fixes from the documentation review
and improve code clarity per reviewer suggestions.

* feat: Address sicarul's feedback on llms.txt implementation

Based on PR review feedback:

1. Move blog content out of 'Optional' section
   - Created new 'Blog & Resources' section with higher priority
   - Blog posts are valuable for LLMs to understand recent features and best practices
   - Case studies remain with blog as related resources

2. Clarify JavaScript/HTML separation
   - Added clear comments explaining ai-bot-tracking.js is ONLY for HTML pages
   - Emphasized that JavaScript does NOT interact with llms.txt file
   - llms.txt is served as plain text, completely separate from website JS

3. Improve blog template visibility
   - Added example blog posts to static llms.txt to show intended output
   - Added comment explaining Hugo dynamically generates blog posts
   - Template filters for high-value posts (IaC, Terraform, Kubernetes topics)

These changes address confusion about file purposes and improve content organization.

* Simplify llms.txt implementation

- Remove AI bot tracking JavaScript entirely
- Remove script reference from assets.html
- Simplify Hugo template by removing complex blog post generation logic
- Update static llms.txt to match simplified structure
- Keep blog section as simple link without specific posts

This addresses PR review feedback about unnecessary complexity and confusion around the JavaScript tracking

* Simplify llms.txt structure

- Merge Tutorials & Examples and Blog & Resources into unified Resources section
- Remove Optional section (What Is Pulumi?, Community)
- Remove CrossGuard reference from IaC section
- Rename Hugo template from index.llmstxt.txt to index.llms.txt
- Update output format name from LLMSTXT to llms in config

Author: asafashirov

AGENTS.md

@@ -10,6 +10,26 @@ This repository contains the source for the Pulumi website, built with [Hugo](ht
 
 ---
 
+## AI Assistant Information (llms.txt)
+
+This repository implements the llms.txt standard for improved AI/LLM consumption:
+
+- **Primary content index**: https://www.pulumi.com/llms.txt
+- **Standard**: Following [llms.txt specification](https://llmstxt.org/)
+- **Content structure**: Hierarchical documentation index optimized for LLM context windows
+- **Focus areas for AI consumption**:
+  - `/docs/iac/` - Infrastructure as Code documentation
+  - `/docs/esc/` - Secrets and configuration management
+  - `/docs/pulumi-cloud/` - Platform capabilities
+  - `/registry/` - Provider and package documentation
+  - `/docs/reference/` - API references for all languages
+- **Markdown versions**: Future enhancement - URLs may support `.md` extension for markdown-only output
+- **Excluding content from llms.txt**: Add `llms_exclude: true` to front matter
+
+When working with Pulumi documentation, consult the llms.txt file first for structured navigation and understanding of the documentation hierarchy.
+
+---
+
 ## Build / Test / Lint Workflow
 
 Agents must use these exact commands:

config/_default/config.yml

@@ -33,6 +33,11 @@ timeout: 300000
 outputFormats:
   rss:
     baseName: rss
+  llms:
+    mediaType: "text/plain"
+    baseName: "llms"
+    isPlainText: true
+    permalinkable: false
 
 # We set the outputs explicitly to just "HTML" as otherwise they default to both
 # "HTML" and "RSS", and we only want to generate RSS for the blog section -- not
@@ -43,6 +48,7 @@ outputs:
   home:
     - HTML
     - JSON
+    - LLMSTXT
   section:
     - HTML
 

layouts/index.llms.txt

@@ -0,0 +1,70 @@
+{{- /* Generate llms.txt file for AI/LLM consumption */ -}}
+{{- /* Static version for now - dynamic generation can be enabled later */ -}}
+# Pulumi
+
+> Infrastructure as code using real programming languages. Write, test, and deploy cloud infrastructure with TypeScript, Python, Go, C#, or Java instead of YAML. Get IDE support, type safety, and automated testing for AWS, Azure, GCP, Kubernetes, and 150+ providers.
+
+## Getting Started
+
+{{- range where .Site.RegularPages "Section" "docs" }}
+  {{- if and (in .RelPermalink "/get-started/") (not (isset .Params "llms_exclude")) }}
+- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc }}{{ end }}
+  {{- end }}
+{{- end }}
+- [Core Concepts](https://www.pulumi.com/docs/iac/concepts/) - Projects, stacks, resources, and state
+
+## Infrastructure as Code (IaC)
+
+- [Why Pulumi?](https://www.pulumi.com/docs/iac/) - Real code vs YAML configuration
+{{- range where .Site.RegularPages "Section" "docs" }}
+  {{- if and (in .RelPermalink "/iac/") (not (in .RelPermalink "/get-started/")) (not (isset .Params "llms_exclude")) }}
+    {{- if or (in .RelPermalink "/languages-sdks/") (in .RelPermalink "/clouds/") (in .RelPermalink "/cli/") (in .RelPermalink "/automation-api/") (in .RelPermalink "/adopting-pulumi/") }}
+- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
+    {{- end }}
+  {{- end }}
+{{- end }}
+
+## Pulumi ESC (Environments, Secrets, Configuration)
+
+{{- range where .Site.RegularPages "Section" "docs" }}
+  {{- if and (in .RelPermalink "/esc/") (not (isset .Params "llms_exclude")) }}
+    {{- if or (eq .RelPermalink "/docs/esc/") (in .RelPermalink "/get-started/") (in .RelPermalink "/environments/") (in .RelPermalink "/integrations/") (in .RelPermalink "/cli/") }}
+- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
+    {{- end }}
+  {{- end }}
+{{- end }}
+
+## Pulumi Cloud Platform
+
+{{- range where .Site.RegularPages "Section" "docs" }}
+  {{- if and (in .RelPermalink "/pulumi-cloud/") (not (isset .Params "llms_exclude")) }}
+    {{- if or (eq .RelPermalink "/docs/pulumi-cloud/") (in .RelPermalink "/deployments/") (in .RelPermalink "/copilot/") (in .RelPermalink "/access-management/") (in .RelPermalink "/admin/") }}
+- [{{ .Title }}]({{ .Permalink }}){{ if .Params.meta_desc }} - {{ .Params.meta_desc | truncate 100 }}{{ end }}
+    {{- end }}
+  {{- end }}
+{{- end }}
+- [Pulumi Insights](https://www.pulumi.com/docs/insights/) - Infrastructure intelligence and search
+
+## Registry (Providers & Packages)
+
+- [AWS Provider](https://www.pulumi.com/registry/packages/aws/) - Complete AWS resource coverage
+- [Azure Native Provider](https://www.pulumi.com/registry/packages/azure-native/) - Full Azure ARM API
+- [Google Cloud Provider](https://www.pulumi.com/registry/packages/gcp/) - Complete GCP coverage
+- [Kubernetes Provider](https://www.pulumi.com/registry/packages/kubernetes/) - Native Kubernetes support
+- [All Providers](https://www.pulumi.com/registry/) - 150+ cloud and SaaS providers
+
+## API Reference
+
+- [Node.js/TypeScript](https://www.pulumi.com/docs/reference/pkg/nodejs/) - Complete TypeScript/JavaScript API
+- [Python](https://www.pulumi.com/docs/reference/pkg/python/) - Complete Python API
+- [Go](https://www.pulumi.com/docs/reference/pkg/go/) - Complete Go API
+- [.NET/C#](https://www.pulumi.com/docs/reference/pkg/dotnet/) - Complete .NET API
+- [Java](https://www.pulumi.com/docs/reference/pkg/java/) - Complete Java API
+
+## Resources
+
+- [Tutorials](https://www.pulumi.com/tutorials/) - Hands-on learning paths
+- [Examples](https://github.com/pulumi/examples) - Production-ready templates
+- [Templates](https://www.pulumi.com/templates/) - Starter templates for common architectures
+- [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
+- [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi

layouts/partials/assets.html

@@ -12,6 +12,7 @@
 {{ $js := resources.Get "js/bundle.js" | fingerprint }}
 <script src="{{ $js.RelPermalink }}" defer></script>
 
+
 <!--
   Our marketing styles diverge a bit from the other parts of the website so we load
   in a different bundle on our marketing pages.

static/llms.txt

@@ -0,0 +1,63 @@
+# Pulumi
+
+> Infrastructure as code using real programming languages. Write, test, and deploy cloud infrastructure with TypeScript, Python, Go, C#, or Java instead of YAML. Get IDE support, type safety, and automated testing for AWS, Azure, GCP, Kubernetes, and 150+ providers.
+
+## Getting Started
+
+- [Installation](https://www.pulumi.com/docs/iac/get-started/install/) - Install Pulumi CLI and tools
+- [AWS Quick Start](https://www.pulumi.com/docs/iac/get-started/aws/) - Deploy first AWS infrastructure
+- [Azure Quick Start](https://www.pulumi.com/docs/iac/get-started/azure/) - Deploy first Azure infrastructure
+- [GCP Quick Start](https://www.pulumi.com/docs/iac/get-started/gcp/) - Deploy first GCP infrastructure
+- [Kubernetes Quick Start](https://www.pulumi.com/docs/iac/get-started/kubernetes/) - Deploy to Kubernetes
+- [Core Concepts](https://www.pulumi.com/docs/iac/concepts/) - Projects, stacks, resources, and state
+
+## Infrastructure as Code (IaC)
+
+- [Why Pulumi?](https://www.pulumi.com/docs/iac/) - Real code vs YAML configuration
+- [Languages & SDKs](https://www.pulumi.com/docs/iac/languages-sdks/) - TypeScript, Python, Go, C#, Java, YAML
+- [Cloud Providers](https://www.pulumi.com/docs/iac/clouds/) - AWS, Azure, GCP, Kubernetes guides
+- [CLI Commands](https://www.pulumi.com/docs/iac/cli/commands/) - Complete command reference
+- [Automation API](https://www.pulumi.com/docs/iac/automation-api/) - Embed Pulumi in your applications
+- [Terraform Migration](https://www.pulumi.com/docs/iac/adopting-pulumi/migrate-terraform/) - Migrate from Terraform
+- [CloudFormation Migration](https://www.pulumi.com/docs/iac/adopting-pulumi/migrate-cloudformation/) - Migrate from CloudFormation
+
+## Pulumi ESC (Environments, Secrets, Configuration)
+
+- [Overview](https://www.pulumi.com/docs/esc/) - Centralized secrets and configuration management
+- [Get Started](https://www.pulumi.com/docs/esc/get-started/) - Quick start guide
+- [Environments](https://www.pulumi.com/docs/esc/environments/) - Define and manage environments
+- [Integrations](https://www.pulumi.com/docs/esc/integrations/) - Integrate with external secret stores
+- [CLI Reference](https://www.pulumi.com/docs/esc/cli/) - ESC command reference
+
+## Pulumi Cloud Platform
+
+- [Overview](https://www.pulumi.com/docs/pulumi-cloud/) - Managed platform capabilities
+- [Deployments](https://www.pulumi.com/docs/pulumi-cloud/deployments/) - Automated deployments
+- [Pulumi Copilot](https://www.pulumi.com/docs/pulumi-cloud/copilot/) - AI-powered infrastructure
+- [Access Management](https://www.pulumi.com/docs/pulumi-cloud/access-management/) - Teams, roles, RBAC
+- [Audit Logs](https://www.pulumi.com/docs/pulumi-cloud/admin/audit-logs/) - Compliance and security
+- [Pulumi Insights](https://www.pulumi.com/docs/insights/) - Infrastructure intelligence and search
+
+## Registry (Providers & Packages)
+
+- [AWS Provider](https://www.pulumi.com/registry/packages/aws/) - Complete AWS resource coverage
+- [Azure Native Provider](https://www.pulumi.com/registry/packages/azure-native/) - Full Azure ARM API
+- [Google Cloud Provider](https://www.pulumi.com/registry/packages/gcp/) - Complete GCP coverage
+- [Kubernetes Provider](https://www.pulumi.com/registry/packages/kubernetes/) - Native Kubernetes support
+- [All Providers](https://www.pulumi.com/registry/) - 150+ cloud and SaaS providers
+
+## API Reference
+
+- [Node.js/TypeScript](https://www.pulumi.com/docs/reference/pkg/nodejs/) - Complete TypeScript/JavaScript API
+- [Python](https://www.pulumi.com/docs/reference/pkg/python/) - Complete Python API
+- [Go](https://www.pulumi.com/docs/reference/pkg/go/) - Complete Go API
+- [.NET/C#](https://www.pulumi.com/docs/reference/pkg/dotnet/) - Complete .NET API
+- [Java](https://www.pulumi.com/docs/reference/pkg/java/) - Complete Java API
+
+## Resources
+
+- [Tutorials](https://www.pulumi.com/tutorials/) - Hands-on learning paths
+- [Examples](https://github.com/pulumi/examples) - Production-ready templates
+- [Templates](https://www.pulumi.com/templates/) - Starter templates for common architectures
+- [Blog](https://www.pulumi.com/blog/) - Latest features, best practices, and announcements
+- [Case Studies](https://www.pulumi.com/case-studies/) - How BMW, Mercedes-Benz, Snowflake use Pulumi