Publish a real robots.txt, and welcome crawlers explicitly (#27)

* feat: Publish a real robots.txt, and welcome crawlers explicitly

layouts/robots.txt has been in this repo for ages without ever being
rendered: Hugo only emits it when enableRobotsTXT is set, and it was not.
So the origin published nothing, and Cloudflare filled the gap with its own
file — the Content Signals legal preamble and no directives at all. No
User-agent line, no Allow, no actual signal. An agent-readiness scan called
it invalid, and it was.

Now Hugo emits the file, and it says what we actually mean: every crawler
allowed, search=yes, ai-input=yes, ai-train=yes. The named AI crawlers
repeat the same permissive group, because a crawler that matches its own
group stops reading the wildcard one.

Alongside that: llms.txt as a custom output format that ranges over the
site's own pages and posts, so it follows the content instead of being
maintained by hand, an ARD capability manifest, and discovery links in the
head. The sitemap Hugo already generated was the one thing the scan did not
complain about.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: Record the open agent-readiness work

The remaining items all need response headers or DNS, which GitHub Pages
does not give us. Worth writing down: this repo already has an nginx config
and a Dockerfile, but only for PR previews — the preview stack can set
headers and production cannot.

Also records why the API-shaped checks are deliberately absent, and one
pre-existing quirk found while working here: the non-production guard in
layouts/robots.txt cannot fire, because params.env is hardcoded to
production, so previews serve the permissive rules too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: sunib

AGENT-READINESS.md

@@ -0,0 +1,171 @@
+# Agent readiness: what is done, what is open
+
+This site should be easy for crawlers, language models, and agents to read. It is a
+public personal site; the whole point is to be found and quoted. So the default answer
+to "may I read this?" is yes, and it should be stated out loud rather than left to be
+inferred.
+
+This file records what is in place, what is still open, and why the open items are
+blocked on hosting decisions rather than on writing code.
+
+## Background: two problems, stacked
+
+An agent-readiness scan reported `robots.txt exists but appears invalid (no User-agent
+directive)`. That turned out to be two separate faults sitting on top of each other.
+
+**1. Hugo was never publishing a robots.txt.** `layouts/robots.txt` has been in this
+repo for a long time, but Hugo only renders it when `enableRobotsTXT` is set, and it
+was not. The template was dead code. The `gh-pages` branch contained no `robots.txt`
+at all — confirmed with `git ls-tree origin/gh-pages`.
+
+**2. Cloudflare answered in our place.** With nothing at the origin, the edge served
+its own file: the Content Signals legal preamble and nothing else. Twenty-odd lines of
+comments explaining what content signals mean, followed by no `User-agent` line, no
+`Allow`, no `Disallow`, and no actual signal. Not a valid robots.txt, and not something
+anyone here wrote. Meanwhile Cloudflare's managed bot preferences were blocking AI
+crawlers by a default nobody chose.
+
+Both are now fixed: `enableRobotsTXT: true` in `config.yml`, and Bot Preference Sync
+disabled at the edge so the origin's file is the one that gets served.
+
+The lesson worth keeping: an empty origin is not a neutral origin. Something else will
+fill the gap and speak on your behalf, and what it says may not be what you would have
+said — or even be valid.
+
+## Done (in this repo)
+
+| What | Where |
+| --- | --- |
+| `enableRobotsTXT`, so Hugo actually emits the file | `config.yml` |
+| `robots.txt` — everyone allowed, Content Signals `search=yes, ai-input=yes, ai-train=yes` | `layouts/robots.txt` |
+| Named AI crawler groups repeating the same permissive rules | `layouts/robots.txt` |
+| `llms.txt`, generated from the site's own pages and posts | `config.yml` (`outputFormats.LLMS`) + `layouts/_default/index.llms.txt` |
+| ARD capability manifest | `static/.well-known/ai-catalog.json` |
+| `<link rel="sitemap">` and `<link rel="alternate">` to `llms.txt` | `layouts/partials/extend_head.html` |
+
+Two things worth knowing about how this is wired:
+
+- A crawler that matches one of the named groups in `robots.txt` ignores the wildcard
+  group entirely (RFC 9309), which is why the permissive rules are repeated rather
+  than inherited.
+- `llms.txt` ranges over `.Site.RegularPages`, so new posts and pages appear in it
+  automatically. Nothing to remember on publish. The sitemap was already generated by
+  Hugo and was the one thing the scan did not complain about.
+
+## Open: blocked on the hosting setup
+
+These need control over HTTP response headers or DNS. Production publishes to
+`gh-pages` (`.github/workflows/publish-stuff.yml`), and GitHub Pages serves static
+files without letting us set headers.
+
+Note the irony: this repo *already contains* an nginx config and a Dockerfile
+(`nginx/default.conf`), but only for per-PR preview environments. The preview stack can
+set response headers; production cannot.
+
+### 1. `Link` response headers (RFC 8288)
+
+Agents look for discovery hints in headers, not just in HTML. The homepage should send
+something like:
+
+```
+Link: </llms.txt>; rel="alternate"; type="text/plain",
+      </sitemap.xml>; rel="sitemap",
+      </.well-known/ai-catalog.json>; rel="service-desc"
+```
+
+The `<link>` tags in `extend_head.html` are the in-page equivalent and cover agents
+that parse HTML, but header-based checks will not see them.
+
+*Blocked by:* no header control on GitHub Pages. Three lines of `add_header` in
+`nginx/default.conf` the moment production runs that container.
+
+### 2. Markdown for Agents
+
+Requests sending `Accept: text/markdown` should get markdown back, with HTML staying
+the default for browsers. Verified as not happening today.
+
+This site is a particularly silly place to be missing it, because everything in
+`content/` *is already markdown*. We render it to HTML at build time and then have no
+way to hand back the source. Hugo can emit a markdown output format alongside HTML
+without much trouble; what is missing is the content negotiation, which is a server
+concern.
+
+*Blocked by:* a Cloudflare feature on this account, where an origin we run could do the
+negotiation directly against markdown we already have.
+
+### 3. DNS for AI Discovery (DNS-AID)
+
+SVCB/HTTPS records under `_index._agents.koudijs.dev` pointing at the discovery
+documents, with DNSSEC on the zone.
+
+*Blocked by:* nothing expensive — this is DNS and is achievable wherever the zone
+lives. Parked because the spec is still an IETF draft
+(`draft-mozleywilliams-dnsop-dnsaid`) and it is the lowest-value item here.
+
+### 4. Check what the edge blocks
+
+Independently of `robots.txt`: if any managed AI-bot blocking is still active in
+Cloudflare's AI Crawl Control, it overrides our permissive rules at the edge and a
+crawler never reaches the origin to read the welcome. Re-verify after any change to the
+proxy setup.
+
+## Observation: previews are crawlable
+
+`layouts/robots.txt` has a guard meant to serve `Disallow: /` for non-production
+builds. It cannot currently fire: the condition is
+`hugo.IsProduction | or (eq .Site.Params.env "production")`, and `params.env` is
+hardcoded to `production` in `config.yml`, so the production branch always wins.
+Preview builds at `pr<N>.preview.koudijs.dev` therefore serve the permissive rules.
+
+This predates the changes here and the risk is low, since preview URLs are not linked
+from anywhere. Left alone deliberately, because the obvious fix — setting
+`HUGO_PARAMS_env: development` in `preview-image.yml` — would also switch off
+OpenGraph, analytics, and schema.org output in previews, making them a less faithful
+copy of production. If it is worth fixing, the cleaner route is to set
+`HUGO_ENVIRONMENT: development` for previews and drop the `params.env` half of the
+condition, so `hugo.IsProduction` alone decides.
+
+## Not applicable
+
+Several agent-readiness checks assume a site with an API behind it. This is a personal
+site and blog, so these are deliberately absent rather than missing:
+
+- `/.well-known/api-catalog` (RFC 9727)
+- `/.well-known/openid-configuration`, `/.well-known/oauth-authorization-server`
+- `/.well-known/oauth-protected-resource` (RFC 9728)
+- `auth.md`
+- `/.well-known/mcp/server-card.json`
+- `/.well-known/agent-skills/index.json` — no skills to publish
+- WebMCP tools
+
+Publishing empty versions of these would satisfy a checker while pointing agents at
+nothing, which is worse than not publishing them.
+
+## The direction
+
+The intent is to move the public sites onto our own cluster and stop depending on
+GitHub Pages. Cloudflare may stay as a proxy, or go entirely. Either way, once there is
+an origin we control:
+
+- `Link` headers become a few lines of nginx config — in a file this repo already has.
+- Markdown content negotiation becomes part of the build, using markdown that is
+  already sitting in `content/`.
+- Nothing about discovery is gated behind someone else's pricing tier, or their default
+  opinion about who may read this.
+
+## How to verify
+
+```sh
+curl -sS https://koudijs.dev/robots.txt
+curl -sS https://koudijs.dev/llms.txt
+curl -sS https://koudijs.dev/sitemap.xml
+curl -sS https://koudijs.dev/.well-known/ai-catalog.json | python3 -m json.tool
+curl -sS -D - -o /dev/null https://koudijs.dev/ | grep -i '^link:'
+curl -sS -D - -o /dev/null -H 'Accept: text/markdown' https://koudijs.dev/ | grep -i '^content-type:'
+```
+
+The last two are the open items: today they return nothing and `text/html`.
+
+Locally, `hugo --minify` writes all of the above into `public/`. To check the preview
+branch of the robots template: `hugo -e development --minify` with
+`HUGO_PARAMS_env=development`.

config.yml

@@ -1,6 +1,9 @@
 baseURL: https://koudijs.dev/
 languageCode: en-us
 title: koudijs.dev
+# Renders layouts/robots.txt. Without this Hugo publishes no robots.txt at all,
+# which left the edge free to answer for us.
+enableRobotsTXT: true
 pagination:
   pagerSize: 5
 
@@ -54,11 +57,21 @@ params:
       url: https://simonkoudijs.substack.com/
 
 
+# llms.txt is generated from the site's own pages (see
+# layouts/_default/index.llms.txt), so it cannot drift from what is published.
+outputFormats:
+  LLMS:
+    mediaType: text/plain
+    baseName: llms
+    isPlainText: true
+    notAlternative: true
+
 outputs:
   home:
   - HTML
   - RSS
   - JSON
+  - LLMS
 
 menu:
   main:

layouts/_default/index.llms.txt

@@ -0,0 +1,34 @@
+{{- /* llms.txt — a plain-text summary of this site for language models and agents.
+       Pages and posts are ranged over, so this file follows the content instead of
+       being maintained by hand. Spec: https://llmstxt.org/ */ -}}
+{{- $trim := "\\s+" -}}
+# {{ .Site.Title }}
+
+> {{ .Site.Params.description }}
+
+Simon Koudijs is a consultant, trainer, and speaker based in Amersfoort, Netherlands,
+trading as Koudijs IT (KvK 97788708). The work spans software delivery, engineering
+workflows and CI/CD, platform engineering, Kubernetes, GitOps, and the practical use of
+AI in engineering — with a bias towards what teams can actually apply over what sounds
+impressive. He is also the founder of ConfigButler (https://configbutler.ai) and the
+author of the Reverse GitOps manifesto (https://reversegitops.dev).
+
+Available for consulting, in-house and public training, and conference talks. To make
+contact, book a slot directly: https://calendar.app.google/c2jMcQgREVgbKEj18
+
+## Pages
+{{ range $p := (where (where .Site.RegularPages "Section" "") "Layout" "!=" "search") -}}
+- [{{ $p.Title }}]({{ $p.Permalink }}): {{ $p.Summary | plainify | replaceRE $trim " " | htmlUnescape | truncate 240 }}
+{{ end }}
+## Posts
+{{ range (where .Site.RegularPages "Section" "posts").ByDate.Reverse -}}
+- [{{ .Title }}]({{ .Permalink }}) — {{ .Date.Format "2 January 2006" }}{{ with .Params.tags }}, tagged {{ delimit . ", " }}{{ end }}: {{ .Summary | plainify | replaceRE $trim " " | htmlUnescape | truncate 240 }}
+{{ end }}
+## Notes for crawlers and agents
+
+- Crawling, indexing, quoting, and training on this content are all permitted. See {{ "robots.txt" | absURL }} for the Content Signals.
+- Machine-readable index of this site: {{ "sitemap.xml" | absURL }}
+- Full-text search index used by the site's own search page: {{ "index.json" | absURL }}
+- RSS feed: {{ "index.xml" | absURL }}
+- Capability manifest: {{ "/.well-known/ai-catalog.json" | absURL }}
+- There is no API to call on this domain; the site is a personal site and blog. Talk slides are published as code at https://github.com/reverse-gitops/talks

layouts/partials/extend_head.html

@@ -1,4 +1,10 @@
 {{- /* Head custom content area start */ -}}
 {{- /*     Insert any custom code (web-analytics, resources, etc.) - it will appear in the <head></head> section of every page. */ -}}
-{{- /*     Can be overwritten by partial with the same name in the global layouts. */ -}}
+
+{{- /* Discovery for crawlers and agents. The HTTP Link header would be the
+       stronger signal, but GitHub Pages cannot set response headers — see
+       AGENT-READINESS.md. */ -}}
+<link rel="sitemap" type="application/xml" href="{{ "sitemap.xml" | absURL }}" />
+<link rel="alternate" type="text/plain" href="{{ "llms.txt" | absURL }}" title="llms.txt summary of this site" />
+
 {{- /* Head custom content area end */ -}}

layouts/robots.txt

@@ -1,7 +1,48 @@
+{{- if hugo.IsProduction | or (eq .Site.Params.env "production") -}}
+# koudijs.dev — Simon Koudijs
+# Every crawler is welcome here: read the site, index it, quote it, train on it.
+# Content Signals: https://contentsignals.org/ (RFC 9309 for the rest)
+
+# As the operator of this site, I consent to the uses below.
+# search=yes: building a search index and linking back here.
+# ai-input=yes: using this content as input to an AI answer or agent.
+# ai-train=yes: using this content to train or fine-tune a model.
+
+User-agent: *
+Content-Signal: search=yes, ai-input=yes, ai-train=yes
+Allow: /
+
+# --- AI crawlers and agents, called out explicitly ---
+# A crawler that matches a group below ignores the wildcard group, so each one
+# repeats the same permissive rules.
+
+User-agent: GPTBot
+User-agent: OAI-SearchBot
+User-agent: ChatGPT-User
+User-agent: ClaudeBot
+User-agent: Claude-Web
+User-agent: Claude-User
+User-agent: Claude-SearchBot
+User-agent: Google-Extended
+User-agent: GoogleOther
+User-agent: PerplexityBot
+User-agent: Perplexity-User
+User-agent: Applebot-Extended
+User-agent: meta-externalagent
+User-agent: Amazonbot
+User-agent: Bytespider
+User-agent: CCBot
+User-agent: cohere-ai
+User-agent: Diffbot
+User-agent: DuckAssistBot
+User-agent: MistralAI-User
+Content-Signal: search=yes, ai-input=yes, ai-train=yes
+Allow: /
+
+Sitemap: {{ "sitemap.xml" | absURL }}
+{{- else -}}
+# Non-production build (PR preview). Keep these out of every index — the
+# production site at https://koudijs.dev/ is the one that should be crawled.
 User-agent: *
-{{- if hugo.IsProduction | or (eq .Site.Params.env "production") }}
-Disallow:
-{{- else }}
 Disallow: /
 {{- end }}
-Sitemap: {{ "sitemap.xml" | absURL }}