SEO and Sitemaps

SEO and Sitemaps

Pages generated per listing

Silo	Exists when
Listing page	always
/for/<client>	for each client that can run the server
/tools, /tools/<tool>, /tools/<tool>/<client>	the server has tools
/prompts/…, /resources/…	the server has prompts / resources
/changelog, /changelog/<kind>	something has changed

The rule throughout: a page exists only where there is something real on it. An empty template repeated across thousands of listings is the doorway pattern search engines penalise.

Sitemaps

sitemap.xml is an index of smaller files, because a single sitemap may hold at most 50,000 URLs and a file over the limit is rejected without warning.

File	Holds
pages.xml	the home page, directory, submit form, book and changelog
servers-N.xml	listing pages
agents-N.xml	listing × client pages
tools-N.xml	tool pages
prompts-N.xml, resources-N.xml	prompt and resource pages, packed to a 40,000-URL budget per file
changelogs-N.xml	changelog pages

Prompts and resources are packed by URL count rather than listing count because they are extremely uneven: one listing has nearly a thousand resources while most have a handful.

Robots

robots.txt allows all content and disallows /live/ and /api/. /live/ is LiveView's transport, whose long-poll fallback carries a fresh token in every URL — an endless supply of new URLs to a crawler that cannot hold a websocket.

Other discovery

New and changed listings are announced through IndexNow, and an Atom feed of new servers is published with WebSub.