Serve robots.txt and a sitemap that lists every tool

[vibhavkatre]

Closes #173. Follows #174.

Both files exist on the deployed site today. Both are useless.

/robots.txt returns 200 with Content-Length: 0. Frappe reads it from a Website Settings field, which is empty. That field is site data, so a value typed into the desk would have to be typed again on the next deployment, and nothing in the repository would record what it said.

/sitemap.xml returns Frappe's stock pages and not one tool:

<url><loc>http://toolbox.localhost:8100/about</loc></url>
<url><loc>http://toolbox.localhost:8100/contact</loc></url>

Frappe builds it from its own page list and from DocTypes with a web view. Toolbox tools are website route rules, so none of them qualify. All fifteen were invisible to anything reading the sitemap.

After

$ curl -s https://frappe.tools/sitemap.xml | grep -c '<loc>'
16                                     # the root and 15 tools; /settings excluded

$ curl -s https://frappe.tools/robots.txt
User-agent: *
Disallow: /api/
Disallow: /app/

Sitemap: https://frappe.tools/sitemap.xml

Both are generated from toolbox/routes.py. The override holds because TemplatePage searches installed apps in reverse order and Toolbox is installed after Frappe — an e2e test asserts the served result rather than the mechanism, so the day that stops being true, it fails.

Two decisions that look like omissions

No Allow: /. Anything not disallowed is already allowed, and the line is not harmless. Google resolves a conflict by longest matching path, but several parsers take the first match — including Python's own robotparser. Under those, a leading blanket allow wins every comparison and voids every Disallow beneath it. This was measured, not assumed: with the line present, /api/method/toolbox.weather.forecast parsed as crawlable. test_it_declares_no_blanket_allow is the guard against putting it back.

/settings is not disallowed. It carries noindex from #174 and it is linked from the navigation. A crawler blocked from fetching a page never reads the noindex on it, so the URL can still surface in results with no description. Blocking and hiding are different instructions, and only one of them belongs on a page we link to.

No lastmod. Frappe's sitemap stamps every entry with today's date, every day. A date that is always wrong is worse than none: a crawler that stops trusting it discounts the real ones later. The sitemap protocol makes the field optional.

Consistency with #174

The sitemap lists the routes seo.py marks indexable, which is the same flag that writes the robots meta tag. A page cannot be told to stay out of the index and be advertised for crawling in the same breath, and test_the_count_matches_what_seo_declares_indexable holds the two together.

Verified

Live on toolbox.localhost:8100: /sitemap.xml is valid XML under ElementTree with 16 URLs and correct application/xml; /robots.txt is text/plain, parses under RobotFileParser, declares the sitemap, and returns the intended verdict for each path — tools crawlable, /settings crawlable, /api/ and /app/ blocked.

backend 169 (was 158) · vitest 610 · e2e 195 passed / 6 skipped (was 189/6) · build clean.

ruff and pre-commit are not installed in this bench, so the Python was checked against the project's conventions by hand rather than by running the hooks.