[3.x] Support pages with more output file extensions (Add a robots.txt generator)

[emmadesilva]

What was built

Hyde now generates a robots.txt out of the box, wired exactly like the PR 5 sitemap/RSS pages:

• RobotsTxtPage extends InMemoryPage and a container-resolved RobotsTxtGenerator, in a new Hyde\Framework\Features\TextGenerators namespace mirroring XmlGenerators (outside Hyde\Pages, so it's exempt from the page unit-test contract). Default output is User-agent: * / Allow: /, plus a Sitemap: line when Features::hasSitemap().
• Config: hyde.robots.enabled (default on) and hyde.robots.disallow (paths written verbatim as Disallow: rules, replacing the Allow: / line). Added to both config stubs.
• Features::hasRobotsTxt() gates registration in HydeCoreExtension::discoverPages() with the D5 skip check. Notably it needs no site URL (robots directives are relative); the only absolute URL is the Sitemap line, gated by the same hasSitemap() condition that registers the sitemap page — the GlobalMetadataBag no-drift precedent.

Verification

• New coverage: registration/disable, generator output shapes, container rebind, real build command output, build manifest, route:list, sitemap self-exclusion, realtime-compiler serving with text/plain, and the D5-mandated end-to-end override tests (booting callback and extension both suppress the generated page) — the epic flagged these as mandatory for PR 6.
• Because the page registers unconditionally on default config, five existing tests asserting exact collections gained a hyde.robots.enabled => false in their setup, alongside their existing sitemap/RSS switches. One subtlety from the run: a count-assertion failure mid-PageCollectionTest deleted the tracked default Blade pages before restoreDefaultPages() could run, which cascaded into ~10 misleading build-test failures until I restored _pages/ from git.
• All four suites were baseline-compared per the CLAUDE.md procedure (run at HEAD, re-run with packages reverted, diff sorted FAILED lines): identical failure lists on FeatureFramework, UnitFramework, and FeatureHyde (remaining failures are the known environmental ones); Realtime Compiler fully green. HydeStan reports no errors.

Commits (5, atomic)

Implementation → existing-test adaptations → new tests → release notes/UPGRADE.md → epic update. The epic's PR 6 section is marked ✅ Implemented with notes recording the decisions: the TextGenerators namespace (with a pointer that PR 7's llms.txt should land there as LlmsTxtGenerator/LlmsTxtPage), the no-site-URL feature gate, verbatim disallow rules (no leading-slash normalization — it would break wildcards and the valid empty rule), and the decision not to add a build:robots command since, unlike sitemap/RSS, there's no removed task to stay compatible with.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (37d3c96) to head (cd348f9).
⚠️ Report is 1 commits behind head on v3/non-html-pages.

Additional details and impacted files

@@                  Coverage Diff                  @@
##             v3/non-html-pages     #2531   +/-   ##
=====================================================
  Coverage               100.00%   100.00%           
- Complexity                1746      1760   +14     
=====================================================
  Files                      175       177    +2     
  Lines                     4360      4395   +35     
=====================================================
+ Hits                      4360      4395   +35     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.