Skip to content

feat(llms): serve /llms.txt + /llms-full.txt from the registry#34

Merged
mhenrixon merged 1 commit into
mainfrom
issue-18-llms-txt
Jul 3, 2026
Merged

feat(llms): serve /llms.txt + /llms-full.txt from the registry#34
mhenrixon merged 1 commit into
mainfrom
issue-18-llms-txt

Conversation

@mhenrixon

@mhenrixon mhenrixon commented Jul 3, 2026

Copy link
Copy Markdown
Owner

Closes #18.

Every docs-kit site now serves the two llmstxt.org artifacts, built from the registry with zero authoring — making the docs AI-consumable without HTML scraping.

What you get

curl https://your-docs.example/llms.txt        # the index
curl https://your-docs.example/llms-full.txt    # every page, concatenated

/llms.txt — H1 brand, an optional > tagline summary, one ## section per nav group, and a tight - [Title](…/page.md) bullet list linking each authored page's Markdown twin. /llms-full.txt — every page's Markdown (the same twin as .md) joined by ---. Both are text/plain; charset=utf-8 and HTTP-cached; unwritten pages are excluded, so no dead links.

Live dogfood output (docs/):

# docs-kit

> Shared Phlex/daisyUI chrome for documentation sites — one shell, sidebar, code kit, and page kit across every docs site.

## Getting started
- [Overview](http://localhost/docs/overview.md)
- [Installation](http://localhost/docs/installation.md)
- [Configuration](http://localhost/docs/configuration.md)
- [Authoring pages](http://localhost/docs/authoring.md)
- [Styling & CSS](http://localhost/docs/styling.md)

## Reference
- [Components](http://localhost/docs/components.md)
- [Code languages](http://localhost/docs/languages.md)
- [On this page](http://localhost/docs/on-this-page.md)
- [Deploy](http://localhost/docs/deploy.md)

Design

Per the issue's Decision: controller in the gem, routes in the host app.

  • DocsKit::LlmsText (lib/docs_kit/llms_text.rb) — a pure, Rails-free builder (.index / .pages / .full). All text shaping is unit-tested without booting Rails.
  • DocsKit::LlmsController (app/controllers/docs_kit/llms_controller.rb) — thin; ships in the gem, routes live in the app so a site keeps full control over path/auth/omission (the engine stays glue-only). #full threads the Rails view context so MarkdownExport can render each page's twin, then hands [title, md] pairs to LlmsText.full.
  • c.tagline config knob (default nil → blockquote line omitted).
  • Install generator scaffolds both routes idempotently; docs-kit.rb.erb documents the tagline knob.
  • Caching: stale?/etag on [DocsKit::VERSION, body].

Two things worth a reviewer's eye

  1. Loader wiring. Rails' own autoloader owns DocsKit::LlmsController from app/controllers/docs_kit/ (docs_kit → DocsKit under the default inflector). The engine already opts app/ out of the gem's zeitwerk loader, and the controller's ActionController::Base superclass is Rails-only — so pushing it from the gem loader both conflicts with Rails' loader and can't load standalone. Letting Rails resolve it is the clean path; verified live (the constant loads and both endpoints serve).
  2. The #config shadow (root cause of a mid-work 500). RequestForgeryProtection does delegate :allow_forgery_protection, to: :config. A first cut defined private def config = DocsKit.configuration on the controller, which shadowed ActionController::Base#config — so csrf_meta_tags (called in DocsUI::Shell's <head> when #full renders a page) resolved allow_forgery_protection against a DocsKit::Configuration and raised NameError. Fixed by renaming the reader to #docs_config; a regression guard in the controller spec asserts no def config on the controller.

Test plan

Spec Covers
spec/docs_kit/llms_text_spec.rb H1/blockquote/H2 groups/tight bullet lists, .md suffixes, absolute + relative URLs, tagline omission, multi-registry order, empty registry, authored-only .pages, .full concatenation. 100%
spec/docs_kit/llms_controller_spec.rb Source wiring: ships at the Rails autoload path, subclasses AC::Base, text/plain, builds via LlmsText, etag, and the #config-shadowing regression guard
spec/docs_kit/configuration_spec.rb c.tagline default + override (Configuration 100%)
spec/generators/install_generator_spec.rb Both routes added + idempotent re-run (no duplicates)

Verification

  • bundle exec rubocop — 78 files, no offenses
  • bundle exec rspec — 321 examples, 0 failures; Configuration / Registry / LlmsText 100%
  • Dogfood: curl /llms.txt (index, 9 working .md links) + curl /llms-full.txt (9 pages, 8 --- rules, 37 KB)
  • Generator re-run does not duplicate routes

Out of scope (per the issue)

MCP (next phase), sitemap/SEO artifacts, per-page exclusion flags.

@mhenrixon
feat(llms): serve /llms.txt + /llms-full.txt from the registry
61a3acd 
Every docs-kit site now serves the two llmstxt.org artifacts, built from the
registry with zero authoring, making the docs AI-consumable without HTML
scraping.

## Summary

- DocsKit::LlmsText (lib/docs_kit/llms_text.rb) — a pure, Rails-free builder:
  .index → the llms.txt index (H1 brand, `> tagline` blockquote, one `##`
  section per nav group, tight `- [Title](abs .md url)` bullet list per authored
  page); .pages → authored pages across every registry; .full → each page's
  Markdown joined by `---`. Unwritten pages (no view_class) are excluded, so no
  dead links.
- DocsKit::LlmsController (app/controllers/docs_kit/llms_controller.rb) — thin,
  ships in the gem; routes live in the host app (the engine stays glue-only).
  Both actions render text/plain; charset=utf-8 and HTTP-cache via
  stale?/etag on [VERSION, body]. #full threads the Rails view context so
  MarkdownExport can render each page's twin.
- c.tagline config knob (default nil → blockquote omitted).
- Install generator scaffolds both routes idempotently; docs-kit.rb.erb
  documents the tagline knob.
- Dogfood: docs/ routes + a tagline; docs route gains (.:format) so its .md
  links resolve.
- README "AI-readable docs (llms.txt)" section.

Rails' own autoloader owns DocsKit::LlmsController from
app/controllers/docs_kit/ (the engine already opts app/ out of the gem's
zeitwerk loader; the controller's ActionController::Base superclass is
Rails-only). The gem controller deliberately does NOT define #config —
RequestForgeryProtection delegates allow_forgery_protection to it, so shadowing
it with DocsKit.configuration broke csrf_meta_tags when #full renders a page's
<head>; the DocsKit config reader is #docs_config.

## Test Coverage

- spec/docs_kit/llms_text_spec.rb — H1/blockquote/H2 groups/tight bullet lists,
  .md suffixes, absolute + relative URLs, tagline omission, multi-registry
  order, empty-registry, authored-only .pages, .full concatenation. 100%.
- spec/docs_kit/llms_controller_spec.rb — source wiring: ships at the Rails
  autoload path, subclasses AC::Base, text/plain, builds via LlmsText, etag,
  and the #config-shadowing regression guard.
- spec/docs_kit/configuration_spec.rb — c.tagline default + override (Config 100%).
- spec/generators/install_generator_spec.rb — both routes added + idempotent
  re-run (no duplicates).

## Verification

- [x] bundle exec rubocop — 78 files, no offenses
- [x] bundle exec rspec — 321 examples, 0 failures; Configuration/Registry/LlmsText 100%
- [x] Dogfood: curl /llms.txt (index, 9 working .md links) + /llms-full.txt (9 pages, 8 rules)
- [x] Generator re-run does not duplicate routes
@mhenrixon mhenrixon self-assigned this Jul 3, 2026
@mhenrixon mhenrixon added the enhancement New feature or request label Jul 3, 2026
@mhenrixon mhenrixon merged commit cd029ca into main Jul 3, 2026
3 checks passed
@mhenrixon mhenrixon deleted the issue-18-llms-txt branch July 4, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@mhenrixon mhenrixon

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(llms): serve /llms.txt + /llms-full.txt from the registry

1 participant

@mhenrixon