Skip to content

fix: serve llms-full.txt as llms-full.md - #1

Merged
L34Z merged 1 commit into
mainfrom
fix/llms-full-md-extension
Jul 31, 2026
Merged

fix: serve llms-full.txt as llms-full.md#1
L34Z merged 1 commit into
mainfrom
fix/llms-full-md-extension

Conversation

@L34Z

@L34Z L34Z commented Jul 31, 2026

Copy link
Copy Markdown
Owner

What

When a source exposes an llms-full.txt (URL rung 1 or repo rung 1), doyo now writes it to disk as llms-full.md instead of llms-full.txt. Content bytes are unchanged — only the extension.

Why

llms-full.txt is Markdown despite the .txt name (that's the llms.txt convention — doyo's own README calls rung 1 "already Markdown"). A Markdown-only downstream indexer skips it:

  • doma (companion search tool) walks *.md only. On a doyo-pulled spacetimedb.com/ folder it indexed 1 chunk (the index.md stub) and ignored the 1.77 MB of real docs. After the rename it indexes 1,676 chunks and searches correctly.

doyo's contract is "a local folder of clean Markdown" — a .txt in that folder was the one thing not labeled as what it is.

How

  • New llms_served_name maps llms-full.txtllms-full.md; everything else is untouched.
  • New served_name composes it with the existing reserve_index collision guard, applied at both the write site and the index.md builder — so disk and index.md always agree.
  • Lives at the write boundary (output.odin), so it covers both URL mode and repo mode with no change to url.odin/select.odin, following the existing reserve_index precedent.
  • The bare llms.txt manifest (a link list, not prose) deliberately keeps .txt and stays out of a prose index.

Testing

  • New golden test output_test.odin pins the rule (llms-full.txt.md, llms.txt stays, ordinary docs untouched, index.mdindex_.md still composes).
  • odin test src → 33/33 pass.
  • End-to-end: rebuilt doyo, re-pulled https://spacetimedb.com/docs → wrote llms-full.md (byte-identical 1,769,303 bytes); doma index → 1,676 chunks; doma search returns ranked hits.

llms-full content is Markdown despite the .txt name (the llms.txt
convention). A Markdown-only downstream indexer — doma walks *.md —
silently skips a .txt and indexes nothing, so doyo's output is unusable
by it. Rename at the write boundary (served_name) so the served file and
index.md agree; the bare llms.txt manifest is a link list, not prose, so
it keeps .txt. Content bytes are unchanged. Adds a golden test.
@L34Z

L34Z commented Jul 31, 2026

Copy link
Copy Markdown
Owner Author

If needed we can make doyo a "pure" getter and doma a format agnostic indexer later but for now this works.

@L34Z
L34Z merged commit d06f6d7 into main Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant