fix: serve llms-full.txt as llms-full.md - #1
Merged
Conversation
llms-full content is Markdown despite the .txt name (the llms.txt convention). A Markdown-only downstream indexer — doma walks *.md — silently skips a .txt and indexes nothing, so doyo's output is unusable by it. Rename at the write boundary (served_name) so the served file and index.md agree; the bare llms.txt manifest is a link list, not prose, so it keeps .txt. Content bytes are unchanged. Adds a golden test.
Owner
Author
|
If needed we can make doyo a "pure" getter and doma a format agnostic indexer later but for now this works. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When a source exposes an
llms-full.txt(URL rung 1 or repo rung 1), doyo now writes it to disk asllms-full.mdinstead ofllms-full.txt. Content bytes are unchanged — only the extension.Why
llms-full.txtis Markdown despite the.txtname (that's the llms.txt convention — doyo's own README calls rung 1 "already Markdown"). A Markdown-only downstream indexer skips it:*.mdonly. On a doyo-pulledspacetimedb.com/folder it indexed 1 chunk (theindex.mdstub) and ignored the 1.77 MB of real docs. After the rename it indexes 1,676 chunks and searches correctly.doyo's contract is "a local folder of clean Markdown" — a
.txtin that folder was the one thing not labeled as what it is.How
llms_served_namemapsllms-full.txt→llms-full.md; everything else is untouched.served_namecomposes it with the existingreserve_indexcollision guard, applied at both the write site and theindex.mdbuilder — so disk andindex.mdalways agree.output.odin), so it covers both URL mode and repo mode with no change tourl.odin/select.odin, following the existingreserve_indexprecedent.llms.txtmanifest (a link list, not prose) deliberately keeps.txtand stays out of a prose index.Testing
output_test.odinpins the rule (llms-full.txt→.md,llms.txtstays, ordinary docs untouched,index.md→index_.mdstill composes).odin test src→ 33/33 pass.https://spacetimedb.com/docs→ wrotellms-full.md(byte-identical 1,769,303 bytes);doma index→ 1,676 chunks;doma searchreturns ranked hits.