GTM-4211: correct llms.txt facts and cover the pages it was missing - #1026
Merged
Conversation
GTM-4211 The llms.txt header carried a benchmark line that did not match our own benchmarks page on any value. It read "Envio 1 min vs The Graph 143 min (Uniswap V2 Factory, Sentio, May 2025)". The benchmarks page has that case at Envio 8s against The Graph 19m, from April 2025, and 143 appears to be the "142x slower" multiplier read as minutes. envio.dev/llms.txt already states it correctly, so the two files disagreed. Changes here: - Header quotes the benchmark exactly as the benchmarks page reports it. - Chain count is interpolated from network-count.json, which is regenerated from the live chain API on every build, instead of a hardcoded "70+" that had drifted well below the real figure. The count is EVM-only by construction, which is why Fuel stays named separately in the sentence. - Showcase entries and standalone pages are now collected. 19 showcase pages and /videos are live and in the sitemap but appeared in no llms file. Showcase reads the same _data.js that renders the pages and standalone pages are globbed from src/pages, so neither needs a second list to keep in step. Both link to the rendered URL, as they have no .md twin. - Relative links in the per-page .md copies are resolved to absolute URLs. The copies are served from the flattened slug URL, not their source directory, so a link like ../../static/img/sync.gif or ../Advanced/hypersync.md resolved to a 404 for anything consuming them. Resolution tries URL space first, matching how the rendered page behaves, then the source tree, and warns on anything it cannot place. - Five pages gained a frontmatter description, replacing the generated "X section of the docs." filler and one empty description. Verified with a full build: builds clean, zero unresolved relative links, zero filler descriptions, and all 19 new showcase URLs plus /videos return 200 live.
GTM-4211 llms-full.txt and llms-full-blog.txt both name llms.txt in their own headers, but llms.txt named neither, so an agent entering through the index had no way to find them. envio.dev/llms.txt already links both. Adds them to Optional.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📝 WalkthroughWalkthroughThe PR adds descriptive metadata to selected documentation pages and extends LLM output generation with dynamic HyperSync coverage, standalone pages, showcase entries, resource links, and relative-link rewriting for Markdown copies. ChangesLLM documentation generation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant DocusaurusConfig
participant PluginGenerateLLMs
participant NetworkData
participant PageSources
participant LLMOutput
DocusaurusConfig->>NetworkData: read generated network data
DocusaurusConfig->>PluginGenerateLLMs: configure LLM sources and dynamic header
PluginGenerateLLMs->>PageSources: collect standalone pages and showcase entries
PageSources-->>PluginGenerateLLMs: return rendered URLs and metadata
PluginGenerateLLMs->>LLMOutput: generate index and Markdown copies
PluginGenerateLLMs->>LLMOutput: rewrite relative links and add resource links
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/plugin-generate-llms.js`:
- Around line 696-705: Update the page directive generation in injectHtmlTags so
it only advertises an appended .md URL for documentation and blog pages,
matching the writeMarkdownCopies filter that excludes standalone pages and
showcase entries; preserve the existing directive for pages that actually
receive Markdown copies.
- Around line 261-288: Update the pageFiles glob options to also ignore Markdown
files under underscore-prefixed directories by adding the "**/_*/**" exclusion.
In the route computation for each collected page, special-case a root-level
index.md or index.mdx so its route is the site root rather than "/index", while
preserving existing nested index and extension handling.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f1e1be96-4149-42c1-9972-cec1dabccb5f
📒 Files selected for processing (7)
docs/HyperIndex/Advanced/config-schema-reference.mddocs/HyperIndex/supported-networks/index.mddocs/HyperIndexV2/Advanced/config-schema-reference.mddocs/HyperIndexV2/Hosted_Service/hosted-service-features.mddocs/HyperIndexV2/Hosted_Service/hosted-service-monitoring.mddocusaurus.config.jsplugins/plugin-generate-llms.js
Comment on lines
+261
to
+288
| const pageFiles = glob.sync("**/*.{md,mdx}", { | ||
| cwd: pagesAbsPath, | ||
| // Partials and data files are prefixed with _ by | ||
| // Docusaurus convention and are not routable. | ||
| ignore: ["**/_*.{md,mdx}"], | ||
| }); | ||
|
|
||
| for (const file of pageFiles) { | ||
| const fullPath = path.join(pagesAbsPath, file); | ||
| const parsed = matter( | ||
| fs.readFileSync(fullPath, "utf-8") | ||
| ); | ||
| const title = parsed.data.title; | ||
| if (!title) continue; | ||
|
|
||
| const route = toPosix(file).replace( | ||
| /(\/index)?\.(md|mdx)$/, | ||
| "" | ||
| ); | ||
|
|
||
| collectedDocs.push({ | ||
| filePath: fullPath, | ||
| relativePath: toPosix( | ||
| path.relative(context.siteDir, fullPath) | ||
| ), | ||
| title, | ||
| description: parsed.data.description || "", | ||
| pageUrl: `${url.replace(/\/$/, "")}/${route}`, |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
fd -t f -e md -e mdx . src/pages | sort
fd -t f -e md -e mdx . src/pages |
rg '(^|/)_.*(/|$)|/index\.mdx?$' || trueRepository: enviodev/docs
Length of output: 172
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- plugin structure and relevant implementation ---'
sed -n '1,380p' plugins/plugin-generate-llms.js
printf '%s\n' '--- Docusaurus configuration and package versions ---'
fd -t f 'package.json|docusaurus.config.*|sidebars.*' . -x sh -c 'echo "--- $1"; sed -n "1,220p" "$1"' sh {}Repository: enviodev/docs
Length of output: 43851
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- plugin configuration and page sources ---'
rg -n -C 5 'plugin-generate-llms|generate-llms|pages:|showcase:|src/pages|pagesDir|pagesAbsPath' \
--glob '!plugins/plugin-generate-llms.js' \
--glob '!package-lock.json' \
--glob '!yarn.lock' .
printf '%s\n' '--- all page files and underscore paths ---'
fd -t f -e md -e mdx . | sort | rg '(^|/)src/pages(/|$)|(^|/)_[^/]*(/|$)' || true
printf '%s\n' '--- route transformation for representative page paths ---'
node - <<'JS'
const paths = [
"index.md",
"index.mdx",
"videos.mdx",
"guide/index.md",
"_partial.md",
"_data/entry.md",
"guide/_partial.md",
"_private/index.md",
];
for (const file of paths) {
const current = file.replace(/(\\/index)?\\.(md|mdx)$/, "");
const proposedPath = file.replace(/\\.(md|mdx)$/, "");
const proposed =
proposedPath === "index" ? "" : proposedPath.replace(/\\/index$/, "");
console.log(JSON.stringify({file, current, proposed}));
}
JSRepository: enviodev/docs
Length of output: 8461
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- plugin configuration and page sources ---'
rg -n -C 5 'plugin-generate-llms|generate-llms|pages:|showcase:|src/pages|pagesDir|pagesAbsPath' \
--glob '!plugins/plugin-generate-llms.js' \
--glob '!package-lock.json' \
--glob '!yarn.lock' .
printf '%s\n' '--- all page files and underscore paths ---'
fd -t f -e md -e mdx . | sort | rg '(^|/)src/pages(/|$)|(^|/)_[^/]*(/|$)' || true
printf '%s\n' '--- route transformation for representative page paths ---'
node - <<'JS'
const paths = [
"index.md",
"index.mdx",
"videos.mdx",
"guide/index.md",
"_partial.md",
"_data/entry.md",
"guide/_partial.md",
"_private/index.md",
];
for (const file of paths) {
const current = file.replace(/(\/index)?\.(md|mdx)$/, "");
const proposedPath = file.replace(/\.(md|mdx)$/, "");
const proposed =
proposedPath === "index" ? "" : proposedPath.replace(/\/index$/, "");
console.log(JSON.stringify({file, current, proposed}));
}
JSRepository: enviodev/docs
Length of output: 8319
Match Docusaurus page routing and exclusions.
The collector is enabled for src/pages. It must also ignore Markdown files inside underscore-prefixed directories, and it must map index.md(x) to the site root instead of /index. Add "**/_*/**" to ignore and special-case the root index route as shown in the proposed fix.
🧰 Tools
🪛 ast-grep (0.45.1)
[warning] 270-270: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(fullPath, "utf-8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/plugin-generate-llms.js` around lines 261 - 288, Update the pageFiles
glob options to also ignore Markdown files under underscore-prefixed directories
by adding the "**/_*/**" exclusion. In the route computation for each collected
page, special-case a root-level index.md or index.mdx so its route is the site
root rather than "/index", while preserving existing nested index and extension
handling.
Comment on lines
+696
to
+705
| // Pages and showcase entries have no markdown source to | ||
| // copy, so they are excluded here and from llms-full. | ||
| writeMarkdownCopies( | ||
| collectedDocs.filter((d) => d.hasMarkdown !== false) | ||
| ); | ||
|
|
||
| const fullDocsPool = collectedDocs.filter( | ||
| (d) => !excludeFromFullPluginIds.has(d.pluginId) | ||
| (d) => | ||
| !excludeFromFullPluginIds.has(d.pluginId) && | ||
| d.hasMarkdown !== false |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Keep the page directive accurate.
These lines exclude standalone pages and showcase entries from .md output. injectHtmlTags still tells every rendered page that appending .md provides a Markdown version. Agents that follow this instruction from a showcase or standalone page receive a URL that this plugin does not generate.
Change the directive to scope Markdown copies to documentation and blog pages. Alternatively, generate copies for these entries.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/plugin-generate-llms.js` around lines 696 - 705, Update the page
directive generation in injectHtmlTags so it only advertises an appended .md URL
for documentation and blog pages, matching the writeMarkdownCopies filter that
excludes standalone pages and showcase entries; preserve the existing directive
for pages that actually receive Markdown copies.
Jordy-Baby
requested review from
keenbeen32
and removed request for
nikbhintade
August 12, 2026 08:48
keenbeen32
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five fixes to the generated
llms.txt, the file AI assistants and answer engines read to understand Envio. The generator itself was working correctly. Everything here is content that was wrong, missing, or drifting.1. The benchmark stat did not match our own benchmarks page
The header read:
Our benchmarks page reports that case as Envio 8s against The Graph 19m, from April 2025. The 143 appears to be the "142x slower" multiplier read as minutes.
envio.dev/llms.txtalready stated it correctly, so the two files disagreed. This is the opening paragraph, so it is the line most likely to be quoted back at us.Now quoted exactly as the benchmarks page reports it.
2. Chain count was hardcoded and had drifted
Said
70+, while the real figure was materially higher. Now interpolated fromnetwork-count.json, which is regenerated from the live chain API on every build. Confirmed working end to end: prebuild moved it 81 to 80 on its own during verification.The count is EVM-only by construction, which is why Fuel stays named separately in the sentence.
3. Twenty live pages appeared in no llms file
19 showcase pages and
/videosare live and in the sitemap, but were in none of the three files.Both now collect from the same sources that render them, so there is no second list to keep in step. Showcase reads
_data.js, standalone pages are globbed fromsrc/pages. Neither has a.mdtwin, so both link to the rendered URL.4. Relative links in the .md copies resolved to 404s
The copies are served from a flattened slug URL rather than their source directory, so source-relative links broke:
../../static/img/sync.gifresolved to a 404, correct asset is/img/sync.gif../Advanced/hypersync.mdresolved to a 404, correct page is/docs/HyperIndex/hypersync.mdNow resolved to absolute URLs. Resolution tries URL space first, matching how the rendered page behaves, then falls back to the source tree, and warns on anything it cannot place.
5. Smaller items
"X section of the docs."filler and one empty description.llms.txtnow points atllms-full.txtandllms-full-blog.txt. Both referenced it, but not the other way round, so an agent entering through the index could not discover them.Verification
Full
yarn build, exit 0.llms-full.txtandllms-full-blog.txtstill match the index exactly, 96 docs pages and 79 posts