Adopt llms.txt spec, add llms-full, ctx generation, and CI#198
Adopt llms.txt spec, add llms-full, ctx generation, and CI#198serefyarar merged 6 commits intodevfrom
Conversation
WalkthroughAdds LLM context source files and prebuilt ctx artifacts, a generator script, a pre-commit hook to auto-generate/stage ctx files, Makefile targets, README docs, and a GitHub Actions workflow to validate llms.txt and build/upload ctx artifacts. New public endpoints are introduced via static files under frontend/public. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor Dev
participant Git as Git (pre-commit)
participant Script as generate-ctx.sh
participant CLI as llms_txt2ctx (llms-txt)
participant FS as Filesystem
Dev->>Git: git commit (staged changes)
Git->>Git: Check staged paths for llms.txt / llms-full.txt
alt LLMS files changed
Git->>Script: Run scripts/generate-ctx.sh
Script->>Script: Ensure CLI (system or .venv)
Script->>CLI: Generate llms-ctx.txt
CLI-->>Script: Output temp ctx
Script->>CLI: Generate llms-ctx-full.txt (--optional)
CLI-->>Script: Output temp full ctx
Script->>FS: Move to repo root and copy to frontend/public/
Git->>Git: Stage llms-ctx*.txt and public copies
else No LLMS changes
Git-->>Dev: Proceed
end
Git-->>Dev: Commit completes (never blocked by failures)
sequenceDiagram
autonumber
actor CI as GitHub Actions
participant Repo as Repo
participant Job1 as validate
participant Job2 as build-ctx
participant Script as scripts/generate-ctx.sh
participant Art as Artifacts
CI->>Repo: Trigger (push/PR/workflow_dispatch)
CI->>Job1: Checkout, setup Python, install llms-txt
Job1->>Job1: Validate llms.txt parsing (minimal & --optional)
Job1-->>Job2: On success
CI->>Job2: Checkout, setup Python
Job2->>Script: Run generate-ctx.sh
Script-->>Job2: llms-ctx.txt / llms-ctx-full.txt (+ public copies)
Job2->>Art: Upload llms-ctx-root and llms-ctx-public (14-day retention)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/llms-ctx.yml (1)
1-80: Fix YAML syntax errors in.github/workflows/llms-ctx.ymlThe workflow file is currently invalid (YAML parse error at line 54: expected block end, found mapping start). This is preventing GitHub Actions from loading the workflow. Please correct the indentation of both the
with:block in thebuild-ctxjob and the misplacedschedule:key.• In the build-ctx job’s Set up Python step:
– Thewith:mapping must include thepython-versionkey indented beneath it.
– Currently,schedule:appears at the same level aswith:, which breaks the mapping.
• The schedule trigger belongs under the top-levelon:section, not underjobs.build-ctx.steps.Suggested fix (excerpt):
on: workflow_dispatch: + schedule: + - cron: '0 3 * * *' push: branches: [ main, dev ] paths: - 'llms.txt' - 'llms-full.txt' - 'scripts/generate-ctx.sh' @@ - fifty-something, + fifty-something jobs: build-ctx: needs: validate runs-on: ubuntu-latest steps: - name: Set up Python uses: actions/setup-python@v5 - with: -schedule: - - cron: '0 3 * * *' - python-version: '3.x' + with: + python-version: '3.x'After applying, verify locally:
python -m pip install pyyaml yamllint python - <<'PY' import yaml yaml.safe_load(open(".github/workflows/llms-ctx.yml")) print("YAML parse OK") PY yamllint .github/workflows/llms-ctx.ymlThis will ensure the workflow is syntactically correct before pushing.
🧹 Nitpick comments (40)
frontend/public/llms-full.txt (3)
3-3: Tighten overview and add provenance metadata (date/version).Consider appending a “Last updated: YYYY-MM-DD, commit: ” line to help downstream consumers detect staleness and cache-bust intelligently. Also, “expanded context” can be more direct: “This full context aggregates additional references…”.
7-11: Polish key-concepts phrasing for clarity and consistency.Minor grammar/style tweaks make these easier for LLMs to ground:
- Prefer “and” over “+”; avoid mixed punctuation; parallelize noun phrases.
Apply this diff:
- - Intent: user-authored goal/need text; may be incognito - - Index: privacy-scoped context containing intents and permissions - - Stake: agent-created link between related intents with confidence + reasoning - - Connection: explicit state transitions (REQUEST, ACCEPT, DECLINE, etc.) + - Intent: user-authored goal/need text; may be incognito. + - Index: privacy-scoped context containing intents and permissions. + - Stake: agent-created link between related intents, with confidence and reasoning. + - Connection: explicit state transitions (REQUEST, ACCEPT, DECLINE, …).
34-36: Prefer raw content URLs for file links to improve machine consumption.Switch blob links to raw where possible to avoid HTML.
- - [Drizzle config](https://github.com/indexnetwork/index/blob/dev/protocol/drizzle.config.ts): DB configuration - - [Protocol env example](https://github.com/indexnetwork/index/blob/dev/protocol/env.example): Required environment variables + - [Drizzle config](https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/drizzle.config.ts): DB configuration + - [Protocol env example](https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/env.example): Required environment variablesNote: the directory link for migrations can remain a normal GitHub “tree” URL since there’s no single raw file.
scripts/hooks/pre-commit (2)
6-8: Broaden the log message and match on either llms file.Small UX win: reflect when either llms.txt or llms-full.txt changed.
-if [[ -n "${changed_llms}" ]]; then - echo "[pre-commit] Detected llms.txt changes; generating ctx files..." +if [[ -n "${changed_llms}" ]]; then + echo "[pre-commit] Detected llms*.txt changes; generating ctx files..."
9-13: Invoke the generator via bash to avoid executable-bit surprises.If scripts/generate-ctx.sh lacks +x locally, calling through bash avoids a failure.
- scripts/generate-ctx.sh || { + bash scripts/generate-ctx.sh || {.github/workflows/llms-ctx.yml (3)
30-41: Cache pip to speed CI and reduce flakiness.Install is repeated across jobs; use setup-python cache to avoid re-downloading.
- - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: '3.x' + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.x' + cache: 'pip'Apply in both jobs.
43-60: Let the build job rely on the script to manage llms-txt; no need to pre-install.Your generator script bootstraps a local venv if the CLI is missing. You can drop the extra pip install step in build-ctx for simplicity.
- - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: '3.x' - - - name: Generate ctx files + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.x' + cache: 'pip' + + - name: Generate ctx files run: | bash scripts/generate-ctx.sh
1-80: Optional: add concurrency control to avoid overlapping runs on busy repos.Prevents nightly and push events from racing and overwriting artifacts.
jobs: + concurrency: + group: llms-ctx-${{ github.ref }} + cancel-in-progress: truescripts/generate-ctx.sh (3)
24-38: Make venv creation more robust and allow pinning llms-txt.Some systems lack python3 alias; also consider pinning for reproducibility.
ensure_cli() { if command -v llms_txt2ctx >/dev/null 2>&1; then return 0 fi # Prefer project-local venv to avoid polluting global env VENV_DIR="${ROOT_DIR}/.venv" if [[ ! -d "${VENV_DIR}" ]]; then echo "[llms] Creating venv at ${VENV_DIR}" - python3 -m venv "${VENV_DIR}" + if command -v python3 >/dev/null 2>&1; then + python3 -m venv "${VENV_DIR}" + else + python -m venv "${VENV_DIR}" + fi fi # shellcheck disable=SC1091 source "${VENV_DIR}/bin/activate" - python -m pip install -q --upgrade pip - python -m pip install -q llms-txt + python -m pip install -q --upgrade pip + python -m pip install -q "${LLMSTXT_PKG:-llms-txt}" }Usage: set LLMSTXT_PKG="llms-txt==" in CI if you want deterministic builds.
41-51: Use the resolved source path and optionally increase workers for speed.Minor improvements; safe defaults.
- local src_path="${ROOT_DIR}/${SRC_FILE}" + local src_path="${SRC_ABS}" echo "[llms] Generating minimal ctx -> ${OUT_MIN}" - if ! llms_txt2ctx "${src_path}" > "${OUT_MIN}"; then + if ! llms_txt2ctx --n_workers "${LLMS_N_WORKERS:-4}" "${src_path}" > "${OUT_MIN}"; then echo "[llms] Warning: minimal ctx generation failed; leaving previous file if present" >&2 fi echo "[llms] Generating full ctx -> ${OUT_FULL}" - if ! llms_txt2ctx --optional True "${src_path}" > "${OUT_FULL}"; then + if ! llms_txt2ctx --optional True --n_workers "${LLMS_N_WORKERS:-4}" "${src_path}" > "${OUT_FULL}"; then echo "[llms] Warning: full ctx generation failed; leaving previous file if present" >&2 fi
67-68: List the public endpoints that Next.js will serve (all four).Your PR mentions /llms.txt and /llms-full.txt as well; include them here for consistency.
-echo "[llms] Done. Public endpoints: /llms-ctx.txt, /llms-ctx-full.txt" +echo "[llms] Done. Public endpoints: /llms.txt, /llms-full.txt, /llms-ctx.txt, /llms-ctx-full.txt"frontend/public/llms-ctx-full.txt (2)
10-22: Align API section with the full context links used elsewhere.To match the “full” set described in llms-full, include “Index access utility” and “Synthesis”.
<Section title="API"> <Doc title="API entry (Express app)" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/index.ts" /> <Doc title="Schema" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/lib/schema.ts" /> - <Doc title="LangGraph config" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/agents/langgraph.json" /> + <Doc title="Index access utility" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/lib/index-access.ts" /> + <Doc title="Synthesis" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/lib/synthesis.ts" /> + <Doc title="LangGraph config" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/agents/langgraph.json" /> </Section>
6-8: Clarify the “omits inlined remote content” note.Slightly rephrase to tell readers how to reproduce embedded content locally/CI.
-This prebuilt ctx includes all links (including Optional) but omits inlined remote content due to restricted network/SSL. Regenerate with network access to embed full documents. +This prebuilt ctx includes all links (including Optional) but omits inlined remote content due to network/SSL limits in this build. Regenerate with network access (e.g., CI nightly or locally) to inline full documents.README.md (2)
212-219: Pre-commit hook prerequisitesDocument that the hook must be executable, and note that
core.hooksPathis repo-local config.Apply:
- Optional pre-commit hook to auto-generate on changes to `llms.txt`/`llms-full.txt`: - Enable hooks path: `git config core.hooksPath scripts/hooks` - The hook will run the generator and stage `llms-ctx*.txt` and public copies. + - Make executable: `chmod +x scripts/hooks/pre-commit` + - Note: `core.hooksPath` is a local setting; teammates must run the command once.
222-230: Keep README and Makefile target names in syncMakefile provides
ctx,ctx-full,ctx-public,ctx-clean, but README only documentsctxandctx-clean. List all targets to reduce drift.Apply:
- Makefile targets: - - `make ctx` – Generate ctx files and copy to `frontend/public/` - - `make ctx-clean` – Remove generated ctx files (root and public) + - `make ctx` – Generate minimal+full ctx and copy to `frontend/public/` + - `make ctx-full` – Force full regeneration from `llms-full.txt` (see Makefile) + - `make ctx-public` – Copy generated ctx files into `frontend/public/` + - `make ctx-clean` – Remove generated ctx files (root and public)llms-ctx-full.txt (3)
11-16: Prefer stable links or a branch strategy for reproducibilityLinking to the moving
devbranch can cause silent context drift. For builds meant to be reproducible, consider pinning to commit SHAs or documenting that contexts intentionally trackdev.Example (pin by commit):
-<Doc title="Repository README" href="https://raw.githubusercontent.com/indexnetwork/index/dev/README.md" /> +<Doc title="Repository README" href="https://raw.githubusercontent.com/indexnetwork/index/<commit-sha>/README.md" />If you intentionally follow
dev, add a short note in stating that behavior.
6-8: Tighten wording on the network/SSL noteThe current note reads like a failure mode. Clarify that omission is intentional to keep repo size small and to avoid network dependence.
-This prebuilt ctx includes all links (including Optional) but omits inlined remote content due to restricted network/SSL. Regenerate with network access to embed full documents. +This prebuilt ctx enumerates all links (including Optional) but intentionally omits inlined remote content to keep the repo lean and network-independent. Regenerate with network access to embed full documents.
1-3: Embed provenance metadata for traceabilityAdd a small metadata block with generation timestamp and tool version.
<Project title="Index Network (Full)"> +<Meta generatedAt="2025-08-26T00:00:00Z" tool="llms-txt" toolVersion="x.y.z" source="llms.txt" optional="true" /> <Summary>frontend/public/llms-ctx.txt (2)
6-8: Adjust the note to reflect intention, not an error stateSame wording improvement as the full ctx file to avoid implying a problem.
-This prebuilt ctx lists key sources but omits inlined content due to restricted network/SSL. Regenerate with network access to embed full documents. +This prebuilt ctx lists key sources and intentionally omits inlined content to keep artifacts small and network-independent. Regenerate with network access to embed full documents.
10-22: Ensure these href’d files exist and keep the public copy in syncMirror the verification check from the root ctx in CI, and consider embedding a minimal for provenance.
Add:
<Project title="Index Network"> +<Meta generatedAt="2025-08-26T00:00:00Z" tool="llms-txt" toolVersion="x.y.z" source="llms.txt" optional="false" /> <Summary>Verification (same script as suggested on the full ctx).
Makefile (1)
14-15: Extend clean target to cover /.well-known copiesIf you adopt the /.well-known publishing above, clean them too.
-ctx-clean: - rm -f llms-ctx.txt llms-ctx-full.txt frontend/public/llms-ctx.txt frontend/public/llms-ctx-full.txt +ctx-clean: + rm -f llms-ctx.txt llms-ctx-full.txt \ + frontend/public/llms-ctx.txt frontend/public/llms-ctx-full.txt \ + frontend/public/.well-known/llms-ctx.txt frontend/public/.well-known/llms-ctx-full.txtllms-ctx.txt (2)
10-16: Reproducibility: consider pinning raw links or documenting branch-trackingSame as the full ctx: either pin to commit SHAs or explicitly state that ctx tracks
dev.
1-4: Add provenance metadataEmbed a minimal metadata tag for build traceability.
<Project title="Index Network"> +<Meta generatedAt="2025-08-26T00:00:00Z" tool="llms-txt" toolVersion="x.y.z" source="llms.txt" optional="false" /> <Summary>frontend/public/llms.txt (5)
3-11: Tighten intro copy and privacy phrasing for clarity + consistency.Minor wording tweaks improve scannability and align with the same phrasing used elsewhere in the PR (confidence “scores”, hyphenation on “double‑opt‑in”). Also make the bullets parallel and remove the “explanations + confidence” shorthand.
- > Private, intent-driven discovery via autonomous agents. Broker agents stake on proposed matches; successful, double opt-in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence. + > Private, intent‑driven discovery via autonomous agents. Broker agents stake on proposed matches; successful, double‑opt‑in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence scores. - - Codebase: Next.js frontend, Node/Express API, Drizzle (PostgreSQL), Redis, LangGraph-based agents - - Privacy: Agents never expose raw intent content across users; outputs are explanations + confidence only - - Setup: See protocol and frontend READMEs for local env, DB migration, and dev scripts - - Scope: Public docs and code references below; no private data or non-public endpoints + - Codebase: Next.js frontend, Node/Express API, Drizzle (PostgreSQL), Redis, LangGraph‑based agents + - Privacy: Agents never expose raw intent content across users; outputs are explanations and confidence scores only + - Setup: See protocol and frontend READMEs for local env, DB migration, and dev scripts + - Scope: Public docs and code references below; no private data or non‑public endpoints
14-19: Confirm raw links are intentional here (generator vs. human readability).You’re using raw.githubusercontent.com links, which are great for ctx generation. If this file is also meant for humans, consider adding a note like “Raw links are used intentionally for ctx generators.” If not, all good.
Would you like me to add a short comment line clarifying the use of raw links?
28-31: Use a consistent label for X/Twitter.Small consistency tweak across files.
- - [X (Twitter)](https://x.com/indexnetwork_): Updates and announcements + - [X](https://x.com/indexnetwork_): Updates and announcements
1-32: Optional: add lightweight metadata (helps LLMs and humans).Consider adding a short metadata footer with last updated date and source-of-truth to prevent drift between root/public copies.
- [Blog](https://blog.index.network): Insights and posts - + + --- + Last-Updated: 2025-08-26 + Source-of-Truth: root llms.txt (public copy kept in sync by CI)
22-25: Link Verification Passed – Consider CI IntegrationAll referenced raw URLs returned HTTP 200, confirming there are no broken links at this time:
- protocol/src/index.ts
- protocol/src/lib/schema.ts
- protocol/src/agents/langgraph.json
To safeguard against future moves and stale references, you may want to incorporate this link-checking script into your CI pipeline.
llms-full.txt (7)
3-3: Tweak tone (“deeper” → “more complex”) and remove “skip” phrasing.Keeps the doc neutral and precise.
- > Private, intent-driven discovery network. This expanded context lists additional references for APIs, architecture, agents, and development to help LLMs answer deeper questions. It includes external resources and optional links that can be skipped for shorter contexts. + > Private, intent‑driven discovery network. This expanded context lists additional references for APIs, architecture, agents, and development to help LLMs answer more complex questions. It includes external resources; optional links can be omitted in shorter contexts.
7-11: Standardize key concept bullets and punctuation.Minor grammar for parallelism and clarity.
- - Intent: user-authored goal/need text; may be incognito - - Index: privacy-scoped context containing intents and permissions - - Stake: agent-created link between related intents with confidence + reasoning - - Connection: explicit state transitions (REQUEST, ACCEPT, DECLINE, etc.) + - Intent: User‑authored goal/need text; may be incognito. + - Index: Privacy‑scoped context containing intents and permissions. + - Stake: Agent‑created link between related intents, with confidence and reasoning. + - Connection: Explicit state transitions (REQUEST, ACCEPT, DECLINE, etc.).
23-27: Avoid colloquial “vibecheck”; prefer descriptive wording.Improves professionalism of the full‑context doc.
- - [Synthesis](https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/lib/synthesis.ts): Synthesis & vibecheck helpers + - [Synthesis](https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/lib/synthesis.ts): Synthesis and validation helpers
34-36: Make link style consistent (raw vs. blob).Other sections use raw links (better for ctx tooling). Consider switching these to raw for uniformity.
- - [Migrations (SQL)](https://github.com/indexnetwork/index/tree/dev/protocol/drizzle): Drizzle SQL migrations - - [Drizzle config](https://github.com/indexnetwork/index/blob/dev/protocol/drizzle.config.ts): DB configuration - - [Protocol env example](https://github.com/indexnetwork/index/blob/dev/protocol/env.example): Required environment variables + - [Migrations (SQL)](https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/drizzle/README.md): Drizzle SQL migrations + - [Drizzle config](https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/drizzle.config.ts): DB configuration + - [Protocol env example](https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/env.example): Required environment variablesIf a raw index/README for migrations does not exist, we can instead link to a representative migration file or keep the current blob URLs.
40-43: Consistent label for X/Twitter and add metadata footer.Keeps alignment with the public copy and provides freshness info.
- - [X (Twitter)](https://x.com/indexnetwork_): Updates and announcements + - [X](https://x.com/indexnetwork_): Updates and announcements + + --- + Last-Updated: 2025-08-26 + Source-of-Truth: llms-full.txt (full context; public copy kept in sync by CI)
14-36: Optional: cross-link related assets to aid LLM crawlers.Add pointers to the generated ctx files so models can discover both forms.
## Docs @@ - [Frontend README](https://raw.githubusercontent.com/indexnetwork/index/dev/frontend/README.md): Next.js app structure and commands - [License](https://raw.githubusercontent.com/indexnetwork/index/dev/LICENSE): MIT + +## Generated Context Artifacts + +- [Minimal context (llms-ctx.txt)](/llms-ctx.txt) +- [Full context (llms-ctx-full.txt)](/llms-ctx-full.txt)
12-44: Enhance CI Link-Check to Follow Redirects and Accept 2xx/3xxThe current link-check script treats any non-200 status as a failure (for example, Discord’s invite URL returns 301), causing false positives. To prevent drift without blocking valid redirects:
• Update the curl invocation to follow redirects (
-L) and treat any 2xx or 3xx response as healthy.
• Incorporate this step into your existing CI workflow (e.g.,.github/workflows/ci.yml).Suggested snippet for your link-check step:
- # Curl HEAD all links in llms-full.txt - set -euo pipefail - rg -nP '\(https?://[^)]+\)' llms-full.txt \ - | sed -E 's/.*\((https?:\/\/[^)]+)\).*/\1/' \ - | sort -u \ - | while read -r url; do - code=$(curl -s -o /dev/null -w "%{http_code}" "$url") - printf "%s -> %s\n" "$url" "$code" - [ "$code" = "200" ] || { echo "Broken: $url"; exit 1; } - done + # Curl all links in llms-full.txt, follow redirects, accept 2xx/3xx + set -euo pipefail + rg -nP '\(https?://[^)]+\)' llms-full.txt \ + | sed -E 's/.*\((https?:\/\/[^)]+)\).*/\1/' \ + | sort -u \ + | while read -r url; do + code=$(curl -sSL -o /dev/null -w "%{http_code}" "$url") + echo "$url -> $code" + if [[ "${code:0:1}" != "2" && "${code:0:1}" != "3" ]]; then + echo "Broken: $url (HTTP $code)" + exit 1 + fi + done• Optionally, you can leverage a dedicated GitHub Action such as stevenvachon/link-checker-action for more features (caching, parallelism, configurable status ranges).
Let me know if you’d like a full example workflow or further help setting this up!
llms.txt (5)
3-11: Mirror the intro copy improvements from the public file.Keep the root and public copies textually identical to avoid ctx diffs.
- > Private, intent-driven discovery via autonomous agents. Broker agents stake on proposed matches; successful, double opt-in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence. + > Private, intent‑driven discovery via autonomous agents. Broker agents stake on proposed matches; successful, double‑opt‑in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence scores. - - Codebase: Next.js frontend, Node/Express API, Drizzle (PostgreSQL), Redis, LangGraph-based agents - - Privacy: Agents never expose raw intent content across users; outputs are explanations + confidence only + - Codebase: Next.js frontend, Node/Express API, Drizzle (PostgreSQL), Redis, LangGraph‑based agents + - Privacy: Agents never expose raw intent content across users; outputs are explanations and confidence scores only
14-19: Note about raw links being deliberate.Same reasoning as the public copy; avoids well‑meaning edits that break ctx generation.
I can append a short HTML comment line stating: “Raw links are intentional to support automated ctx generation.”
22-25: Add stability guardrails for moving files.Consider linking to stable tags (e.g., nightly tag) in CI‑generated variants to reduce breakage when refactors land on dev.
If you want, I can add a Makefile flag to target a tag/sha for ctx generation and update the workflow to use it on nightly.
28-31: Consistent label for X/Twitter.Align naming across all docs.
- - [X (Twitter)](https://x.com/indexnetwork_): Updates and announcements + - [X](https://x.com/indexnetwork_): Updates and announcements
1-32: Optional: add metadata footer (last updated + source-of-truth).Helps consumers and downstream automations reconcile copies.
- [Blog](https://blog.index.network): Insights and posts - + + --- + Last-Updated: 2025-08-26 + Source-of-Truth: llms.txt (ctx is generated from this file by CI)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (13)
.github/workflows/llms-ctx.yml(1 hunks)Makefile(1 hunks)README.md(1 hunks)frontend/public/llms-ctx-full.txt(1 hunks)frontend/public/llms-ctx.txt(1 hunks)frontend/public/llms-full.txt(1 hunks)frontend/public/llms.txt(1 hunks)llms-ctx-full.txt(1 hunks)llms-ctx.txt(1 hunks)llms-full.txt(1 hunks)llms.txt(1 hunks)scripts/generate-ctx.sh(1 hunks)scripts/hooks/pre-commit(1 hunks)
🧰 Additional context used
🪛 LanguageTool
frontend/public/llms.txt
[grammar] ~7-~7: There might be a mistake here.
Context: ...stgreSQL), Redis, LangGraph-based agents - Privacy: Agents never expose raw intent ...
(QB_NEW_EN)
[grammar] ~14-~14: There might be a mistake here.
Context: ...md): Overview, quick start, architecture - [How It Works](https://raw.githubusercont...
(QB_NEW_EN)
[grammar] ~15-~15: There might be a mistake here.
Context: ...Technical architecture and privacy model - [Integration Guide](https://raw.githubuse...
(QB_NEW_EN)
[grammar] ~16-~16: There might be a mistake here.
Context: ...IDE.md): API usage and integration steps - [Protocol README](https://raw.githubuserc...
(QB_NEW_EN)
[grammar] ~17-~17: There might be a mistake here.
Context: ...ADME.md): Backend server, routes, and DB - [Frontend README](https://raw.githubuserc...
(QB_NEW_EN)
[grammar] ~22-~22: There might be a mistake here.
Context: .../index.ts): Mounts REST endpoints under /api - [Schema](https://raw.githubusercontent.co...
(QB_NEW_EN)
[grammar] ~23-~23: There might be a mistake here.
Context: ...s, intents, indexes, stakes, connections - [LangGraph config](https://raw.githubuser...
(QB_NEW_EN)
frontend/public/llms-ctx-full.txt
[grammar] ~1-~1: Hier könnte ein Fehler sein.
Context:
Private, intent-driven discovery via autonomous agents. Agents stake on proposed matches; successful, double opt-in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence.
This prebuilt ctx includes all links (including Optional) but omits inlined remote content due to restricted network/SSL. Regenerate with network access to embed full documents.(QB_NEW_DE)
llms-full.txt
[style] ~3-~3: Consider a different adjective to strengthen your wording.
Context: ...ts, and development to help LLMs answer deeper questions. It includes external resourc...
(DEEP_PROFOUND)
[grammar] ~7-~7: There might be a mistake here.
Context: ...uthored goal/need text; may be incognito - Index: privacy-scoped context containing...
(QB_NEW_EN)
[grammar] ~8-~8: There might be a mistake here.
Context: ...ntext containing intents and permissions - Stake: agent-created link between relate...
(QB_NEW_EN)
[grammar] ~9-~9: There might be a mistake here.
Context: ...ated intents with confidence + reasoning - Connection: explicit state transitions (...
(QB_NEW_EN)
[grammar] ~24-~24: There might be a mistake here.
Context: ...s, intents, indexes, stakes, connections - [Index access utility](https://raw.github...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ...-access.ts): Permission handling helpers - [Synthesis](https://raw.githubusercontent...
(QB_NEW_EN)
[grammar] ~34-~34: There might be a mistake here.
Context: ...rotocol/drizzle): Drizzle SQL migrations - [Drizzle config](https://github.com/index...
(QB_NEW_EN)
[grammar] ~35-~35: There might be a mistake here.
Context: ...col/drizzle.config.ts): DB configuration - [Protocol env example](https://github.com...
(QB_NEW_EN)
llms-ctx.txt
[grammar] ~1-~1: Hier könnte ein Fehler sein.
Context:
Private, intent-driven discovery via autonomous agents. Agents stake on proposed matches; successful, double opt-in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence.
This prebuilt ctx lists key sources but omits inlined content due to restricted network/SSL. Regenerate with network access to embed full documents.(QB_NEW_DE)
llms-ctx-full.txt
[grammar] ~1-~1: Hier könnte ein Fehler sein.
Context:
Private, intent-driven discovery via autonomous agents. Agents stake on proposed matches; successful, double opt-in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence.
This prebuilt ctx includes all links (including Optional) but omits inlined remote content due to restricted network/SSL. Regenerate with network access to embed full documents.(QB_NEW_DE)
frontend/public/llms-full.txt
[style] ~3-~3: Consider a different adjective to strengthen your wording.
Context: ...ts, and development to help LLMs answer deeper questions. It includes external resourc...
(DEEP_PROFOUND)
[grammar] ~7-~7: There might be a mistake here.
Context: ...uthored goal/need text; may be incognito - Index: privacy-scoped context containing...
(QB_NEW_EN)
[grammar] ~8-~8: There might be a mistake here.
Context: ...ntext containing intents and permissions - Stake: agent-created link between relate...
(QB_NEW_EN)
[grammar] ~9-~9: There might be a mistake here.
Context: ...ated intents with confidence + reasoning - Connection: explicit state transitions (...
(QB_NEW_EN)
[grammar] ~24-~24: There might be a mistake here.
Context: ...s, intents, indexes, stakes, connections - [Index access utility](https://raw.github...
(QB_NEW_EN)
[grammar] ~25-~25: There might be a mistake here.
Context: ...-access.ts): Permission handling helpers - [Synthesis](https://raw.githubusercontent...
(QB_NEW_EN)
[grammar] ~34-~34: There might be a mistake here.
Context: ...rotocol/drizzle): Drizzle SQL migrations - [Drizzle config](https://github.com/index...
(QB_NEW_EN)
[grammar] ~35-~35: There might be a mistake here.
Context: ...col/drizzle.config.ts): DB configuration - [Protocol env example](https://github.com...
(QB_NEW_EN)
frontend/public/llms-ctx.txt
[grammar] ~1-~1: Hier könnte ein Fehler sein.
Context:
Private, intent-driven discovery via autonomous agents. Agents stake on proposed matches; successful, double opt-in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence.
This prebuilt ctx lists key sources but omits inlined content due to restricted network/SSL. Regenerate with network access to embed full documents.(QB_NEW_DE)
llms.txt
[grammar] ~7-~7: There might be a mistake here.
Context: ...stgreSQL), Redis, LangGraph-based agents - Privacy: Agents never expose raw intent ...
(QB_NEW_EN)
[grammar] ~14-~14: There might be a mistake here.
Context: ...md): Overview, quick start, architecture - [How It Works](https://raw.githubusercont...
(QB_NEW_EN)
[grammar] ~15-~15: There might be a mistake here.
Context: ...Technical architecture and privacy model - [Integration Guide](https://raw.githubuse...
(QB_NEW_EN)
[grammar] ~16-~16: There might be a mistake here.
Context: ...IDE.md): API usage and integration steps - [Protocol README](https://raw.githubuserc...
(QB_NEW_EN)
[grammar] ~17-~17: There might be a mistake here.
Context: ...ADME.md): Backend server, routes, and DB - [Frontend README](https://raw.githubuserc...
(QB_NEW_EN)
[grammar] ~22-~22: There might be a mistake here.
Context: .../index.ts): Mounts REST endpoints under /api - [Schema](https://raw.githubusercontent.co...
(QB_NEW_EN)
[grammar] ~23-~23: There might be a mistake here.
Context: ...s, intents, indexes, stakes, connections - [LangGraph config](https://raw.githubuser...
(QB_NEW_EN)
README.md
[grammar] ~204-~204: There might be a mistake here.
Context: ... official CLI (requires network access): - Install: `python3 -m venv .venv && sourc...
(QB_NEW_EN)
[grammar] ~205-~205: There might be a mistake here.
Context: ...(requires network access): - Install: python3 -m venv .venv && source .venv/bin/activate && pip install llms-txt - Generate minimal: `llms_txt2ctx llms.txt...
(QB_NEW_EN)
[grammar] ~206-~206: There might be a mistake here.
Context: ...install llms-txt - Generate minimal:llms_txt2ctx llms.txt > llms-ctx.txt - Generate full:llms_txt2ctx --optional ...
(QB_NEW_EN)
[grammar] ~207-~207: There might be a mistake here.
Context: ....txt > llms-ctx.txt - Generate full:llms_txt2ctx --optional True llms.txt > llms-ctx-full.txt - Tip: Ensure links inllms.txt` point to...
(QB_NEW_EN)
[grammar] ~212-~212: There might be a mistake here.
Context: ...ration and optional auto-run - Script: scripts/generate-ctx.sh - Creates/uses .venv, installs `llms-txt...
(QB_NEW_EN)
[grammar] ~213-~213: There might be a mistake here.
Context: ...ctx files, copies to frontend/public/. - Run: bash scripts/generate-ctx.sh - O...
(QB_NEW_EN)
[grammar] ~216-~216: There might be a mistake here.
Context: ...n changes to llms.txt/llms-full.txt: - Enable hooks path: `git config core.hook...
(QB_NEW_EN)
[grammar] ~217-~217: There might be a mistake here.
Context: ...llms-full.txt: - Enable hooks path: git config core.hooksPath scripts/hooks - The hook will run the generator and stag...
(QB_NEW_EN)
[grammar] ~227-~227: There might be a mistake here.
Context: ...ightly at 03:00 UTC, and manual dispatch - Validates: parses llms.txt using `llms...
(QB_NEW_EN)
[grammar] ~228-~228: There might be a mistake here.
Context: ... using llms_txt2ctx (minimal and full) - Builds: ctx files and uploads them as ar...
(QB_NEW_EN)
🪛 checkmake (0.2.2)
Makefile
[warning] 3-3: Missing required phony target "all"
(minphony)
[warning] 3-3: Missing required phony target "clean"
(minphony)
[warning] 3-3: Missing required phony target "test"
(minphony)
🪛 actionlint (1.7.7)
.github/workflows/llms-ctx.yml
53-53: could not parse as YAML: yaml: line 53: did not find expected key
(syntax-check)
🪛 YAMLlint (1.37.1)
.github/workflows/llms-ctx.yml
[error] 55-55: syntax error: expected , but found ''
(syntax)
🔇 Additional comments (7)
frontend/public/llms-full.txt (1)
23-27: Review comment resolved: endpoint descriptions are accurateAll referenced files exist in protocol/src, and the Express app in protocol/src/index.ts mounts REST routes under
/apiexactly as described. The snippet in llms-full.txt correctly reflects:
- protocol/src/index.ts: mounts
/api/auth,/api/users,/api/upload,/api/intents,/api/stakes,/api/connections,/api/indexes,/api/integrations,/api/vibecheck, and/api/synthesis.- protocol/src/lib/schema.ts: defines tables for users, intents, indexes, stakes, and connections.
- protocol/src/lib/index-access.ts: implements permission-handling helpers.
- protocol/src/lib/synthesis.ts: provides synthesis and vibecheck utilities.
No mismatches or broken links detected.
frontend/public/llms-ctx-full.txt (1)
2-4: LGTM overall; concise summary reads well.The privacy framing is clear and consistent with the rest of the docs.
README.md (1)
205-208: Update README to document both--optionalflag formsTo avoid confusion across different CLI versions, show both the flag-only and explicit-value syntax for
--optionaland prefer the flag-only form:
- Generate full (flag-only form):
llms_txt2ctx --optional llms.txt > llms-ctx-full.txt- Generate full (explicit-value form, if required by your CLI):
llms_txt2ctx --optional true llms.txt > llms-ctx-full.txtDiff suggestion in
README.mdaround lines 205–208:- - Generate full: `llms_txt2ctx --optional True llms.txt > llms-ctx-full.txt` + - Generate full: `llms_txt2ctx --optional llms.txt > llms-ctx-full.txt` + (If your CLI requires an explicit value: `llms_txt2ctx --optional true llms.txt > llms-ctx-full.txt`)Please verify in your local environment (e.g. by running
llms_txt2ctx --help) which syntax your installed version accepts and update the docs accordingly.llms-ctx-full.txt (1)
18-22: Integrate CI path validation for ctx referencesI ran the supplied script and confirmed that all referenced files exist as-is. To prevent future breakages when these files move, incorporate this check into your CI pipeline (for example, as a GitHub Actions step).
frontend/public/llms-ctx.txt (1)
1-4: LGTM: clear summary and minimal ctx structureThe minimal context is concise and mirrors the root ctx file. Good.
Makefile (1)
3-3: Phony targets coverage is fine; ignore “all/clean/test” lint nagsThe repo appears to use workspace scripts for builds/tests. No need to add stub
all/testtargets unless your tooling requires them.llms-ctx.txt (1)
18-22: Verify referenced code paths exist; keep schema and config in syncAdd/extend CI check to ensure
protocol/src/index.ts,protocol/src/lib/schema.ts, andprotocol/src/agents/langgraph.jsonexist and are maintained when refactoring.Use the shell script provided in the llms-ctx-full.txt comment.
| on: | ||
| workflow_dispatch: | ||
| push: | ||
| branches: [ main, dev ] | ||
| paths: | ||
| - 'llms.txt' | ||
| - 'llms-full.txt' | ||
| - 'scripts/generate-ctx.sh' | ||
| pull_request: | ||
| branches: [ main, dev ] | ||
| paths: | ||
| - 'llms.txt' | ||
| - 'llms-full.txt' | ||
| - 'scripts/generate-ctx.sh' | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add nightly schedule under on: and tighten security permissions.
Move schedule here; also set least-privilege defaults.
on:
workflow_dispatch:
push:
branches: [ main, dev ]
@@
pull_request:
branches: [ main, dev ]
paths:
- 'llms.txt'
- 'llms-full.txt'
- 'scripts/generate-ctx.sh'
+ schedule:
+ - cron: '0 3 * * *'
+
+permissions:
+ contents: read📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| on: | |
| workflow_dispatch: | |
| push: | |
| branches: [ main, dev ] | |
| paths: | |
| - 'llms.txt' | |
| - 'llms-full.txt' | |
| - 'scripts/generate-ctx.sh' | |
| pull_request: | |
| branches: [ main, dev ] | |
| paths: | |
| - 'llms.txt' | |
| - 'llms-full.txt' | |
| - 'scripts/generate-ctx.sh' | |
| on: | |
| workflow_dispatch: | |
| push: | |
| branches: [ main, dev ] | |
| paths: | |
| - 'llms.txt' | |
| - 'llms-full.txt' | |
| - 'scripts/generate-ctx.sh' | |
| pull_request: | |
| branches: [ main, dev ] | |
| paths: | |
| - 'llms.txt' | |
| - 'llms-full.txt' | |
| - 'scripts/generate-ctx.sh' | |
| schedule: | |
| - cron: '0 3 * * *' | |
| permissions: | |
| contents: read |
🤖 Prompt for AI Agents
.github/workflows/llms-ctx.yml lines 3-17: add a nightly schedule entry under
the existing on: block (e.g., schedule: - cron: '0 00 * * *' to run once nightly
UTC) and tighten workflow permissions by adding a top-level permissions stanza
that uses least-privilege defaults (for example set default read-only access
like permissions: contents: read actions: read and only add any write scopes
required by the script in a narrow, explicit way); ensure the schedule is
alongside push/pull_request and the permissions stanza is at the top level of
the workflow file.
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| schedule: | ||
| - cron: '0 3 * * *' | ||
| python-version: '3.x' | ||
|
|
There was a problem hiding this comment.
Fix YAML: stray schedule is nested under a step, breaking the workflow.
The schedule trigger must live under on:. Its current placement causes a YAML parse error and prevents CI from running.
Apply this minimal fix to remove the bad block:
- name: Set up Python
uses: actions/setup-python@v5
with:
- schedule:
- - cron: '0 3 * * *'
python-version: '3.x'📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - name: Set up Python | |
| uses: actions/setup-python@v5 | |
| with: | |
| schedule: | |
| - cron: '0 3 * * *' | |
| python-version: '3.x' | |
| - name: Set up Python | |
| uses: actions/setup-python@v5 | |
| with: | |
| python-version: '3.x' |
🧰 Tools
🪛 actionlint (1.7.7)
53-53: could not parse as YAML: yaml: line 53: did not find expected key
(syntax-check)
🪛 YAMLlint (1.37.1)
[error] 55-55: syntax error: expected , but found ''
(syntax)
🤖 Prompt for AI Agents
.github/workflows/llms-ctx.yml around lines 50 to 56: there is a stray
`schedule:` block incorrectly nested under the "Set up Python" step which breaks
the workflow; remove the entire `schedule:` stanza (and its indented cron line)
from this step so the `with:` block only contains `python-version: '3.x'`, and
if you intend a scheduled run move the `schedule`/`cron` entry under the
top-level `on:` key instead.
| <Project title="Index Network"> | ||
| <Summary> | ||
| Private, intent-driven discovery via autonomous agents. Agents stake on proposed matches; successful, double opt-in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence. | ||
| </Summary> | ||
|
|
||
| <Note> | ||
| This prebuilt ctx lists key sources but omits inlined content due to restricted network/SSL. Regenerate with network access to embed full documents. | ||
| </Note> | ||
|
|
||
| <Section title="Docs"> | ||
| <Doc title="Repository README" href="https://raw.githubusercontent.com/indexnetwork/index/dev/README.md" /> | ||
| <Doc title="How It Works" href="https://raw.githubusercontent.com/indexnetwork/index/dev/HOWITWORKS.md" /> | ||
| <Doc title="Integration Guide" href="https://raw.githubusercontent.com/indexnetwork/index/dev/docs/INTEGRATION_GUIDE.md" /> | ||
| <Doc title="Protocol README" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/README.md" /> | ||
| <Doc title="Frontend README" href="https://raw.githubusercontent.com/indexnetwork/index/dev/frontend/README.md" /> | ||
| </Section> | ||
|
|
||
| <Section title="API"> | ||
| <Doc title="API entry (Express app)" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/index.ts" /> | ||
| <Doc title="Schema" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/lib/schema.ts" /> | ||
| <Doc title="LangGraph config" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/agents/langgraph.json" /> | ||
| </Section> | ||
|
|
||
| </Project> |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Serve under /.well-known in addition to root
To match the spec and your README, also publish this file as frontend/public/.well-known/llms-ctx.txt (see Makefile change below).
🧰 Tools
🪛 LanguageTool
[grammar] ~1-~1: Hier könnte ein Fehler sein.
Context:
Private, intent-driven discovery via autonomous agents. Agents stake on proposed matches; successful, double opt-in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence.
This prebuilt ctx lists key sources but omits inlined content due to restricted network/SSL. Regenerate with network access to embed full documents.(QB_NEW_DE)
🤖 Prompt for AI Agents
In frontend/public/llms-ctx.txt lines 1-24: the context file is only published
at /llms-ctx.txt but needs to also be available at /.well-known/llms-ctx.txt to
match the spec and README; fix by ensuring the build/static/public pipeline
places a copy at frontend/public/.well-known/llms-ctx.txt (or adjust the static
assets step to symlink or copy this file into a .well-known directory during
build), and update the Makefile/static-server config so the file is served from
both paths (or that .well-known is included in the list of published public
assets).
| <Project title="Index Network"> | ||
| <Summary> | ||
| Private, intent-driven discovery via autonomous agents. Agents stake on proposed matches; successful, double opt-in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence. | ||
| </Summary> | ||
|
|
||
| <Note> | ||
| This prebuilt ctx lists key sources but omits inlined content due to restricted network/SSL. Regenerate with network access to embed full documents. | ||
| </Note> | ||
|
|
||
| <Section title="Docs"> | ||
| <Doc title="Repository README" href="https://raw.githubusercontent.com/indexnetwork/index/dev/README.md" /> | ||
| <Doc title="How It Works" href="https://raw.githubusercontent.com/indexnetwork/index/dev/HOWITWORKS.md" /> | ||
| <Doc title="Integration Guide" href="https://raw.githubusercontent.com/indexnetwork/index/dev/docs/INTEGRATION_GUIDE.md" /> | ||
| <Doc title="Protocol README" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/README.md" /> | ||
| <Doc title="Frontend README" href="https://raw.githubusercontent.com/indexnetwork/index/dev/frontend/README.md" /> | ||
| </Section> | ||
|
|
||
| <Section title="API"> | ||
| <Doc title="API entry (Express app)" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/index.ts" /> | ||
| <Doc title="Schema" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/lib/schema.ts" /> | ||
| <Doc title="LangGraph config" href="https://raw.githubusercontent.com/indexnetwork/index/dev/protocol/src/agents/langgraph.json" /> | ||
| </Section> | ||
|
|
||
| </Project> |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Serve under /.well-known as well
Mirror the public publishing recommendation so clients can discover the canonical path.
🧰 Tools
🪛 LanguageTool
[grammar] ~1-~1: Hier könnte ein Fehler sein.
Context:
Private, intent-driven discovery via autonomous agents. Agents stake on proposed matches; successful, double opt-in connections earn rewards. Privacy is preserved by keeping raw user intents private while sharing only agent reasoning and confidence.
This prebuilt ctx lists key sources but omits inlined content due to restricted network/SSL. Regenerate with network access to embed full documents.(QB_NEW_DE)
🤖 Prompt for AI Agents
In llms-ctx.txt around lines 1-24, the context file lists public doc links but
omits a canonical discovery path; add a mirror of the public publishing
recommendation by adding an entry that exposes the project context at a
/.well-known path (for example /.well-known/index-network) so clients can
discover the canonical path; then update the API (protocol/src/index.ts) to
serve the same context under that /.well-known route (returning the same
content/manifest) so both locations are equivalent and discoverable.
| ctx-full: | ||
| LLMSTXT_FILE=llms.txt bash scripts/generate-ctx.sh | ||
|
|
There was a problem hiding this comment.
Likely wrong source for “full” generation
ctx-full still uses LLMSTXT_FILE=llms.txt. If the intent is to build from llms-full.txt, switch the input.
Apply:
-ctx-full:
- LLMSTXT_FILE=llms.txt bash scripts/generate-ctx.sh
+ctx-full:
+ LLMSTXT_FILE=llms-full.txt bash scripts/generate-ctx.sh📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ctx-full: | |
| LLMSTXT_FILE=llms.txt bash scripts/generate-ctx.sh | |
| ctx-full: | |
| LLMSTXT_FILE=llms-full.txt bash scripts/generate-ctx.sh |
🤖 Prompt for AI Agents
In Makefile around lines 8 to 10, the ctx-full target is using
LLMSTXT_FILE=llms.txt but should use the full input file; change the environment
variable to LLMSTXT_FILE=llms-full.txt so scripts/generate-ctx.sh reads
llms-full.txt for the "full" context generation and verify the scripts still
accept that filename (update the script invocation or variable name if needed).
| ctx-public: ctx | ||
| cp -f llms-ctx.txt llms-ctx-full.txt frontend/public/ | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion
Make publishing idempotent and ensure directory existence
Use mkdir -p to avoid failures if frontend/public/ is missing in some environments, and also publish to /.well-known/.
-ctx-public: ctx
- cp -f llms-ctx.txt llms-ctx-full.txt frontend/public/
+ctx-public: ctx
+ mkdir -p frontend/public/.well-known
+ cp -f llms-ctx.txt llms-ctx-full.txt frontend/public/
+ cp -f llms-ctx.txt frontend/public/.well-known/llms-ctx.txt
+ cp -f llms-ctx-full.txt frontend/public/.well-known/llms-ctx-full.txt🤖 Prompt for AI Agents
In Makefile around lines 11 to 13, the publish step assumes frontend/public/
exists and only copies to that directory; make it idempotent by creating the
target directories first (mkdir -p frontend/public and mkdir -p
frontend/public/.well-known) and then copy the files to both frontend/public/
and frontend/public/.well-known/ so the operation succeeds even if directories
are missing and the context is published under .well-known as requested.
| - `/.well-known`: The project exposes LLM-friendly indexes: | ||
| - `GET /llms.txt` – Concise index for LLMs (spec-compliant) | ||
| - `GET /llms-full.txt` – Expanded index with more references | ||
| - `GET /llms-ctx.txt` – Prebuilt context (docs + API; excludes Optional) | ||
| - `GET /llms-ctx-full.txt` – Prebuilt context including Optional links | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion
Clarify and align endpoint paths with the llms.txt spec
You say “/.well-known: The project exposes …” but the listed endpoints are root-scoped (/llms*.txt). For spec compliance and best interoperability, serve under /.well-known and optionally keep the root aliases. Also update text to reflect both.
Apply this diff to the README section:
- - `/.well-known`: The project exposes LLM-friendly indexes:
- - `GET /llms.txt` – Concise index for LLMs (spec-compliant)
- - `GET /llms-full.txt` – Expanded index with more references
- - `GET /llms-ctx.txt` – Prebuilt context (docs + API; excludes Optional)
- - `GET /llms-ctx-full.txt` – Prebuilt context including Optional links
+ - Endpoints (spec-compliant location first, root aliases optional):
+ - `GET /.well-known/llms.txt` (alias: `/llms.txt`)
+ - `GET /.well-known/llms-full.txt` (alias: `/llms-full.txt`)
+ - `GET /.well-known/llms-ctx.txt` (alias: `/llms-ctx.txt`)
+ - `GET /.well-known/llms-ctx-full.txt` (alias: `/llms-ctx-full.txt`)Follow-up: ensure files are published under frontend/public/.well-known/ (see Makefile suggestion below).
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - `/.well-known`: The project exposes LLM-friendly indexes: | |
| - `GET /llms.txt` – Concise index for LLMs (spec-compliant) | |
| - `GET /llms-full.txt` – Expanded index with more references | |
| - `GET /llms-ctx.txt` – Prebuilt context (docs + API; excludes Optional) | |
| - `GET /llms-ctx-full.txt` – Prebuilt context including Optional links | |
| - Endpoints (spec-compliant location first, root aliases optional): | |
| - `GET /.well-known/llms.txt` (alias: `/llms.txt`) | |
| - `GET /.well-known/llms-full.txt` (alias: `/llms-full.txt`) | |
| - `GET /.well-known/llms-ctx.txt` (alias: `/llms-ctx.txt`) | |
| - `GET /.well-known/llms-ctx-full.txt` (alias: `/llms-ctx-full.txt`) |
🤖 Prompt for AI Agents
In README.md around lines 198 to 203, the documentation lists llms*.txt
endpoints at root but should clarify they are served under /.well-known for spec
compliance and note optional root aliases; update the text to state that the
canonical paths are /.well-known/llms.txt, /.well-known/llms-full.txt,
/.well-known/llms-ctx.txt, and /.well-known/llms-ctx-full.txt and mention that
root-scoped aliases (e.g., /llms.txt) are kept optionally for convenience; also
add a brief follow-up note instructing that the files must be published to
frontend/public/.well-known/ (and reference the Makefile change suggestion).
| # | ||
| # Usage: | ||
| # scripts/generate-ctx.sh | ||
| # LLMSTXT_FILE=llms.txt scripts/generate-ctx.sh # alternate source file |
There was a problem hiding this comment.
Fix usage comment: the alternate source should be llms-full.txt.
This typo can mislead developers.
-# LLMSTXT_FILE=llms.txt scripts/generate-ctx.sh # alternate source file
+# LLMSTXT_FILE=llms-full.txt scripts/generate-ctx.sh # alternate source file📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # LLMSTXT_FILE=llms.txt scripts/generate-ctx.sh # alternate source file | |
| # LLMSTXT_FILE=llms-full.txt scripts/generate-ctx.sh # alternate source file |
🤖 Prompt for AI Agents
In scripts/generate-ctx.sh around line 10, the usage comment references the
wrong alternate source file ("llms.txt"); update the comment to refer to
"llms-full.txt" so the help/usage line accurately reflects the intended
alternate source.
| if [[ ! -f "${ROOT_DIR}/${SRC_FILE}" ]]; then | ||
| echo "[llms] Error: ${SRC_FILE} not found in repo root" >&2 | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Support absolute and non-root paths for LLMSTXT_FILE.
Current check forces files to live under repo root. Allow absolute paths or relative paths outside root.
-echo "[llms] Using source: ${SRC_FILE}"
-if [[ ! -f "${ROOT_DIR}/${SRC_FILE}" ]]; then
- echo "[llms] Error: ${SRC_FILE} not found in repo root" >&2
- exit 1
-fi
+echo "[llms] Using source: ${SRC_FILE}"
+# Resolve source to an absolute path. Prefer explicit path if provided.
+if [[ -f "${SRC_FILE}" ]]; then
+ SRC_ABS="$(cd "$(dirname "${SRC_FILE}")" && pwd)/$(basename "${SRC_FILE}")"
+elif [[ -f "${ROOT_DIR}/${SRC_FILE}" ]]; then
+ SRC_ABS="${ROOT_DIR}/${SRC_FILE}"
+else
+ echo "[llms] Error: ${SRC_FILE} not found (checked ${SRC_FILE} and ${ROOT_DIR}/${SRC_FILE})" >&2
+ exit 1
+fiAlso update downstream references from "${ROOT_DIR}/${SRC_FILE}" to "${SRC_ABS}".
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if [[ ! -f "${ROOT_DIR}/${SRC_FILE}" ]]; then | |
| echo "[llms] Error: ${SRC_FILE} not found in repo root" >&2 | |
| exit 1 | |
| fi | |
| echo "[llms] Using source: ${SRC_FILE}" | |
| # Resolve source to an absolute path. Prefer explicit path if provided. | |
| if [[ -f "${SRC_FILE}" ]]; then | |
| SRC_ABS="$(cd "$(dirname "${SRC_FILE}")" && pwd)/$(basename "${SRC_FILE}")" | |
| elif [[ -f "${ROOT_DIR}/${SRC_FILE}" ]]; then | |
| SRC_ABS="${ROOT_DIR}/${SRC_FILE}" | |
| else | |
| echo "[llms] Error: ${SRC_FILE} not found (checked ${SRC_FILE} and ${ROOT_DIR}/${SRC_FILE})" >&2 | |
| exit 1 | |
| fi |
🤖 Prompt for AI Agents
In scripts/generate-ctx.sh around lines 19 to 22, the script currently enforces
that LLMSTXT_FILE (SRC_FILE) must exist under the repo root; change the
existence check to support absolute paths and relative paths outside root by
resolving SRC_FILE to an absolute path (e.g., SRC_ABS) and testing that file
path (if [[ ! -f "${SRC_ABS}" ]]; then ... ); then update all downstream
references in the script that currently use "${ROOT_DIR}/${SRC_FILE}" to use the
resolved "${SRC_ABS}" variable instead so the script works for absolute and
non-root-relative paths.
…update Adopt llms.txt spec, add llms-full, ctx generation, and CI
2 participants
This PR standardizes our LLM-facing docs and automates ctx generation.
Highlights
Notes
After merge
Summary by CodeRabbit