Added stateless llms.txt service with dependency injection#28042
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (18)
✅ Files skipped from review due to trivial changes (3)
🚧 Files skipped from review as they are similar to previous changes (13)
WalkthroughAdds LLMS support: a markdown conversion module, a createLlmsService factory producing llms.txt and llms-full.txt (index and bounded full export), an Express handler mounting plaintext endpoints, per-entry .md routes and controller branches to serve/redirect markdown, discovery middleware and header handling, labs/config defaults, tests, and package/workspace dependency updates. Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
ghost/core/test/unit/frontend/services/llms/markdown.test.js (1)
144-167: ⚡ Quick winConsider adding edge-case tests for better coverage.
The integration test provides good coverage of the happy path, but consider adding tests for:
formatIsoDatewith invalid date strings (would catch the issue flagged in markdown.js)renderEntryMarkdownwith missing title, no metadata fieldsrenderEntryMarkdownBodyfallback tohtmlToPlaintext.excerpt()when bothhtmlandplaintextare falsyThese additions would improve test maintainability by isolating failure points and documenting expected behavior for edge cases.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ghost/core/test/unit/frontend/services/llms/markdown.test.js` around lines 144 - 167, Add unit tests to cover the edge cases: 1) for formatIsoDate, add tests that pass invalid/empty date strings and assert it handles them (returns empty string or expected fallback) to catch parsing errors; 2) for renderEntryMarkdown, add a test with an entry missing title and all optional metadata (no url, authors, tags, custom_excerpt, html, plaintext) and assert the output still renders safely (no undefined values and expected minimal placeholders or sections omitted); 3) for renderEntryMarkdownBody, add a test where both html and plaintext are falsy and assert it falls back to calling htmlToPlaintext.excerpt() (mock/stub htmlToPlaintext.excerpt to verify it was invoked and its output used). Reference the functions formatIsoDate, renderEntryMarkdown, renderEntryMarkdownBody, and htmlToPlaintext.excerpt when adding these tests.ghost/core/core/frontend/services/llms/service.js (1)
197-216: Consider memory implications oflimit: 'all'for large sites.The index generation fetches all public pages/posts at once with
limit: 'all'. While thehtmlcolumn is excluded to reduce memory usage, sites with tens of thousands of entries may still experience memory pressure during index generation.For future optimization, consider adding a pagination option for the index or documenting expected memory requirements for large sites.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ghost/core/core/frontend/services/llms/service.js` around lines 197 - 216, fetchIndexEntries currently calls models.Post.findPage with limit: 'all', which can cause high memory usage for very large sites; change it to use paginated fetching instead of 'all' by calling models.Post.findPage in a loop (or using offset/page parameters) and accumulating entries in batches, using the same order/filter/columns and early filtering of entries without urls, so you only hold one batch in memory at a time; update fetchIndexEntries to iterate pages until no more results (or expose a pagination option) rather than requesting all rows at once from models.Post.findPage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ghost/core/core/frontend/services/llms/markdown.js`:
- Around line 82-88: formatIsoDate currently calls new Date(value).toISOString()
without validating the date, which can throw RangeError for invalid values;
update formatIsoDate to validate the parsed Date (use new Date(value) then check
isNaN(date.getTime()) or !isFinite(date)) and return null (or a safe fallback)
when the value is invalid, otherwise return date.toISOString(); reference the
function name formatIsoDate and the calls to new Date(...) and toISOString()
when making the change.
In `@ghost/core/core/frontend/services/llms/service.js`:
- Around line 17-28: Add unit tests for fetchPublicEntry: mock
api.pagesPublic.read and api.postsPublic.read and assert the correct controller
is called depending on resourceType ('pages' uses api.pagesPublic, 'posts' uses
api.postsPublic), that the call receives id, formats: 'html,plaintext', include:
'authors,tags' and context with the provided member, that the function returns
the first item from response[responseKey] when present (exercise optional
chaining behavior), and that it returns null when response or
response[responseKey] is empty/undefined; use spies/mocks to validate the
controller selection and context forwarding for fetchPublicEntry.
In `@pnpm-workspace.yaml`:
- Line 54: The dependency entry for node-html-markdown is using a caret range
(^2.0.0) which breaks reproducible builds; update the catalog dependency
declaration for node-html-markdown to use the exact pinned version (change
^2.0.0 to 2.0.0) so it matches the other pinned catalog dependencies and
complies with strict catalogMode.
---
Nitpick comments:
In `@ghost/core/core/frontend/services/llms/service.js`:
- Around line 197-216: fetchIndexEntries currently calls models.Post.findPage
with limit: 'all', which can cause high memory usage for very large sites;
change it to use paginated fetching instead of 'all' by calling
models.Post.findPage in a loop (or using offset/page parameters) and
accumulating entries in batches, using the same order/filter/columns and early
filtering of entries without urls, so you only hold one batch in memory at a
time; update fetchIndexEntries to iterate pages until no more results (or expose
a pagination option) rather than requesting all rows at once from
models.Post.findPage.
In `@ghost/core/test/unit/frontend/services/llms/markdown.test.js`:
- Around line 144-167: Add unit tests to cover the edge cases: 1) for
formatIsoDate, add tests that pass invalid/empty date strings and assert it
handles them (returns empty string or expected fallback) to catch parsing
errors; 2) for renderEntryMarkdown, add a test with an entry missing title and
all optional metadata (no url, authors, tags, custom_excerpt, html, plaintext)
and assert the output still renders safely (no undefined values and expected
minimal placeholders or sections omitted); 3) for renderEntryMarkdownBody, add a
test where both html and plaintext are falsy and assert it falls back to calling
htmlToPlaintext.excerpt() (mock/stub htmlToPlaintext.excerpt to verify it was
invoked and its output used). Reference the functions formatIsoDate,
renderEntryMarkdown, renderEntryMarkdownBody, and htmlToPlaintext.excerpt when
adding these tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 9d3c2241-d955-43a8-9c5c-0ec254852617
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (13)
ghost/core/core/frontend/services/llms/handler.jsghost/core/core/frontend/services/llms/markdown.jsghost/core/core/frontend/services/llms/service.jsghost/core/core/frontend/services/routing/controllers/entry.jsghost/core/core/frontend/web/middleware/llms-discovery.jsghost/core/core/frontend/web/middleware/static-theme.jsghost/core/core/frontend/web/site.jsghost/core/core/server/web/shared/middleware/pretty-urls.jsghost/core/core/shared/config/defaults.jsonghost/core/package.jsonghost/core/test/unit/frontend/services/llms/markdown.test.jsghost/core/test/unit/frontend/services/llms/service.test.jspnpm-workspace.yaml
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #28042 +/- ##
==========================================
- Coverage 73.82% 73.62% -0.21%
==========================================
Files 1532 1536 +4
Lines 130096 130745 +649
Branches 15605 15637 +32
==========================================
+ Hits 96046 96259 +213
- Misses 33061 33520 +459
+ Partials 989 966 -23
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ErisDS
force-pushed
the
llms-txt-stateless
branch
from
406f910 to
76dbf55
Compare
Contributor
E2E Tests FailedTo view the Playwright test report locally, run: REPORT_DIR=$(mktemp -d) && gh run download 26237926440 -n playwright-report -D "$REPORT_DIR" && npx playwright show-report "$REPORT_DIR" |
16 tasks
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ghost/core/core/frontend/services/llms/markdown.js`:
- Around line 56-60: The Accept header check in getAcceptedMarkdownContentType
is case-sensitive and will miss values like "Text/Markdown"; normalize the
header value (e.g., call toLowerCase() on req.get('Accept') or use a
case-insensitive match) before testing for 'text/markdown' or 'text/plain', then
run the existing includes checks against the normalized string so both mixed-
and upper-case media types are accepted.
In `@ghost/core/core/frontend/services/llms/service.js`:
- Around line 233-235: The current conditional only sorts entries by URL on the
first page (if (type === 'page' && pageNum === 1) { entries.sort(...) }),
causing inconsistent ordering across later paginated page exports; move or apply
the entries.sort((left, right) => left.url.localeCompare(right.url)) so that
sorting happens for all page exports before pagination (or remove the pageNum
check) so every page uses the same DB ordering; update the code paths that build
page entries (referencing variables type, pageNum, and entries) to ensure
sorting is applied consistently regardless of pageNum.
In `@ghost/core/core/frontend/services/routing/controllers/entry.js`:
- Around line 96-99: The response for negotiated markdown (built with
renderEntryMarkdown and using markdownContentType and llmsIndexUrl) is cacheable
but varies by Accept; add a Vary header for Accept before sending the response
(use res.vary('Accept') or res.set('Vary', 'Accept')) so shared caches don’t
serve markdown to HTML clients, then continue to set Cache-Control and send as
before.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 3d913420-8017-43c8-8706-3404b7c6faef
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (14)
ghost/core/core/frontend/services/llms/handler.jsghost/core/core/frontend/services/llms/markdown.jsghost/core/core/frontend/services/llms/service.jsghost/core/core/frontend/services/routing/controllers/entry.jsghost/core/core/frontend/web/middleware/llms-discovery.jsghost/core/core/frontend/web/middleware/static-theme.jsghost/core/core/frontend/web/site.jsghost/core/core/server/web/shared/middleware/pretty-urls.jsghost/core/core/shared/config/defaults.jsonghost/core/core/shared/labs.jsghost/core/package.jsonghost/core/test/unit/frontend/services/llms/markdown.test.jsghost/core/test/unit/frontend/services/llms/service.test.jspnpm-workspace.yaml
✅ Files skipped from review due to trivial changes (3)
- ghost/core/core/frontend/web/middleware/static-theme.js
- ghost/core/package.json
- ghost/core/core/shared/config/defaults.json
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
ghost/core/test/unit/frontend/services/llms/service.test.js (1)
210-210:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winTruncation message assertion doesn't match the configured budget.
The test sets
fullTxtBudget: 1024(1 KiB) on line 199, but this assertion expects the truncation message to say "Truncated after 5 MiB". Either:
- The service is hardcoding "5 MiB" in the message (misleading users/LLMs about the actual limit), or
- The service correctly uses the actual budget and this assertion will fail.
Update the assertion to match the actual budget used, or verify the service implementation includes the correct value in the truncation message.
🛠️ Suggested fix
If the message should reflect the actual budget:
- assert.match(llmsFullTxt, /Truncated after 5 MiB/); + assert.match(llmsFullTxt, /Truncated after 1024 bytes|Truncated at budget/);Or if the message should be generic:
- assert.match(llmsFullTxt, /Truncated after 5 MiB/); + assert.match(llmsFullTxt, /Truncated/);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ghost/core/test/unit/frontend/services/llms/service.test.js` at line 210, The test assertion under llms/service.test.js is checking for "Truncated after 5 MiB" while the test config sets fullTxtBudget: 1024 (1 KiB); update the assertion to reflect the configured budget by either formatting the expected truncation message using the fullTxtBudget value (e.g., derive "Truncated after 1 KiB" from fullTxtBudget) or change the assertion to a generic pattern that matches any truncation message (e.g., /Truncated after/), so the expectation in the test aligns with the behavior of the code that produces the truncation message.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@ghost/core/test/unit/frontend/services/llms/service.test.js`:
- Line 210: The test assertion under llms/service.test.js is checking for
"Truncated after 5 MiB" while the test config sets fullTxtBudget: 1024 (1 KiB);
update the assertion to reflect the configured budget by either formatting the
expected truncation message using the fullTxtBudget value (e.g., derive
"Truncated after 1 KiB" from fullTxtBudget) or change the assertion to a generic
pattern that matches any truncation message (e.g., /Truncated after/), so the
expectation in the test aligns with the behavior of the code that produces the
truncation message.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 432a75b1-3c7c-43b0-8e6d-9bc896ce879b
⛔ Files ignored due to path filters (1)
ghost/core/test/e2e-api/admin/__snapshots__/config.test.js.snapis excluded by!**/*.snap
📒 Files selected for processing (4)
ghost/core/core/frontend/services/llms/markdown.jsghost/core/core/frontend/services/llms/service.jsghost/core/core/frontend/services/routing/controllers/entry.jsghost/core/test/unit/frontend/services/llms/service.test.js
ErisDS
added
preview
ErisDS
force-pushed
the
llms-txt-stateless
branch
from
648e236 to
9029929
Compare
ErisDS
added
preview
ErisDS
force-pushed
the
llms-txt-stateless
branch
from
322e648 to
757b6b3
Compare
ErisDS
added
preview
ErisDS
force-pushed
the
llms-txt-stateless
branch
from
c1c7a23 to
9d55b2a
Compare
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (2)
ghost/core/test/unit/frontend/services/routing/collection-router.test.js (2)
68-70: ⚡ Quick winVerify the mounted handler, not just the route pattern.
The assertion only checks the route pattern (
mountRouteSpy.args[3][0]), but doesn't verify the second argument (the handler). Other route assertions in this test (lines 58, 62, 66) check both the pattern and the controller. Since the markdown route uses an anonymous middleware wrapper, consider verifying thatmountRouteSpy.args[3][1]is a function, or add a note explaining why handler verification is deferred to integration tests.🧪 Proposed enhancement to verify the handler
// markdown variant assert.equal(mountRouteSpy.args[3][0], '/:slug.md'); +assert.equal(typeof mountRouteSpy.args[3][1], 'function');🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ghost/core/test/unit/frontend/services/routing/collection-router.test.js` around lines 68 - 70, The test currently only asserts the markdown route pattern via mountRouteSpy.args[3][0] but omits verifying the mounted handler; update the test around the markdown variant to also assert that mountRouteSpy.args[3][1] is a function (i.e., the mounted handler is present and callable) — locate the assertions using mountRouteSpy in collection-router.test.js and add a check for the second argument being a function, or alternatively add a short comment explaining that handler verification is covered by integration tests if you intentionally choose not to assert it here.
117-119: ⚡ Quick winVerify the mounted handler, not just the route pattern.
Same issue as the "default" test case: only the route pattern is verified, not the handler. Consider verifying that
mountRouteSpy.args[3][1]is a function for consistency with the other route assertions in this test.🧪 Proposed enhancement to verify the handler
// markdown variant assert.equal(mountRouteSpy.args[3][0], '/blog/:year/:slug.md'); +assert.equal(typeof mountRouteSpy.args[3][1], 'function');🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ghost/core/test/unit/frontend/services/routing/collection-router.test.js` around lines 117 - 119, The test currently only asserts the route pattern (mountRouteSpy.args[3][0] === '/blog/:year/:slug.md') but should also verify the mounted handler; update the assertion to check that mountRouteSpy.args[3][1] is a function (similar to other assertions in this test) so you confirm the handler was mounted correctly — locate the assertion using mountRouteSpy.args[3][0] and add a corresponding check that mountRouteSpy.args[3][1] is of type function.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@ghost/core/test/unit/frontend/services/routing/collection-router.test.js`:
- Around line 68-70: The test currently only asserts the markdown route pattern
via mountRouteSpy.args[3][0] but omits verifying the mounted handler; update the
test around the markdown variant to also assert that mountRouteSpy.args[3][1] is
a function (i.e., the mounted handler is present and callable) — locate the
assertions using mountRouteSpy in collection-router.test.js and add a check for
the second argument being a function, or alternatively add a short comment
explaining that handler verification is covered by integration tests if you
intentionally choose not to assert it here.
- Around line 117-119: The test currently only asserts the route pattern
(mountRouteSpy.args[3][0] === '/blog/:year/:slug.md') but should also verify the
mounted handler; update the assertion to check that mountRouteSpy.args[3][1] is
a function (similar to other assertions in this test) so you confirm the handler
was mounted correctly — locate the assertion using mountRouteSpy.args[3][0] and
add a corresponding check that mountRouteSpy.args[3][1] is of type function.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 66a34aca-8096-4558-a670-20d1224543bf
⛔ Files ignored due to path filters (2)
ghost/core/test/e2e-api/admin/__snapshots__/config.test.js.snapis excluded by!**/*.snappnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (18)
apps/admin-x-settings/src/components/settings/advanced/labs/private-features.tsxghost/core/core/frontend/services/llms/handler.jsghost/core/core/frontend/services/llms/markdown.jsghost/core/core/frontend/services/llms/service.jsghost/core/core/frontend/services/routing/collection-router.jsghost/core/core/frontend/services/routing/controllers/entry.jsghost/core/core/frontend/services/routing/static-pages-router.jsghost/core/core/frontend/web/middleware/llms-discovery.jsghost/core/core/frontend/web/middleware/static-theme.jsghost/core/core/frontend/web/site.jsghost/core/core/server/web/shared/middleware/pretty-urls.jsghost/core/core/shared/config/defaults.jsonghost/core/core/shared/labs.jsghost/core/package.jsonghost/core/test/unit/frontend/services/llms/markdown.test.jsghost/core/test/unit/frontend/services/llms/service.test.jsghost/core/test/unit/frontend/services/routing/collection-router.test.jspnpm-workspace.yaml
🚧 Files skipped from review as they are similar to previous changes (16)
- ghost/core/core/frontend/web/middleware/static-theme.js
- ghost/core/core/frontend/services/routing/static-pages-router.js
- pnpm-workspace.yaml
- apps/admin-x-settings/src/components/settings/advanced/labs/private-features.tsx
- ghost/core/package.json
- ghost/core/core/frontend/web/middleware/llms-discovery.js
- ghost/core/core/server/web/shared/middleware/pretty-urls.js
- ghost/core/core/frontend/web/site.js
- ghost/core/core/shared/config/defaults.json
- ghost/core/core/frontend/services/routing/controllers/entry.js
- ghost/core/test/unit/frontend/services/llms/service.test.js
- ghost/core/core/frontend/services/routing/collection-router.js
- ghost/core/core/frontend/services/llms/handler.js
- ghost/core/core/frontend/services/llms/service.js
- ghost/core/test/unit/frontend/services/llms/markdown.test.js
- ghost/core/core/frontend/services/llms/markdown.js
ErisDS
force-pushed
the
llms-txt-stateless
branch
from
9d55b2a to
16f7cad
Compare
Ghost is moving toward stateless, compute-on-request architecture.
This implements llms.txt/llms-full.txt generation, per-entry .md
export, Accept: text/markdown content negotiation, and discovery
headers using that approach — no in-memory cache, no event listeners,
full dependency injection via factory functions.
Gated behind a config flag ("llms": false by default) so the feature
can be tested before broad rollout. The llms_enabled setting (already
in main) acts as the user-facing toggle within an enabled deployment.
Key design decisions:
- Factory functions (createLlmsService, createLlmsHandler,
createLlmsDiscovery) receive all dependencies explicitly
- Uses urlServiceFacade for lazy routing compatibility
- Paginated DB queries for llms-full.txt (100/batch, 5 MiB budget)
- Index queries exclude html column to reduce memory
- Cache-Control headers on all responses for CDN/proxy caching
- pretty-urls extension bypass scoped to .md and .txt only
- markdown.js is fully pure (no Ghost singleton imports)
Ref #27984
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
A private labs flag (llmsTxt) is more flexible than a config flag because it can be toggled per-site via the API without a server restart, and support can enable it for individual customers. Routes are now always mounted; isEnabled() checks the labs flag at request time alongside the llms_enabled setting and is_private. This fits the stateless architecture — zero boot-time gating. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Config API acceptance test snapshots the full labs object. Adding the llmsTxt private feature flag requires updating the snapshot to include the new key. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Handle Accept header case-insensitively (RFC 7231) - Add Vary: Accept to content-negotiated markdown responses - Remove inconsistent page sorting in fetchFullEntries (DB order via 'id asc' is already stable; index uses fetchIndexEntries which sorts all pages by URL) - Make fullTxtBudget injectable to avoid 5MB allocation in tests that was causing OOM on CI runners Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Register /:slug.md routes on CollectionRouter and StaticPagesRouter
instead of a catch-all /.+\.md$/ regex on the site app
- Entry controller handles .md requests using the already-loaded entry
from entryLookup, eliminating the duplicate fetchPublicEntry call
- Content negotiation (Accept: text/markdown) uses the same shared
serveMarkdown helper with DI via req.app.get('llmsService')
- Remove mountMarkdownRoutes and parallel URL resolution path from
handler.js
- Expose llmsService on siteApp for consistent dependency injection
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Errors from llms.txt/llms-full.txt routes now get tagged with [llms] in logs and llms.serve_index/llms.serve_full in Sentry before being forwarded to Express error handling. Centralizes try/catch in serveLlms instead of duplicating across four route handlers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These flags were promoted to GA on main while our branch still listed them as private, causing duplicate flag detection to fail in CI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The URL service needs tags/authors to substitute :primary_tag and :primary_author in permalink patterns. Without withRelated, those placeholders resolve to 'all'/'undefined', producing wrong URLs. Also added published_at to index query for date-based permalinks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
An AI agent requesting the .md URL of a members-only post now gets a 403 with an explanation instead of a silent 404, so it can distinguish restricted content from missing content. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Added all tags (not just primary) and entry type (post/page) to the per-entry .md output so AI agents can classify content by topic and distinguish evergreen pages from time-bound posts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ErisDS
force-pushed
the
llms-txt-stateless
branch
from
16f7cad to
9cd67af
Compare
Contributor
|
Actionable comments posted: 0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
llmsTxt) +llms_enabledsetting as user-facing togglewithRelated: ['tags', 'authors']for correct permalink resolution[llms]prefix) and Sentry reporting on handler errorsRef #27984
Architecture
Routing approach
Per-entry .md URLs are registered as suffix routes directly on CollectionRouter and StaticPagesRouter, reusing the existing
entryLookuppipeline. This avoids a double-fetch (the broad regex approach needed a separatefetchPublicEntrycall) and respects all existing routing features (_respectDominantRouter, redirect handling, etc.).DI and service wiring
createLlmsService()receives all dependencies as constructor params. The service is created once at boot insite.jsand exposed viasiteApp.set('llmsService', llmsService)— entry controller accesses it viareq.app.get('llmsService').What is NOT included (intentionally)
llms_enabledsetting already exists in main (merged via Added llms_enabled site setting #27995), this PR adds the labs gate onlyTest plan
Accept: text/markdownon a public post returns markdown withVary: AcceptLinkandX-Llms-Txtpointing to /llms.txt