docs: reduce llms.txt size from ~196K to ~84K characters by marcel-rbro · Pull Request #2470 · apify/apify-docs

marcel-rbro · 2026-04-28T08:10:53Z

Summary

Reduce llms.txt from ~196K to ~84K characters (57% reduction, under 100K target)
Replace live fetching of external repo llms.txt with a curated static file (scripts/llms-external-curated.txt) that keeps overview, concept, and guide pages but drops ~400 individual class/interface/enum reference entries
Exclude individual API endpoint pages from the Docusaurus llms-txt plugin via excludeRoutes, keeping only Introduction category pages and Getting started
Exclude legacy Academy content (old JS scraping course, node-js/python tutorial collections, exercise solutions, marketing playbook deep pages) and outdated legal documents
llms-full.txt remains unchanged - it still fetches all content from external repos

Test plan

npm run build succeeds
wc -c build/llms.txt is under 100K characters
build/llms.txt contains API Introduction pages but no individual endpoint pages
build/llms.txt contains no legacy Academy content or reference class entries
build/llms-full.txt size is unchanged

🤖 Generated with Claude Code

The llms.txt file was over 200K characters, well above the recommended 100K limit. This reduces it to ~84K by: 1. Replacing live fetching of external repo llms.txt files with a curated static file that keeps overview/guide/concept pages but drops individual class/interface/enum reference entries (~67K saved) 2. Excluding individual API endpoint pages from the plugin output via excludeRoutes, keeping only Introduction category pages (~17K saved) 3. Excluding legacy Academy content (old JS course, node-js/python tutorial collections, exercise solutions, marketing playbook deep pages) and outdated legal documents (~27K saved) The llms-full.txt file remains unchanged - it still fetches and includes all content from external repos. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

apify-service-account · 2026-04-28T08:14:55Z

Preview for this PR was built for commit 6ae0861 and is ready at https://pr-2470.preview.docs.apify.com!

jancurn · 2026-04-28T09:22:01Z

Great stuff. Pls can we move Academy and Legal docs to the end? It's less relevant for agents, and this way we increase the chance they will see the important bits

Also, this part (the heading) looks bit strange:

Let's make the "Source:" something else than H1, and improve the formatting

jancurn · 2026-04-28T09:24:00Z

Also, please improve the section captions, e.g. ## sdk => ## Apify SDK, they look bit lame now

TC-MO · 2026-04-28T09:34:01Z

Are legal docs relevant for agents at all? Maybe we could trim them completely? 🤔

jancurn · 2026-04-28T09:40:35Z

Are legal docs relevant for agents at all?

Not for the agents, but definitely for the companies running those agents. But let's sort the sections by importance, so that if agents trim the text, the important stuff is kept

- Add includeOrder to put Platform docs and API before Academy/Legal - Fix curated file: replace H1 headings with H2, move source URLs to blockquotes, improve section names (e.g. "## sdk" -> "## Apify SDK for Python") Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Move section reordering from the Docusaurus plugin (includeOrder) into joinLlmsFiles.mjs so it can control the final order after external curated content is appended. Academy and Legal are now at the very end of the file, after all SDK/client/CLI sections. Section order: Platform -> API -> JS Client -> Python Client -> JS SDK -> Python SDK -> CLI -> Academy -> Legal Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

apify-service-account · 2026-04-28T09:58:52Z

Preview for this PR was built for commit aa09e428 and is ready at https://pr-2470.preview.docs.apify.com!

TC-MO

LGTM

jancurn · 2026-04-28T10:30:53Z

One last nit - can we add the following intro to the very top?

Let's add there:

Apify is the largest marketplace of tools for AI. 25,000 ready-made Actors to automate your business. Get real-time web data, track competitors, generate leads, analyze sentiment, and orchestrate your apps. Actors are created by a global community of builders earning over $1M every month. Apify takes care of infrastructure, billing, and distribution.

And also, make sure we include somewhere link to https://apify.com/pricing.md as that's quite important too

jancurn · 2026-04-28T10:31:42Z

Small bug here - Unicode emojis are not rendered correctly:

jancurn · 2026-04-28T10:32:25Z

And this part is also weird:

jancurn · 2026-04-28T10:32:54Z

And missing newline here:

jancurn · 2026-04-28T10:48:30Z

Oh and one more - can we add an integration test that ensures the llms.txt is below let's say 90k chars?

- Add Apify marketing blurb and pricing link to siteDescription - Remove Source blockquotes from curated file (not useful for LLMs) - Normalize list indentation to consistent 2-space nesting Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

apify-service-account · 2026-04-28T11:22:14Z

Preview for this PR was built for commit 029b5d2a and is ready at https://pr-2470.preview.docs.apify.com!

marcel-rbro · 2026-04-28T11:22:24Z

I've resolved the nits but the test is a bigger change (also, purely vibecoded). Can you please review the integration test, @webrdaniel?

Add scripts/checkLlmsSize.mjs that verifies both llms.txt and llms-full.txt exist after build and checks llms.txt character count: - Warning if over 90,000 characters - Error (exit 1) if over 100,000 characters Runs in CI after build via npm run test:llms-size. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

apify-service-account · 2026-04-28T11:29:43Z

Preview for this PR was built for commit 44025b03 and is ready at https://pr-2470.preview.docs.apify.com!

webrdaniel

looks good

jancurn · 2026-04-28T11:38:23Z

Thanks. Two more nits and it will be perfect :)

The Changelog probably shouldn't be the parent of the sub-items

One more missing new-line before this heading:

- Move Changelog to end of each section's sub-items so the indent script doesn't make concept/guide pages children of Changelog - Reorder entries to put Quick start first in each section - Fix reorderSections to ensure blank line between all ## sections Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

apify-service-account · 2026-04-28T11:50:43Z

Preview for this PR was built for commit 0ce18ce3 and is ready at https://pr-2470.preview.docs.apify.com!

marcel-rbro · 2026-04-28T11:54:36Z

🚀

webrdaniel · 2026-04-29T14:08:23Z

+                        '/legal/candidate-referral-program-terms',
+                        // Misc singleton pages
+                        '/open-source',
+                        '/sdk',


@marcel-rbro @TC-MO By including the pages here, the markdown variants themselves were removed. (e.g. https://docs.apify.com/sdk.md now returns 404)
I just want to verify if this ok and if it was intended.

Good catch. Definitely not okay, all pages need to have their Markdown counterpart (+ .md) working. Can we add integration tests to ensure these pages work

The tests are not checking the homepages of the child repos, but the setup is there already:

https://github.com/apify/apify-docs/actions/runs/25109134352/workflow#L67

This could fix it: #2480

github-actions Bot assigned marcel-rbro Apr 28, 2026

github-actions Bot added the t-docs Issues owned by technical writing team. label Apr 28, 2026

marcel-rbro marked this pull request as ready for review April 28, 2026 08:11

marcel-rbro requested a review from TC-MO April 28, 2026 08:19

TC-MO requested a review from webrdaniel April 28, 2026 08:49

marcel-rbro and others added 2 commits April 28, 2026 11:52

webrdaniel approved these changes Apr 28, 2026

View reviewed changes

TC-MO approved these changes Apr 28, 2026

View reviewed changes

marcel-rbro requested a review from webrdaniel April 28, 2026 11:21

webrdaniel approved these changes Apr 28, 2026

View reviewed changes

marcel-rbro merged commit d01804b into master Apr 28, 2026
15 checks passed

marcel-rbro deleted the feat/reduce-llms-txt-size branch April 28, 2026 11:54

webrdaniel reviewed Apr 29, 2026

View reviewed changes

webrdaniel mentioned this pull request Apr 29, 2026

docs: omit .md pages from llms.txt without removing them completely #2480

Open

4 tasks

Conversation

marcel-rbro commented Apr 28, 2026

Summary

Test plan

Uh oh!

apify-service-account commented Apr 28, 2026

Uh oh!

jancurn commented Apr 28, 2026

Uh oh!

jancurn commented Apr 28, 2026

Uh oh!

TC-MO commented Apr 28, 2026

Uh oh!

jancurn commented Apr 28, 2026

Uh oh!

apify-service-account commented Apr 28, 2026

Uh oh!

TC-MO left a comment

Choose a reason for hiding this comment

Uh oh!

jancurn commented Apr 28, 2026

Uh oh!

jancurn commented Apr 28, 2026

Uh oh!

jancurn commented Apr 28, 2026

Uh oh!

jancurn commented Apr 28, 2026

Uh oh!

jancurn commented Apr 28, 2026

Uh oh!

apify-service-account commented Apr 28, 2026

Uh oh!

marcel-rbro commented Apr 28, 2026

Uh oh!

apify-service-account commented Apr 28, 2026

Uh oh!

webrdaniel left a comment

Choose a reason for hiding this comment

Uh oh!

jancurn commented Apr 28, 2026

Uh oh!

apify-service-account commented Apr 28, 2026

Uh oh!

Uh oh!

marcel-rbro commented Apr 28, 2026

Uh oh!

webrdaniel Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

jancurn Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

B4nan Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

webrdaniel Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants