Support for llm.txt files

[gusfune]

Goals

Add llm.txt support for dynamic content in similar fashion of sitemap.xml and robots.txt is implemented

Non-Goals

Generate the content for the website based on the site, just stick to a similar format compared to the current similar implementations.

Background

From the proposal format website: https://llmstxt.org/

Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety. Converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise.

While websites serve both human readers and LLMs, the latter benefit from more concise, expert-level information gathered in a single, accessible location. This is particularly important for use cases like development environments, where LLMs need quick access to programming documentation and APIs.

A few companies already adopting the format:

• Strapi
• Vultr
• Langbase
• Checkly
• CircleCI

Proposal

Implement a llm.(js|ts) file that would live in the app directory in similar way we utilise sitemap or robots.txt.

[icyJoseph]

Hi,

Justo echo this entry - #80692 (comment) - I understand your request is more in line with the functionality we have for sitemap, robots etc... but just so that people know there's a way to do it yourself.

Vercel also has https://vercel.com/docs/llms.txt

And since yesterday, so does Next.js https://nextjs.org/docs/llms.txt - I was just about to announce this -

[gusfune]

Absolutely. I believe the "official support" as a functionality is more as a show of support/endorse a standard to push for wider adoption than something that is a technical limitation. I'm also running that in similar fashion. Thanks for the link, did not see that discussion from when I searched the topic.

[icyJoseph]

No worries.

I did some quick searching around regarding llms.txt and found mixed opinions. Though, some big players do have it. In a way it is a low entry barrier. Perhaps it has to become more stablished.

This kind of discussion entry signals that the standard is growing though. At the moment is that some engagement. We'll see what the core team thinks about it.

[josepholiveira]

@icyJoseph

I might be reviving something here, but I think a feature focused on LLMSEO might actually make a lot of sense.

With the llmstxt convention improving day by day, we’re now seeing a growing need to support .html.md pages as well.

For example, today we can do something like this:

/app/blog/tags/[slug]
├─ page.tsx
└─ llms.txt/route.ts

But what we can’t do yet is something like:

/app/blog/tags/[slug]
├─ page.tsx
└─ [slug].html.md

I noticed that Vercel’s docs seem to handle this — adding a .md suffix to the end of files — but I’m curious how that’s achieved under the hood, or if we can achieve that using the routing itself.

[bke-daniel]

Currently, I am working on a plugin for that. It works like the static metadata export. It's production-ready by end of this week. See https://github.com/bke-daniel/next-llms-txt

[josepholiveira]

Whoa, great tool! @bke-daniel 🙌

I was about to open a discussion on your repo, but I noticed it’s not enabled yet.

Do you plan on supporting dynamic routes at some point?

I’m wondering if the only way to achieve that currently would be through rewrites, since we can’t really do what I mentioned here:
#81182 (reply in thread)

[bke-daniel]

@josepholiveira
Thank you! Also thank you for point out the disabled discussions. I am just working on that thing like a maniac and completely overlooked it. It's enabled now. See bke-daniel/next-llms-txt#5

Absolutely, on my book dynamic routes must be supported as well.

I’m wondering if the only way to achieve that currently would be through rewrites, since we can’t really do what I mentioned here: #81182 (reply in thread)

No, I don't think so. I've experimented with different approaches at the beginning. Then I decided to somewhat roll back and serve the initial purpose rather than having a full blown assault on LLMs.txt / *.html.md.
My plan is to work my way forward from here.

[pieterjanscheir]

Any news here, how is one best to implement llms.text in nextjs today?

[gusfune]

Right now you can "drop" a manual llms.txt in the public folder. For dynamic a simple llm/route.ts could do the trick and rewrite that in next.config to be a .txt route (that's how I am doing in a project). I sent a proposal today for incorporating this as a feature, so it's up to the NextJS maintainer team to pick it up

[icyJoseph]

You can name it llms.txt/route.ts though? and no rewrite needed.

[gusfune]

I don't see why wouldn't work, but I avoid putting dots in folder names, at some point it gave me problems with "some" filesystem, so I keep that away just in case. Might be just superstition.

[icyJoseph]

Being defensive in software dev is a safe bet most of the time 👍

[bke-daniel]

Any news here, how is one best to implement llms.text in nextjs today?

My plugin has production readiness and works well on multiples pages.
Feel free to check it out.

[gusfune]

I didn't want to wait further so I made a PR implementing the future directly into NextJS, refer to #90580 for details on the PR progress.