Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for generating llms-full.txt #146

Merged
merged 1 commit into from
Mar 29, 2025

Conversation

aglitchman
Copy link
Contributor

@aglitchman aglitchman commented Mar 27, 2025

This PR adds the generation of the llms-full.txt file, which contains all Defold documentation in a single page in the Markdown format. (almost all of it - the next huge step is to add /ref/ to the file)

So:

  1. in update.py a step is added to process temporary files to the _llms folder
  2. The _llms folder is added to gitignore and ignored by Jekyll because of the underscore. This path is used because it is convenient to see the result of the script how the files are converted.
  3. in update.py the files are assembled into one big llms-full.txt. In this file all references are processed so that they are absolute. This is necessary so that the file can be viewed outside the context of the site, as one large convenient document.

This PR only adds llms-full.txt, without llms.txt (which is actually a collection of links and requires all pages on the site to have .md format as well, in addition to the finished html). LLM is evolving very quickly and from my research everyone is now using one big llms-full.txt file at once for query context for AI.

The format of llms-full.txt is based on https://duckdb.org/duckdb-docs.md, as the best reference for that filetype.

Fixes defold/doc#522

@aglitchman
Copy link
Contributor Author

One question - do I need to add llms-full.txt to the commit or will CI do it automatically?

@aglitchman
Copy link
Contributor Author

aglitchman commented Mar 27, 2025

Preview of the file to test it if anyone is curious - llms-full.txt

230376 tokens.

@britzl
Copy link
Collaborator

britzl commented Mar 28, 2025

One question - do I need to add llms-full.txt to the commit or will CI do it automatically?

There's a step commit_changes() which should commit any changes

@britzl
Copy link
Collaborator

britzl commented Mar 28, 2025

almost all of it - the next huge step is to add /ref/ to the file

Will you submit another PR for this or should the review wait?

@aglitchman
Copy link
Contributor Author

Will you submit another PR for this or should the review wait?

I think it's okay to accept this one. I'd like to get feedback from the community - useful or not. Next, in another PR, I will add the API from /ref/ as well.

For me it's already useful, i.e. AI started to write a bit more clear code.

@britzl britzl merged commit 1ed8b31 into defold:master Mar 29, 2025
@aglitchman
Copy link
Contributor Author

There's a step commit_changes() which should commit any changes

Aha. The PR merge didn't trigger the CI, only trigged github pages rebuild, so there is no file llms-full.txt in the repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add llms-full.txt file to help LLM systems better understand and utilize docs
2 participants