Add support for generating llms-full.txt #146
Merged
+119
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the generation of the llms-full.txt file, which contains all Defold documentation in a single page in the Markdown format. (almost all of it - the next huge step is to add /ref/ to the file)
So:
update.py
a step is added to process temporary files to the_llms
folder_llms
folder is added to gitignore and ignored by Jekyll because of the underscore. This path is used because it is convenient to see the result of the script how the files are converted.update.py
the files are assembled into one bigllms-full.txt
. In this file all references are processed so that they are absolute. This is necessary so that the file can be viewed outside the context of the site, as one large convenient document.This PR only adds
llms-full.txt
, withoutllms.txt
(which is actually a collection of links and requires all pages on the site to have .md format as well, in addition to the finished html). LLM is evolving very quickly and from my research everyone is now using one bigllms-full.txt
file at once for query context for AI.The format of
llms-full.txt
is based on https://duckdb.org/duckdb-docs.md, as the best reference for that filetype.Fixes defold/doc#522