when there are huge amount of source files mentioning the same concept, over a period of time OpenKB will continue to append and bloat up the <concept>.md till a size that is unreadable by LLM as it is way higher than context windows limit. This is also true to <entities>.md as well.
Need a feature to auto turn the <concept>.md into the main index for the <concept>-<child-concept>.md structure once it is over the threshold limit , in order to keep the md file size light weight for LLM navigation.
This mechanism should also be applied to the <concept>-<child-concept>.md to spawn off <concept>-<child-concept>-<grandchild-concept>.md.
These spawn off structure files need to be updated in the wiki main index.md and the subsequent openKB add or recompile will auto recognize them and will not overwrite the main <concept>.md index
related: #203
when there are huge amount of source files mentioning the same concept, over a period of time OpenKB will continue to append and bloat up the
<concept>.mdtill a size that is unreadable by LLM as it is way higher than context windows limit. This is also true to<entities>.mdas well.Need a feature to auto turn the
<concept>.mdinto the main index for the<concept>-<child-concept>.mdstructure once it is over the threshold limit , in order to keep the md file size light weight for LLM navigation.This mechanism should also be applied to the
<concept>-<child-concept>.mdto spawn off<concept>-<child-concept>-<grandchild-concept>.md.These spawn off structure files need to be updated in the wiki main
index.mdand the subsequentopenKB addorrecompilewill auto recognize them and will not overwrite the main<concept>.mdindexrelated: #203