Weighted Field-Filtering

With this release comes weighted field-filtering matching, and a bit of context token budget management. For those using this tool for story telling it will shine.

By default the Context Manager will heavily bias a search (75%) along the entity field (NPCs, Names, Protagonist, etc) of your allotted match count. Furthermore, if there are multiple entities, it will balance returned matches for each of them. (15 matches allowed @%75 = 11 matches reserved for entity field-filtering. 11/2 = 5 for Jane, 5 for John, etc).

Dropped nltk in favor of using a built-in from difflib import SequenceMatcher for performing similar sentence de-duplication. At 15 maximum RAG result matches, it is impressive to see a total of 16K tokens found, while eliminating 80% of them afterwards (~3200 remaining unique tokens), meaning the LLM was certainly given pertinent information without bulge. This helps the LLM stay focused with its system prompt and not be overwhelmed by history context.

Added several methods folks can use to pre-populate their RAGs. The process now filters the chunks properly through the pre-processor, thus properly tagging the chunks for field-filtering matching. The process is of course slower, but the end results will be much more accurate.

  --import-pdf         Path to pdf to pre-populate main RAG
  --import-txt         Path to txt to pre-populate main RAG
  --import-web         URL to pre-populate main RAG
  --import-dir         Path to recursively find and import assorted files (*.md *.html, *.txt)

when encountering .html files, importing will use beautifulsoup4 to extract plain text