WIP Inference program to interact with large language models of the GGUF format. Requirements: A working installation of llama-cpp-python, ChromaDB and NRCLex is needed. Mistral models of the GGUF format found on Huggingface currently appear to be broken, the model files need to be downloaded, turned into GGUF and quantized locally.
This code is licensed under a AGPL 3.0 (and later) license