running gpt4 all models

[abcnow]

.env

Generic

TEXT_EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2
TEXT_EMBEDDINGS_MODEL_TYPE=HF # LlamaCpp or HF
USE_MLOCK=false

Ingestion

PERSIST_DIRECTORY=db
DOCUMENTS_DIRECTORY=source_documents
INGEST_CHUNK_SIZE=500
INGEST_CHUNK_OVERLAP=50
INGEST_N_THREADS=4

Generation

MODEL_TYPE=GPT4All # GPT4All or LlamaCpp
MODEL_PATH=thebloke/ggml-gpt4all-j-v1.3-groovy.bin
MODEL_TEMP=0.8
MODEL_N_CTX=1024 # Max total size of prompt+answer
MODEL_MAX_TOKENS=256 # Max size of answer
MODEL_STOP=[STOP]
CHAIN_TYPE=betterstuff
N_RETRIEVE_DOCUMENTS=100 # How many documents to retrieve from the db
N_FORWARD_DOCUMENTS=100 # How many documents to forward to the LLM, chosen among those retrieved
N_GPU_LAYERS=4

Python version

3.11.3

System

OS Monterey version 12.6.8

CASALIOY version

latest

Information

• The official example scripts
• My own modified scripts

Related Components

• Document ingestion
• GUI
• Prompt answering

Reproduction

I switched to gpt4all model since the vicuna's answer were below average (with the docs you have there by default).. and it is giving me this error:
declare -x PWD="/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main"
declare -x SDKMAN_CANDIDATES_API="https://api.sdkman.io/2"
declare -x SDKMAN_CANDIDATES_DIR="/Users/ac/.sdkman/candidates"
declare -x SDKMAN_DIR="/Users/ac/.sdkman"
declare -x SDKMAN_PLATFORM="Darwin"
declare -x SDKMAN_VERSION="5.7.2+323"
declare -x SHELL="/bin/bash"
declare -x SHLVL="2"
declare -x SSH_AUTH_SOCK="/private/tmp/com.apple.launchd.OyMPfNIIOU/Listeners"
declare -x TERM="xterm-256color"
declare -x TERM_PROGRAM="vscode"
declare -x TERM_PROGRAM_VERSION="1.81.1"
declare -x TMPDIR="/var/folders/bt/wyx4_3gx33q2cj3b_frrbz780000gn/T/"
declare -x USER="ac"
declare -x VSCODE_GIT_ASKPASS_EXTRA_ARGS="--ms-enable-electron-run-as-node"
declare -x VSCODE_GIT_ASKPASS_MAIN="/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/git/dist/askpass-main.js"
declare -x VSCODE_GIT_ASKPASS_NODE="/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)"
declare -x VSCODE_GIT_IPC_HANDLE="/var/folders/bt/wyx4_3gx33q2cj3b_frrbz780000gn/T/vscode-git-8d97e97cde.sock"
declare -x VSCODE_INJECTION="1"
declare -x VSCODE_NONCE="890b781c-7725-4872-be2f-ec2096aa5f39"
declare -x VSCODE_SHELL_LOGIN="1"
declare -x XPC_FLAGS="0x0"
declare -x XPC_SERVICE_NAME="application.com.microsoft.VSCode.13059730707.13059730714"
declare -x _CE_CONDA=""
declare -x _CE_M=""
declare -x __CFBundleIdentifier="com.microsoft.VSCode"
declare -x __CF_USER_TEXT_ENCODING="0x1F5:0x0:0x0"

The default interactive shell is now zsh.
To update your account to use zsh, please run chsh -s /bin/zsh.
For more details, please visit https://support.apple.com/kb/HT208050.
(base)
ac at acs-iMac in ~/Documents/software/ai/casalioy/test0/CASALIOY-main
$ /Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/.venv/bin/python -u "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py"
found local model dir at models/sentence-transformers/all-MiniLM-L6-v2
found local model file at models/thebloke/ggml-gpt4all-j-v1.3-groovy.bin
gptj_model_load: loading model from 'models/thebloke/ggml-gpt4all-j-v1.3-groovy.bin' - please wait ...
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx = 2048
gptj_model_load: n_embd = 4096
gptj_model_load: n_head = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot = 64
gptj_model_load: f16 = 2
gptj_model_load: ggml ctx size = 4505.45 MB
gptj_model_load: memory_size = 896.00 MB, n_mem = 57344
gptj_model_load: ................................... done
gptj_model_load: model size = 3609.38 MB / num tensors = 285

Enter a query: source /Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/.venv/bi
n/activate
Traceback (most recent call last):
File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py", line 135, in
main()
File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py", line 131, in main
qa_system.prompt_once(query)
File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/startLLM.py", line 105, in prompt_once
res = self.qa(query)
^^^^^^^^^^^^^^
File "/Users/ac/Documents/software/ai/casalioy/test0/CASALIOY-main/casalioy/CustomChains.py", line 80, in call
> model_n_ctx - self.llm.dict()["max_tokens"]
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'max_tokens'
(base)

I am not sure if you have a particular model in mind that could work for gpt4all implementation.. thanks in advance!

Expected behavior

program it is stopping right there...it should answer the question.. it is just stopping the process...

[su77ungr]

I would recommend sticking with llamacpp. The ReadMe states a model that's over six months in age and was just a test model. The quality of models improved incredible since this repo was created. I have to check compatibility of the LlamaCpp version to run Llama2 models

Nonetheless the last time I tested it, it was running with ggml-gpt4all-j-v1.3-groovy.bin