ValidationError: 1 validation error for LlamaCpp n_gpu_layers extra fields not permitted (type=value_error.extra)

[neeewwww]

.env

Generic

TEXT_EMBEDDINGS_MODEL=sentence-transformers/all-MiniLM-L6-v2
TEXT_EMBEDDINGS_MODEL_TYPE=HF # LlamaCpp or HF
USE_MLOCK=true

Ingestion

PERSIST_DIRECTORY=db
DOCUMENTS_DIRECTORY=source_documents
INGEST_CHUNK_SIZE=500
INGEST_CHUNK_OVERLAP=50

Generation

MODEL_TYPE=LlamaCpp # GPT4All or LlamaCpp
MODEL_PATH=eachadea/ggml-vicuna-7b-1.1/ggml-vic7b-q5_1.bin
MODEL_TEMP=0.8
MODEL_N_CTX=1024 # Max total size of prompt+answer
MODEL_MAX_TOKENS=256 # Max size of answer
MODEL_STOP=[STOP]
CHAIN_TYPE=betterstuff
N_RETRIEVE_DOCUMENTS=100 # How many documents to retrieve from the db
N_FORWARD_DOCUMENTS=100 # How many documents to forward to the LLM, chosen among those retrieved
N_GPU_LAYERS=4

Python version

3.11.3

System

Windows 11

CASALIOY version

Main

Information

• The official example scripts
• My own modified scripts

Related Components

• Document ingestion
• GUI
• Prompt answering

Reproduction

/

D:\120hz\CASALIOY>python casalioy/ingest.py
found local model dir at models\sentence-transformers\all-MiniLM-L6-v2
found local model file at models\eachadea\ggml-vicuna-7b-1.1\ggml-vic7b-q5_1.bin

Delete current database?(Y/N): y
Deleting db...
Scanning files
Processing ABRACEEL_process_230519.pdf
Processing 89 chunks
Creating a new collection, size=384
Saving 89 chunks
Saved, the collection now holds 89 documents.
Processed ABRACEEL_process_230519.pdf
100.0% [=======================================================================================>] 1/ 1 eta [00:00]
Done

D:\120hz\CASALIOY>python casalioy/startLLM.py
found local model dir at models\sentence-transformers\all-MiniLM-L6-v2
found local model file at models\eachadea\ggml-vicuna-7b-1.1\ggml-vic7b-q5_1.bin
llama.cpp: loading model from models\eachadea\ggml-vicuna-7b-1.1\ggml-vic7b-q5_1.bin
llama_model_load_internal: format = ggjt v2 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 1024
llama_model_load_internal: n_embd = 4096
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 32
llama_model_load_internal: n_layer = 32
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 9 (mostly Q5_1)
llama_model_load_internal: n_ff = 11008
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 72.75 KB
llama_model_load_internal: mem required = 6612.59 MB (+ 1026.00 MB per state)
....................................................................................................
llama_init_from_file: kv self size = 512.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ D:\120hz\CASALIOY\casalioy\startLLM.py:135 in │
│ │
│ 132 │
│ 133 │
│ 134 if name == "main": │
│ ❱ 135 │ main() │
│ 136 │
│ │
│ D:\120hz\CASALIOY\casalioy\startLLM.py:123 in main │
│ │
│ 120 # noinspection PyMissingOrEmptyDocstring │
│ 121 def main() -> None: │
│ 122 │ session = PromptSession(auto_suggest=AutoSuggestFromHistory()) │
│ ❱ 123 │ qa_system = QASystem(get_embedding_model()[0], persist_directory, model_path, model_ │
│ 124 │ while True: │
│ 125 │ │ query = prompt_HTML(session, "\nEnter a query: ").strip() │
│ 126 │ │ if query == "exit": │
│ │
│ D:\120hz\CASALIOY\casalioy\startLLM.py:57 in init │
│ │
│ 54 │ │ │ case "LlamaCpp": │
│ 55 │ │ │ │ from langchain.llms import LlamaCpp │
│ 56 │ │ │ │ │
│ ❱ 57 │ │ │ │ llm = LlamaCpp( │
│ 58 │ │ │ │ │ model_path=model_path, │
│ 59 │ │ │ │ │ n_ctx=n_ctx, │
│ 60 │ │ │ │ │ temperature=model_temp, │
│ │
│ in pydantic.main.BaseModel.init:341 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValidationError: 1 validation error for LlamaCpp
n_gpu_layers
extra fields not permitted (type=value_error.extra)

Expected behavior

Question Prompt.

[hippalectryon-0]

CASALIOY version
Main

Can you confirm that you're on commit f29182b ? It sounds like you're on a previous release that didn't include the n_gpu_layers arg

[neeewwww]

Yep, I'm using the latest files. Just confirmed.

[hippalectryon-0]

Did you also rerun poetry install ? What's the output of poetry show ?

[neeewwww]

Rerun the "Build from Source" procedures solves the problem.
Question: Each new version need to re-run the poetry?

[hippalectryon-0]

Each time we update the requirements, you need to run "poetry install" to update them.

We'll add a section to the README.