No input for the question: "Enter a query:" NEVER shows

[AngelTs]

C:\privateGPT-main>python ingest.py
Loading documents from source_documents
Loaded 1 documents from source_documents
Split into 90 chunks of text (max. 500 characters each)
Using embedded DuckDB with persistence: data will be stored in: db

C:\privateGPT-main>python privateGPT.py
Using embedded DuckDB with persistence: data will be stored in: db

C:\privateGPT-main>

[Elicherla01]

Are you using IDE or directly running from command prompt?

[AngelTs]

Are you using IDE or directly running from command prompt?

Directly from command-prompt. LLM model is in "C:\privateGPT-main\models\ggml-gpt4all-j-v1.3-groovy.bin" and these 2 scripts are in "C:\privateGPT-main". The embedding model was automatic download (via 1st start of ingest.py script) in "C:\Users\Angel.cache\torch\sentence_transformers\sentence-transformers_all-MiniLM-L6-v2\pytorch_model.bin"

[AngelTs]

".env" file contains:
PERSIST_DIRECTORY=db
EMBEDDINGS_MODEL_NAME=sentence-transformers/all-MiniLM-L6-v2
MODEL_TYPE=GPT4All
MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin
MODEL_N_CTX=1000

[PulpCattel]

This is a duplicate of #281 and #295

[AngelTs]

Are you using IDE or directly running from command prompt?

When i use the absolute path for embedding model:
EMBEDDINGS_MODEL_NAME=C:\Users\Angel.cache\torch\sentence_transformers\sentence-transformers_all-MiniLM-L6-v2\pytorch_model.bin

the following error occurs:

OSError: It looks like the config file at 'C:\Users\Angel.cache\torch\sentence_transformers\sentence-transformers_all-MiniLM-L6-v2\pytorch_model.bin' is not a valid JSON file.

[PulpCattel]

@AngelTs Your .env file was good before. See the other issues I linked above, it's probably a problem with your hardware, your CPU is too old.

[AngelTs]

@AngelTs Your .env file was good before. See the other issues I linked above, it's probably a problem with your hardware, your CPU is too old.

I started without problems oobabooga with many different models, even with 6B, AUTOMATIC1111, Dolly ...

[PulpCattel]

Maybe they use the GPU or something? Or some online API? In any event, if you think there's still an issue, it's better if you close this one and comment in the one already open to avoid creating unnecessary duplicate issues.

[VGONUL]

I did as followed.

I created another subfolder like models for sentence-transformers,and put all files here.then it worked

PERSIST_DIRECTORY=db
MODEL_TYPE=GPT4All
MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin
EMBEDDINGS_MODEL_NAME=sentence-transformers/all-MiniLM-L6-v2
MODEL_N_CTX=1000

but Im getting response as followed.
I dont know why

Using embedded DuckDB with persistence: data will be stored in: db
gptj_model_load: loading model from 'models/ggml-gpt4all-j-v1.3-groovy.bin' - please wait ...
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx = 2048
gptj_model_load: n_embd = 4096
gptj_model_load: n_head = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot = 64
gptj_model_load: f16 = 2
gptj_model_load: ggml ctx size = 4505.45 MB
gptj_model_load: memory_size = 896.00 MB, n_mem = 57344
gptj_model_load: ................................... done
gptj_model_load: model size = 3609.38 MB / num tensors = 285

Enter a query: can you summarize text with 10 sentences?
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'
gpt_tokenize: unknown token 'Γ'
gpt_tokenize: unknown token 'Ç'
gpt_tokenize: unknown token 'Ö'

[AngelTs]

I receive the error -1073741795, so the problem is in my CPU - it doesn't support AVX and AVX2 instructions.
Three options here:

1.Downgrade to TensorFlow 1.5 - it is almost impossible to work, because this version use old python, so very possible all other installed libraries in the new version to python to be impossible to install on this old version of python!;
2.Installing TensorFlow from Source (Custom Build) with optimization flags, which exclude AVX2 - I can't find custom build under native Windows, so the process to compile 250 MB source code, which is not target to native windows will be VERY funny;
3.Use GPU Llama Index - must have at least 15GB of VRAM.

I select the thie last option and here is my example python script (hope to help to another solo soldier):

from langchain.embeddings.huggingface import HuggingFaceEmbeddings
from langchain.llms import GPT4All
from llama_index.node_parser.simple import SimpleNodeParser
from llama_index.langchain_helpers.text_splitter import TokenTextSplitter
from llama_index import (
GPTVectorStoreIndex,
LangchainEmbedding,
LLMPredictor,
ServiceContext,
StorageContext,
download_loader,
PromptHelper
)
PyMuPDFReader = download_loader("PyMuPDFReader")

from llama_index.prompts.prompts import SimpleInputPrompt

query_wrapper_prompt = SimpleInputPrompt(
"Below is an instruction that describes a task. "
"Write a response that appropriately completes the request.\n\n"
"### Instruction:\n{query_str}\n\n### Response:"
)

import torch
from llama_index.llm_predictor import HuggingFaceLLMPredictor

hf_predictor = HuggingFaceLLMPredictor(
max_input_size=2048,
max_new_tokens=256,
temperature=0.25,
do_sample=False,
query_wrapper_prompt=query_wrapper_prompt,
tokenizer_name="./models/ggml-gpt4all-j-v1.3-groovy.bin",
model_name="./models/ggml-gpt4all-j-v1.3-groovy.bin",
device_map="auto",
tokenizer_kwargs={"max_length": 2048},
model_kwargs={"torch_dtype": torch.bfloat16}
)

embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2"))

service_context = ServiceContext.from_defaults(chunk_size_limit=512, llm_predictor=hf_predictor, embed_model=embed_model)

index = GPTVectorStoreIndex.from_documents(documents, service_context=service_context)
index.storage_context.persist(persist_dir="./source_documents")

query_engine = index.as_query_engine(streaming=True, similarity_top_k=3, service_context=service_context)
response_stream = query_engine.query("What are the main climate risks to our Oceans?")
response_stream.print_response_stream()

---

=>BUT the following error occured:
...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 4: invalid continuation byte
During handling of the above exception, another exception occurred:
...
OSError: It looks like the config file at './models/ggml-gpt4all-j-v1.3-groovy.bin' is not a valid JSON file.

I think it is time to me to flag game over. Cheers!