-
Notifications
You must be signed in to change notification settings - Fork 363
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New implementation of locally hosted paper-qa according to the README causes a pydantic error #243
Comments
I seem to have the same issue with Gemini: it looks like the pydantic setting wants there to be a 'name' field in both LLamaEmbeddingModel and GoogleGenerativeAIEmbeddings(). Fuller error: My Docs():
This is what
And the error:
Edit: Line 159 in 350225c
data['embedding'] is assumed to be a string, not any model. This code kind of works:
This will use the local SentenceTransformerEmbeddingModel() instead of Gemini. Then I run into the next error as it assumes that llm_model is wrapped within |
Sorry about that - yes Use this syntax instead for passing a model (I updated the README - thanks) from paperqa import Docs, LlamaEmbeddingModel, OpenAILLMModel
from openai import AsyncOpenAI
# start llamap.cpp client with
local_client = AsyncOpenAI(
base_url="http://localhost:8080/v1",
api_key = "sk-no-key-required"
)
docs = Docs(client=local_client,
embedding_model=LlamaEmbeddingModel(),
llm_model=OpenAILLMModel(config=dict(model="my-llm-model", temperature=0.1, frequency_penalty=1.5, max_tokens=512))) For @philippbayer - your example would be using Langchain (I assume!): docs = Docs(llm = "langhcain",
client = ChatGoogleGenerativeAI(model='gemini-pro'),
embedding = "langchain",
embedding_client = GoogleGenerativeAIEmbeddings(model="models/embedding-001")) |
This code (and in the readme)
still generates this error: pydantic_core._pydantic_core.ValidationError: 1 validation error for LlamaEmbeddingModel |
I think the problem in both cases is that LlamaEmbeddingModel() requires a name argument - its that instantiation that's complaining, not Docs() |
also embedding_model isn't a valid argument for Docs
|
embedding_client perhaps instead?
|
That fixed the pydantic issue, but revealed another issue OpenAI API gets embeddings via POST to this endpoint '/embeddings' (https://platform.openai.com/docs/api-reference/embeddings) But the locally hosted llamafile server gets embeddings via this endpoint '/embedding' (https://github.com/Mozilla-Ocho/llamafile/blob/main/llama.cpp/server/README.md#api-endpoints) This results in a File Not Found API error when trying to add documents and embed text on a locally hosted LLM. Llamafile is currently on version 0.6.2 which was last synchronized with llama.cpp on 1-27-2024. However the llama.cpp commit that fixes this problem wasn't committed until 1-29-2024 (ggerganov/llama.cpp@9461329). So until the next version of llamafile syncs to a llama.cpp sync after that date paperqa is not compatible with a llamafile I believe. |
Hi @alexanderchang1 - maybe try using SentenceTransformer? from paperqa import Docs, OpenAILLMModel, print_callback
docs = Docs(client=client,
embedding="sentence-transformers",
llm_result_callback=print_callback,
llm_model=OpenAILLMModel(config=dict(model="my-llama.cpp-llm", temperature=0.1, max_tokens=512))) or I added a new embedding that is just keyword based: from paperqa import Docs, OpenAILLMModel, print_callback
docs = Docs(client=client,
embedding="sparse",
llm_result_callback=print_callback,
llm_model=OpenAILLMModel(config=dict(model="my-llama.cpp-llm", temperature=0.1, max_tokens=512))) |
Hi @whitead, Still getting the File Not Found error due to /embeddings and /embedding conflicts in llamafile. |
Hi, @whitead i got it to work with the following method. llamafile servers are currently incompatible and result in File Note Found error due to endpoint differences. Instead, a user has to install the latest llama-cpp-python bindings for web server (https://github.com/abetlen/llama-cpp-python#web-server), and then run the command locally.
Then you can run it locally via
And the remainder of the code is the same, however the performance drops a lot using sentence transformers. Is there an updated version on how to use LlamaEmbeddingModel instead? Or any other model, my hope is to eventually use mistral 8x7B. |
The text was updated successfully, but these errors were encountered: