Understanding the question/answering process and its costs

[bbscout]

Can someone explain to me what the process is behind the scenes when calling the OpenAI API?

I understand how embedding works (#1). But how much text from the embedding is included in following requests? And why there are for example 2 requests for one question or even 5 when using ChatOpenAI?

Example:

I tried simple question (in Czech because my embeddings are in Czech) "How old must the camp leader be at least?". The chain made two API calls with 5565 tokens in total. And the response was "The minimum age for the camp leader is 18 according to Junák – český skaut." It's not very cost effective when using text-davinci. For one simple question I pay around 0,11 USD.

I simply tried replace OpenAI() with ChatOpenAI() which uses gpt-3.5-turbo-0301. The chain made 5 requests (4,643 prompt + 278 completion = 4,921 tokens). The price is 10x lower and also less tokens are used.

chain = VectorDBQAWithSourcesChain.from_llm(llm=ChatOpenAI(temperature=0), vectorstore=store)

Is it possible to affect how long "embeddings" will be included in the request?

Thanks for any information.

[aabdelfattah]

I also made 2 trials which costed 0.8$ (~ 0.4$ per question).
I am wondering about your change, it doesn't seem as a one liner because also more stuff needs to be included . Can you please elaborate how did you replace OpenAI() with ChatOpenAI() ?

[lmsanch]

Exactly the same experience for me.

[sfx101]

Use a tool like Proxyman to get more insights.

And yes, the Langchain code in this particular repo makes two requests, 1 for a kind of summarizing the context from Faiss and the other for generating the answer.
As I said, Proxyman or Charles might be useful to get more insights on costs, tokens etc

[aabdelfattah]

Use a tool like Proxyman to get more insights. And yes, the Langchain code in this particular repo makes two requests, 1 for a kind of summarizing the context from Faiss and the other for generating the answer. As I said, Proxyman or Charles might be useful to get more insights on costs, tokens etc

This is probably out of the context of the main question, but how did you get proxyman to decrypt https

I always get this

Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 8.0 seconds as it raised APIConnectionError: Error communicating with OpenAI: HTTPSConnectionPool(host='api.openai.com', port=443): Max retries exceeded with url:

[aabdelfattah]

I also made 2 trials which costed 0.8$ (~ 0.4$ per question). I am wondering about your change, it doesn't seem as a one liner because also more stuff needs to be included . Can you please elaborate how did you replace OpenAI() with ChatOpenAI() ?

Answering my own question: here is a commit which does the intended replacement

[sfx101]

Here: https://docs.proxyman.io/debug-devices/python

[yixin0829]

Weights and Biases also released it Prompts feature that supports langchain integration. https://docs.wandb.ai/guides/prompts/quickstart

Think it might be really helpful if you wanna try.