How can I configure the final prompt of ConversationalRetrievalChain?

[matthiasroder]

I am using a ConversationalRetrievalChain and would like to change the final prompt of the chain. How would I go about that?

I understand that the ConversationalRetrievalChain calls the StuffDocumentChain at some point, which collates documents from the retriever. This context ist then passed to an LLMChain for generating the final answer. It is the prompt of this last step that I want to change.

I found out, that the prompt for that step is located here: langchain/libs/langchain/langchain/chains/question_answering/stuff_prompt.py

If I change that prompt in the source code I get exactly what I want.

I am trying to understand however, how I can pass the prompt I need as an argument to ConversationalRetrievalCHain in my python code without changing the source code of langchain.

Thank you very much for any help with this!

[dosubot[bot]]

🤖

Hello,

Based on the information you provided and the context from the LangChain repository, there are a couple of ways you can change the final prompt of the ConversationalRetrievalChain without modifying the LangChain source code.

1. You can change the main prompt in ConversationalRetrievalChain by passing it in via combine_docs_chain_kwargs if you instantiate the chain using from_llm. Here is how you can do it:

ConversationalRetrievalChain.from_llm(combine_docs_chain_kwargs={"prompt": your_prompt}))

2. Alternatively, you can use load_qa_chain with memory and a custom prompt. Here is an example:

from langchain.chains.question_answering import load_qa_chain
template = """
  {Your_Prompt}
  
  CONTEXT:
  {context}
  
  QUESTION: 
  {query}

  CHAT HISTORY: 
  {chat_history}
  
  ANSWER:
  """

prompt = PromptTemplate(input_variables=["chat_history", "query", "context"], template=template)
memory = ConversationBufferMemory(memory_key="chat_history", input_key="query")

chain = load_qa_chain(ChatOpenAI(temperature=0), chain_type="stuff", memory=memory, prompt=prompt)

3. Another way is to create the ConversationalRetrievalChain without the combine_docs_chain_kwargs and memory parameters. Then, manually set the SystemMessagePromptTemplate for the llm_chain in the combine_docs_chain of the ConversationalRetrievalChain:

qa = ConversationalRetrievalChain.from_llm(
    llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3}),
    return_source_documents=True,
    verbose=True,
    condense_question_llm = llm,
    chain_type="stuff",
    get_chat_history=lambda h : h,
)
qa.combine_docs_chain.llm_chain.prompt.messages[0] = SystemMessagePromptTemplate.from_template(sys_prompt)

Define your own sys_prompt, set the context and try.

Please note that these are general examples and might not work as is, because the actual implementation of the ConversationalRetrievalChain class is not provided in the context.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• Issue: Changing Prompt (from Default) when Using ConversationalRetrievalChain?
• ValidationError: 1 validation error for ConversationalRetrievalChain chain_type_kwargs extra fields not permitted (type=value_error.extra)
• libs/langchain/langchain/chains/retrieval_qa/prompt.py
• libs/langchain/langchain/chains/conversational_retrieval/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nithinsikinam]

Hi

i am trying to add system prompt to my existing ConversationalRetrievelChain.fromLLm

memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
conversation_chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    retriever=vectorstore.as_retriever(),
    memory=memory,
)

how do i do this

[muke5hy]

@dosu-bot Can you give some example with Custom system prompt for ConversationalRetrievalChain with chat history

[DanQuin]

@nithinsikinam In my case using option 3 I got what I needed (I'm using langchain==0.0.310), my prompt is something like this:

custom_template = """Given the following conversation and a follow-up message, \
rephrase the follow-up message to a stand-alone question or instruction that \
represents the user's intent, add all context needed if necessary to generate a complete and \
unambiguous question or instruction, only based on the history, don't make up messages. \
Maintain the same language as the follow up input message.

Chat History:
{chat_history}

Follow Up Input: {question}
Standalone question or instruction:"""

        qa = ConversationalRetrievalChain.from_llm(
            llm=llm,
            verbose=verbose,
            memory=memory,
            return_generated_question=True,
            retriever=retriever,
            condense_question_prompt=PromptTemplate.from_template(custom_template),
        )

I hope I have helped you.

[muke5hy]

Not sure what I am doing wrong here but I got an error

def get_answer(question, chat_history=[]):
    try:
        embeddings = get_embeddings()

        CONNECTION_STRING=os.environ["DATABASE_URL"]
        llm = get_llm()
        vectorstore = PGVector(
            collection_name=settings.COLLECTION_NAME,
            connection_string=CONNECTION_STRING,
            embedding_function=embeddings,
        )

        retriever = vectorstore.as_retriever(search_type="similarity", search_kwargs={"k":2})

        custom_template = """Given the following conversation and a follow-up message, \
        rephrase the follow-up message to a stand-alone question or instruction that \
        represents the user's intent, add context if necessary to generate a complete and \
        unambiguous question or instruction, only based on the history, don't make up messages. \
        Maintain the same language as the follow up input message.

        Chat History:
        {chat_history}

        Follow Up Input: {question}
        Standalone question or instruction:"""

        sys_prompt = PromptTemplate(
            template=custom_template, input_variables=["chat_history", "question"]
        )

        chain = ConversationalRetrievalChain.from_llm(
            llm=llm,
            retriever=retriever,
        )
        chain.combine_docs_chain.llm_chain.prompt.messages[0] = SystemMessagePromptTemplate.from_template(sys_prompt)
        result = chain({"question": question, "chat_history": chat_history})

    except Exception as e:
        raise Exception(e)

    return result["answer"]

The error is

Exception: expected str, got PromptTemplate

Which is occurring for following line

chain.combine_docs_chain.llm_chain.prompt.messages[0] = SystemMessagePromptTemplate.from_template(sys_prompt)

[DanQuin]

Sorry for the confusion @muke5hy, I've now updated my last post and included the langchain version.
I hope I've helped.

[mbkamran]

@mdsaifk I've tried your solution. It works fine, but there is a bit of problem. It tends to iterate back in the answer multiple times therefore producing multiple answers and questions. Refer to the picture below:

Do you know why this is happening?

[mbkamran]

@mdsaifk I have a feeling that context is not being passed properly

[mdsaifk]

yes that could but i have created new approach for handeling chat hiatory with langchain is refining our with new prompt which will look to previous chat crete one related query you can try that
this is using gemini

def refined_query(chat_history ,user_question):
generation_config = {
"temperature": 0,
"top_p": 1,
"top_k": 1,
"max_output_tokens": 256,
}

  model = genai.GenerativeModel(model_name="gemini-pro",
                                generation_config=generation_config)

prompt1_1 = f"Given the following last conversation log, if user question is related to previous user question then formulate a question that would be the most relevant to provide the user with an answer otherwise don't formulate question leave it as it is.\n\nCONVERSATION LOG: \n{chat_history}\n\nuser question: {user_question}\n\nRefined user question :"
  
  prompt_parts = [prompt1_1,]
  response = model.generate_content(prompt_parts)
  return response.text