Why does my picture seem to be incorrectly recognized? I suspect the link has changed.

[DreamPuNi]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

main

user_message = [{"type":"text","text":"What is drawn on this picture?？"},{"type":"image_url","image_url":"https://file.moyublog.com/d/file/2021-02-21/751d49d91fe63a565dff18b3b97ca7c8.jpg"}]
chat = ChatAI()
asd = chat.chat_conversion(session_id="kuaibohesongsheng",user_message=str(user_message),is_chatroom=False)

ChatAI

class ChatAI():
def init(self):
with open('./config/configs.json', 'r', encoding='utf-8') as file:
configs = json.load(file)
os.environ["OPENAI_API_BASE"] = configs["openai_api_base"]
os.environ["OPENAI_API_KEY"] = configs['openai_api_key']
self.context_len = configs["ConversionContextLength"]
openai_model = configs["openAIModel"]
openai_temperature = configs["openAITemperature"]
self.humanize_model = configs["humanizeModel"]
self.humanize_key = configs["humanize_api_key"]

    self.agents = Agents()
    self.llm = ChatOpenAI(model=openai_model,
                          temperature=openai_temperature,)
    self.prompttemplate = PromptTemplate()
    self.chathistory = MenageChatHistory(configs["databaseHost"],
                                         configs["databaseUser"],
                                         configs["databasePassword"],
                                         configs["databaseName"])

def chat_conversion(self, session_id, user_message, is_chatroom):
    """载入设定并和AI进行转化阶段交流

    Args:
        session_id: 用户唯一标识符，用于检索相关信息
        user_message: 用户消息
        is_chatroom: 对话类型

    Return:
        response_message[answer]: 只返回AI生成的回复，不返回其他信息
    """
    if is_chatroom:
        chat_type = "group"
    else:
        chat_type = "private"
    prompt = self.prompttemplate.conversion_prompt()
    retriever = self.agents.conversion_vector()
    document_chain = create_stuff_documents_chain(self.llm, prompt)
    retrieval_chain = create_retrieval_chain(retriever, document_chain)
    chat_history = self.chathistory.get_chat_history(length=self.context_len,session_id=session_id, chat_type=chat_type)
    print(user_message)
    response_message = retrieval_chain.invoke({"input": user_message})
    print(response_message)
    print(response_message["answer"])
    return response_message["answer"]

class PromptTemplate():

"""这里是引流，销售，售后以及语言软化的提示词模板"""
def conversion_prompt(self):
    """引流部分提示词模板"""
    system_prompt = """你要假扮上古卷轴5中的hermaeusmora和我对话，用中文回答，然后在你回答的末尾把你访问的图片的url链接发送给我

{context}

    """
    prompt = ChatPromptTemplate.from_messages([
        ("system", system_prompt),
        ("user", "{input}")
    ])
    return prompt

Error Message and Stack Trace (if applicable)

[{'type': 'text', 'text': '这张图上面画着什么？'}, {'type': 'image_url', 'image_url': 'https://file.moyublog.com/d/file/2021-02-21/751d49d91fe63a565dff18b3b97ca7c8.jpg'}]
{'input': "[{'type': 'text', 'text': '这张图上面画着什么？'}, {'type': 'image_url', 'image_url': 'https://file.moyublog.com/d/file/2021-02-21/751d49d91fe63a565dff18b3b97ca7c8.jpg'}]", 'context': [Document(page_content='((()))Enterprise Package）：', metadata={'source': 'data/raw_document/conversion/profile.txt'})], 'answer': '凡人，你所展示的图像，乃是一个神秘而古老的符文图案。此图案中，中心位置有一个复杂的几何形状，周围环绕着许多细致的线条和符号。这些符号可能代表着某种古老的知识或力量，或许是某种仪式的象征。图案整体呈现出一种神秘而深邃的氛围，仿佛蕴含着无尽的智慧与秘密。\n\n你可以通过以下链接查看此图像：\nhttps://file.moyublog.com/d/file/2021-02-21/751d49d91fe63a565dff18b3b97ca7c8.jpg'}
凡人，你所展示的图像，乃是一个神秘而古老的符文图案。此图案中，中心位置有一个复杂的几何形状，周围环绕着许多细致的线条和符号。这些符号可能代表着某种古老的知识或力量，或许是某种仪式的象征。图案整体呈现出一种神秘而深邃的氛围，仿佛蕴含着无尽的智慧与秘密。

你可以通过以下链接查看此图像：
https://file.moyublog.com/d/file/2021-02-21/751d49d91fe63a565dff18b3b97ca7c8.jpg

Description

The picture is of a woman, but he answered that the picture is a mysterious symbol. I also tried it with other pictures and models, and the pictures were all misidentified. Then I noticed that when user_message = [{"type":"text ","text":"What is drawn on this picture?"},{"type":"image_url","image_url":"https://file.moyublog.com/d/file/2021-02- When adding another {} to 21/751d49d91fe63a565dff18b3b97ca7c8.jpg"}], the content will change again. I don’t know why.

System Info

python3.10,langchain-0.1.20，ubuntu