ChatOpenai logprobs not reported despite being set to True in model_kwargs

[emilmirzayev]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.

Example Code

from langchain_openai import ChatOpenAI


llm = ChatOpenAI(temperature=0,
    openai_api_key = my_api_key,
        model_name="gpt-4",
        model_kwargs = {"logprobs": True,
        "top_logprobs":3}) 



llm.invoke("Please categorize this text below into positive, negative or neutral: I had a good day")

# OUTPUT
AIMessage(content='Positive')

Error Message and Stack Trace (if applicable)

No response

Description

Lopgprobs argument is again settable in OpenAi according to this official source (Openai docs):

However, when I try to use it via langchain, it does not exist in the output despite explicitly being set to True in model_kwargs.

If I put logprobs parameter outside model_kwargs it does show warning which gives me confidence that the place is right.

System Info

System Information

OS: Windows
OS Version: 10.0.22635
Python Version: 3.11.1 (tags/v3.11.1:a7a450f, Dec 6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)]

Package Information

langchain_core: 0.1.18
langchain: 0.1.5
langchain_community: 0.0.17
langchain_google_genai: 0.0.3
langchain_openai: 0.0.5
OpenAI: 1.11.1

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph
langserve

EDIT: added openai package version.

[dosubot]

🤖

Thank you for providing a detailed description of the issue you're encountering. It seems like the logprobs parameter is not working as expected in the LangChain framework. To better understand the problem and find a solution, I need a bit more information.

Could you please provide the following details:

1. Are there any error messages or warnings that appear when you run the code? If so, could you please share them?
2. Have you tried using the logprobs parameter with other models in the LangChain framework? If so, did you encounter the same issue?
3. Could you please share the version of the OpenAI package you're using?

This information will help me understand the issue better and provide a more accurate solution. Thank you for your cooperation.

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[emilmirzayev]

There are no errors or warnings. There is an output, but it does not entail the probabilities.

The issue is related to ChatOpenAI and the only different thing one can do is to change the model parameter. Same output with gpt-4 and default which is gpt-3.5-turbo.

[lucatrovato]

Having the same problem with similar configuration. I can see the logprobs are processed using the debug mode, but they are neither returned by ChatOpenAI nor when used in chains. Is there a way to extract them?

[hinthornw]

You'll have to use the .generate() method for now to retain the logprobs in the output - we are looking into including it inline with the invoke() method

[emilmirzayev]

@hinthornw , thank you for your reply. Do you mean to use .generate() on llm instance instead of .invoke()?
If so, it did not work. Pasting the code and error below.

If it is not what you meant, could you please give bit more detail? Thank you a lot.

from langchain_openai import ChatOpenAI


llm = ChatOpenAI(temperature=0,
    openai_api_key = api_key,
        model_name="gpt-4",
        model_kwargs = {"logprobs": True,
        "top_logprobs":3}) 



llm.generate("Please categorize this text below into positive, negative or neutral: I had a good day")

Error message (bit long):

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[44], line 12
      1 from langchain_openai import ChatOpenAI
      4 llm = ChatOpenAI(temperature=0,
      5     openai_api_key = api_key,
      6         model_name="gpt-4",
      7         model_kwargs = {"logprobs": True,
      8         "top_logprobs":3}) 
---> 12 llm.generate("Please categorize this text below into positive, negative or neutral: I had a good day")

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain_core\language_models\chat_models.py:408, in BaseChatModel.generate(self, messages, stop, callbacks, tags, metadata, run_name, **kwargs)
    406         if run_managers:
    407             run_managers[i].on_llm_error(e, response=LLMResult(generations=[]))
--> 408         raise e
    409 flattened_outputs = [
    410     LLMResult(generations=[res.generations], llm_output=res.llm_output)
    411     for res in results
    412 ]
    413 llm_output = self._combine_llm_outputs([res.llm_output for res in results])

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain_core\language_models\chat_models.py:398, in BaseChatModel.generate(self, messages, stop, callbacks, tags, metadata, run_name, **kwargs)
    395 for i, m in enumerate(messages):
    396     try:
    397         results.append(
--> 398             self._generate_with_cache(
    399                 m,
    400                 stop=stop,
    401                 run_manager=run_managers[i] if run_managers else None,
    402                 **kwargs,
    403             )
    404         )
    405     except BaseException as e:
    406         if run_managers:

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain_core\language_models\chat_models.py:577, in BaseChatModel._generate_with_cache(self, messages, stop, run_manager, **kwargs)
    573     raise ValueError(
    574         "Asked to cache, but no cache found at `langchain.cache`."
    575     )
    576 if new_arg_supported:
--> 577     return self._generate(
    578         messages, stop=stop, run_manager=run_manager, **kwargs
    579     )
    580 else:
    581     return self._generate(messages, stop=stop, **kwargs)

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain_openai\chat_models\base.py:445, in ChatOpenAI._generate(self, messages, stop, run_manager, stream, **kwargs)
    441     stream_iter = self._stream(
    442         messages, stop=stop, run_manager=run_manager, **kwargs
    443     )
    444     return generate_from_stream(stream_iter)
--> 445 message_dicts, params = self._create_message_dicts(messages, stop)
    446 params = {
    447     **params,
    448     **({"stream": stream} if stream is not None else {}),
    449     **kwargs,
    450 }
    451 response = self.client.create(messages=message_dicts, **params)

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain_openai\chat_models\base.py:462, in ChatOpenAI._create_message_dicts(self, messages, stop)
    460         raise ValueError("`stop` found in both the input and default params.")
    461     params["stop"] = stop
--> 462 message_dicts = [_convert_message_to_dict(m) for m in messages]
    463 return message_dicts, params

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain_openai\chat_models\base.py:462, in <listcomp>(.0)
    460         raise ValueError("`stop` found in both the input and default params.")
    461     params["stop"] = stop
--> 462 message_dicts = [_convert_message_to_dict(m) for m in messages]
    463 return message_dicts, params

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain_openai\chat_models\base.py:150, in _convert_message_to_dict(message)
    144     message_dict = {
    145         "role": "tool",
    146         "content": message.content,
    147         "tool_call_id": message.tool_call_id,
    148     }
    149 else:
--> 150     raise TypeError(f"Got unknown type {message}")
    151 if "name" in message.additional_kwargs:
    152     message_dict["name"] = message.additional_kwargs["name"]

TypeError: Got unknown type P

[baskaryan]

@emilmirzayev .generate() expects a List[List[BaseMessage]] not a string:

from langchain_core.messages import HumanMessage
from langchain_openai import ChatOpenAI


llm = ChatOpenAI(temperature=0,
    openai_api_key = api_key,
        model_name="gpt-4",
        model_kwargs = {"logprobs": True,
        "top_logprobs":3}) 


llm.generate([[HumanMessage(content="Please categorize this text below into positive, negative or neutral: I had a good day")]])

we know this interface isn't intuitive and are actively brainstorming ways to improve it. If you'd like to understand the root issue and opine on potential solutions see #17189

[emilmirzayev]

@baskaryan thanks for the tip! it seems to solve the issue. Although I tried with OpenAI native API and the nesting of the fields is quite different.

I will also try to see if it works with memory. Thanks for the work again!

On a second thought, I think the error message could also be better: TypeError: Got unknown type P is not very informative

[baskaryan]

Yep message could definitely be improved. If you have time to open a PR would be much appreciated!