handleLLMNewToken called with empty tokens when using function calling

[gramliu]

When calling ChatOpenAI.predictMessages with an OpenAI function call specified, the handleLLMNewToken callback is repeatedly called with empty tokens. This is likely because when OpenAI pushes new chunks to the HTTP event-stream, the structure of the response data is slightly different.

For normal text responses, the schema of a choice follows:

{
  "delta": { "content": string },
  "index": number,
   "finish_reason: null
}

With function calling, however, it now looks like:

{
  "delta": {
    "function_call": {
       "name": string,
       "arguments": string
    }
  },
  "index": number,
  "finish_reason": null
}

Edit: This does seem to be the case: see

[vladholubiev]

Here are some of the ideas:

Single callback

callbacks: [
    {
      handleLLMNewFunctionCall(function_call: {name: string;args:string}) {
        /*
        function_call = {
          name: "xxx",
          args: 'yyy'
        }
         */
      },
    }
  ]

Separate callbacks for function name and argument

callbacks: [
    {
      handleLLMNewFunctionCallName(function_call_name: string) {
        /*
        function_call_name = "xxx"
         */
      },
      handleLLMNewFunctionCallArgs(function_call_args: string) {
        /*
        function_call_args = "yyy"
         */
      }
    }
  ]

From my observations, looks like OpenAI always emits entire function_name in stream as a first chunk. So probably second option makes more sense.

[vladholubiev]

Looks like a PR was submitted #2025

[dosubot]

Hi, @gramliu! I'm Dosu, and I'm helping the langchainjs team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding, the issue you reported is related to the handleLLMNewToken callback receiving empty tokens when using function calling in ChatOpenAI.predictMessages. It seems that there are two possible solutions suggested by vladholubiev: using a single callback or separate callbacks for the function name and argument. Additionally, there is a comment mentioning that a pull request has been submitted to address the issue.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the langchainjs repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution to langchainjs!