[R-304] support more providers for `is_finished()` logic

[jjmachan]

Suggest improvements to the parser for figuring out if the model finish reasons so that the default parser can be improved

you can also define your own custom parser and parse it too

_R-304

[ahgraber]

Llama models hosted on TogetherAI use {"finish_reason": "eos"}

{
  "content": "The letter 'r' appears twice in the word 'strawberry'.",
  "additional_kwargs": {
    "refusal": null
  },
  "response_metadata": {
    "token_usage": {
      "completion_tokens": 16,
      "prompt_tokens": 54,
      "total_tokens": 70,
      "completion_tokens_details": null,
      "prompt_tokens_details": null
    },
    "model_name": "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
    "system_fingerprint": null,
    "finish_reason": "eos",
    "logprobs": null
  },
  "type": "ai",
  "name": null,
  "id": "run-16ae060b-4bc2-4624-b07e-dce650033d6c-0",
  "example": false,
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 54,
    "output_tokens": 16,
    "total_tokens": 70,
    "input_token_details": {},
    "output_token_details": {}
  }
}

[jjmachan]

hey @ahgraber thanks for reporting this as always 🙂
I'll add it to that

[spackows]

While you're at it, IBM watsonx LLMs give eos_token when generation finishes:

generation_info={'finish_reason': 'eos_token'}

[malikaltakrori]

@spackows not sure if you are still waiting for an update or if you already figured it out, but I found this solution and just added to it the 'eos_token'

[jjmachan]

hey @malikaltakrori would you be interested in adding that as a PR? would really appreciate it - if not I'll make it 🙂

[malikaltakrori]

Hi @jjmachan,
Sorry for the late reply. Github emails still go to my school's inbox.
I would actually love to do so! (if you havn't already).

[malikaltakrori]

@jjmachan Done!

[jjmachan]

thanks a million @malikaltakrori ❤