Passing Custom Metadata Through to LiteLLM Proxy?

[dkindlund]

Hey @HenryHengZJ , in Flowise, I have an existing chatflow defined and it's connected to a LiteLLM proxy using the ChatOpenAI chat model node.

When I call the predictions API, I'm looking to try and include metadata that can get sent to the LiteLLM proxy -- something like this:

curl https://.../api/v1/prediction/... \
     -X POST \
     -d '{"question": "Hey, how are you?", "overrideConfig": { "metadata": { "litellm_key": "litellm_value" }}}' \
     -H "Content-Type: application/json"

And when Flowise sends the completion API call over to the LiteLLM proxy, it would sort of look like:

curl --location 'http://0.0.0.0:4000/chat/completions' \
    --header 'Content-Type: application/json' \
    --data '{
    "model": "gpt-3.5-turbo",
    "messages": [
        {
        "role": "user",
        "content": "Hey, how are you?"
        }
    ],
    "metadata": {
        "litellm_key": "litellm_value"
    }
}'

My questions are:

1. Does this sort of functionality already exist in Flowise? If so, can you point me to the right overrideConfig parameter to use?
2. If it does not exist already, would this be easy to add?

The reason why I'm looking for this, is because I not only have Langfuse connected in Flowise, but I also have Langfuse connected in LiteLLM, and I'm looking for an easy way to track the same Trace across both sources.

Ref: https://docs.litellm.ai/docs/proxy/logging#logging-metadata-to-langfuse

Thanks in advance!

[phasmax]

@dkindlund did you manage to figure this out? I'm trying to make use of the LiteLLM prompt management feature, where you can pass the prompt_id and prompt_parameters to liteLLM, and it'll proxy it through to Langfuse. I tried the baseOptions JSON but that just crashes it completely - get connection failture.

[rossm-mf]

it can serve as inspiration but I can make work openwebui to litellm using this filter https://github.com/thiswillbeyourgithub/openwebui_custom_pipes_filters/blob/main/filters/add_metadata.py