SDK-Python :- Cost calculation for custom model

[kshitijbisen]

I am using a custom model which return token counts and content. i have defined model defination on langfuse ui with usage details as prompttoken and cadidatetoken as keys but not able to get cost getting 0 cost only.

Llm.invoke(query)-> AIMessage(content,tokenDetail)

[inkeep-ai-assistant[bot]]

To get cost calculation working for your custom model in Langfuse, the keys in your usage details (usage_details) must exactly match the usage type keys defined in your model definition on Langfuse. If your model definition uses keys like prompttoken and cadidatetoken, but your SDK sends different keys (such as input, output, prompt_tokens, completion_tokens, etc.), cost will not be calculated and will remain zero(1).

Make sure:

• The keys in your usage_details dictionary match those in your custom model definition.
• The model definition in Langfuse UI uses the same keys as those passed from your code.
• Cost will only be calculated if both the usage details and the model definition keys align(1).

For example, if your model definition uses prompttoken and cadidatetoken, you must send:

generation.update(
    usage_details={
        "prompttoken": <value>,
        "cadidatetoken": <value>
    }
)

If you want to change to the standard keys (input, output, etc.), update both the model definition and your SDK usage to match(1).

You can review more details here: Model Usage & Cost Tracking.

📚 Sources:

1. Model Usage & Cost Tracking

---

Have another question? Just tag @inkeep.

[kshitijbisen]

@inkeep i am using langchain default callback handler

[inkeep-ai-assistant[bot]]

When using the Langchain default callback handler with Langfuse, cost calculation depends on the model name and usage details being passed correctly from Langchain to Langfuse. If you are using a custom model, ensure the following:

• The model name sent from Langchain matches the custom model definition you created in the Langfuse UI(1).
• The usage detail keys (e.g., prompttoken, cadidatetoken) in your custom model definition must exactly match the keys sent by Langchain. If Langchain sends keys like input or output, your model definition in Langfuse should use those same keys(1).

If Langchain is not sending the correct model name or usage detail keys by default, you may need to override or extend the callback handler to ensure they are set as expected(2).

You can review more about custom model handling and cost calculation here: Model Usage & Cost Tracking, how to pass custom user managed model name to calculate the cost via langchain.

📚 Sources:

1. Model Usage & Cost Tracking
2. how to pass custom user managed model name to calculate the cost via langchain

---

Have another question? Just tag @inkeep.

[kshitijbisen]

@inkeep where should i pass the usage details model name is matching what should be format of llm response for proper cost calculation

[inkeep-ai-assistant[bot]]

For cost calculation to work in Langfuse when using a custom model, the usage details must be passed to Langfuse in a way that exactly matches the keys defined in your model definition. When using Langchain with the Langfuse callback handler, you need to ensure:

• The model name in your Langfuse model definition matches the one sent from Langchain.
• The usage details (token counts) are included in the LLM response and are passed to Langfuse with keys that match your model definition.

If usage details are not sent automatically by Langchain, you may need to update the generation with the usage details after you get the LLM response. The correct format is a dictionary where keys match your model definition (e.g., if your model uses "prompttoken" and "cadidatetoken", use those keys):

generation.update(
    usage_details={
        "prompttoken": <value>,
        "cadidatetoken": <value>
    }
)

(1)

The model name should be set so it matches your Langfuse model definition, and the usage details must use the same key names as in your model definition. If your LLM response provides token counts under different names, map them to the correct keys before passing them to Langfuse.

If you want to pass a custom model name in Langchain, you can do so using the metadata or model attributes when invoking the chain or initializing the handler(2).

📚 Sources:

1. Model Usage & Cost Tracking
2. how to pass custom user managed model name to calculate the cost via langchain

---

Have another question? Just tag @inkeep.

[kshitijbisen]

@inkeep how does langchain passes the details to langfuse

[inkeep-ai-assistant[bot]]

When using Langchain with Langfuse, the Langfuse callback handler maps Langchain events (chains, LLMs, tools, retrievers, agents) to Langfuse observations. For LLM calls, Langchain passes details such as model name, prompts, responses, and usage (token counts) if available from the LLM provider, into the Langfuse generation observation(1).

If you want to pass additional information (such as session_id, user_id, tags, or custom usage details), you can use the metadata parameter in your Langchain chain invocation. Here’s the format used for passing details:

response = chain.invoke(
    {"topic": "cats"},
    config={
        "callbacks": [langfuse_handler],
        "metadata": {
            "langfuse_session_id": "session_123",
            "langfuse_user_id": "user_456",
            "langfuse_tags": ["production", "humor-bot"],
            "custom_field": "additional metadata"
        }
    }
)

The callback handler will automatically extract and forward model name, prompt, response, and available usage details (like token counts) to Langfuse. If your LLM or chain does not provide usage details by default, you may need to update the generation observation with usage details after the call(1).

📚 Sources:

1. Python SDK (v3)

---

Have another question? Just tag @inkeep.

[kshitijbisen]

What is the key for usage details to be passed in metadata for cost tracking

[kshitijbisen]

@inkeep

[inkeep-ai-assistant[bot]]

There is no documented key for passing usage details via the metadata field for cost tracking in Langfuse. Usage details for cost tracking must be provided using the usage_details parameter (or usage in some SDK versions) when updating or creating a generation observation, not as part of metadata(1)(2)(3). The keys in your usage_details dictionary must match exactly the usage type keys defined in your model definition on Langfuse for cost tracking to work(1).

Example usage (Python low-level SDK):

generation = trace.generation(
    name="llm-generation",
    usage_details={
        "input": 50,
        "output": 49,
        "total": 99,
    }
)

(2), (3)

Passing usage details in metadata will not trigger cost tracking. Always use usage_details (or usage) with the correct keys for cost calculation(1)(2)(3).

📚 Sources:

1. Model Usage & Cost Tracking
2. Python SDK (Low-level)
3. How do I set token usage details using low level SDK?

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @kshitijbisen, can you see the usage tokens in Langfuse but the costs are still not calculated?

[kshitijbisen]

@jannikmaierhoefer I wasn’t able to see token usage in Langfuse initially, since I didn’t know how to pass the token usage data to Langfuse via the langchain-langfuse callback handler.

I modified the _generate method of my custom model class and made it return token usage inside the llm_output parameter of the ChatResult object.
After doing this, Langfuse was able to pick up the usage correctly, based on the model definition I had already set up in the Langfuse UI with proper usage name mapping.