See LLM Generation when using llm-as-a-judge

[nitin302]

Describe the feature or potential improvement

Currently when using llm-as-a-judge, we can see the comment and score value given by the LLM Judge for a given trace.

But more additional features are required:

1. See the exact LLM generation used for judging. This will help us see what exactly happened in LLM call - LLM input/output, token consumption, LLM cost etc.
2. We need to also understand cost of llm-as-a-judge aggregated at evaluator level. This will help us understand cost of the evaluator to adjust sampling rate.
3. See what model was used in the score level. This will help us compare between different LLM judges and choose the one that offers the best performance for cost.
4. Add support for multi-modal input for llm-as-a-judge.
5. Add ability run evaluation for a single selected trace. This will help run evaluations manually from UI. Also, we can test before enabling the feature.

Additional information

No response

[jannikmaierhoefer]

Thanks @nitin302 for adding this! We are tracking this here: https://github.com/orgs/langfuse/discussions/7166

[nitin302]

@jannikmaierhoefer sure, cost is only 1 part of this feature request. please see others as well.

[jannikmaierhoefer]

Thanks @nitin302, I added them to the other thread for deduplication.