Metrics & Scores

[pgvetrivel]

Describe your question

I am using self hosted langfuse ui. I have a golden data which specifies 'positive' or 'negative'. Similarly I have llm generated output as 'positive' or 'negative'. I need following metrics

1. For every data point (Per-Item Comparison), I need to compare the llm output with golden data and specify weather it is right or wrong. If it is right, market it as 1 else 0.
2. Need to calculate overall confusion matrix.
   How to configure this in langfuse ui

I did a search in github. It is recommended to use 'Custom Evaluators via SDK'. Is this the only option currently available?

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[jannikmaierhoefer]

For every data point (Per-Item Comparison), I need to compare the llm output with golden data and specify weather it is right or wrong. If it is right, market it as 1 else 0.

Have a look at LLM-as-a-Judge for this: https://langfuse.com/docs/evaluation/experiments/experiments-via-ui#optional-set-up-llm-as-a-judge

Need to calculate overall confusion matrix.
This is not supported yet. You would need to export your evaluation results: https://langfuse.com/faq/all/retrieve-experiment-scores