Add answer-generation as part of Dataset-/Experiment-UI

[HackNSnack]

Describe the feature or potential improvement

We are looking to have an all-in-1 platform where users can both create, upload, generate, and evaluate their datasets.

The main feature we miss, is the ability to generate an output in a column prior to evaluation.
So e.g. I'd like to:

1. Run an experiment populating a column with an LLM output
2. Run another experiment to actually evaluate the question-answer pair

Additional information

Is this currently possible? Or is there some way to get close to this?
I'm not overly familiar with Langfuse yet, so I'm not fully aware of the functionalities available.

Any feedback is welcome! :)

[jannikmaierhoefer]

Hi @HackNSnack, yes, you can

1. run a Dataset Experiment: https://langfuse.com/docs/evaluation/experiments/experiments-via-ui and
2. use LLM-as-a-Judge to evaluate the Dataset Run results: https://langfuse.com/docs/evaluation/experiments/experiments-via-ui#optional-set-up-llm-as-a-judge

Let me know if this answers your question!

[HackNSnack]

Yes! Thanks so much for the pointers.

Got everything working now! I just needed to figure out that I could map the variables in the prompts to both the dataset items, as well as the trace outputs😅

Thank you!