datasets evaluator filtering options？

[CryoThrust]

Describe your question

When using datasets run, I can connect an evaluator using the SDK. This evaluator is created by me in the interface. Do I need to specify the filter conditions? If I don't specify anything, will it affect other business traces?

Also, if my filter conditions are completely clear and can specify the trace of my dataset item, do I not need to specify the evaluator? On this point, I would like to ask how this is considered in the design and what the correct usage is. Looking forward to your reply.

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

v3.112.0 OSS

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[marliessophie]

Hey, good questions. I will use the terms experiment/dataset run interchangeably.

There are two possible types of evaluators if you will, when running the experiment SDK:

1. Defined in code: if you define any custom code evaluator for the current experiment, then it will run only in the scope of the given experiment. There will be no execution traces automatically by langfuse. The local evaluator will simply post scores to langfuse that are attached to your experiment traces.
2. Defined in langfuse ui: these will only work if you run an experiment using a langfuse dataset. Say you have an evaluator you set up for the given dataset in the langfuse ui. Then, when you run the experiment, the evaluator will run on the resulting dataset run items. This will produce execution traces and scores in langfuse.

Does this clear things up?