Create a custom evaluator which runs on coding log

[sandeep-07]

Create a custom evaluator that calculates the correctness of an ai result on the dataset item. User should be able to define the custom coding logic without running it through LLM , and user can run this evaluator directly from playground

Additional information

No response

[marliessophie]

Hey, thanks for sharing feedback. I'd suggest using the experiment SDK. This allows you to define any custom code evaluator and automatically runs it against your experiment items and posts the results to langfuse. Please view docs for more information: https://langfuse.com/changelog/2025-09-17-experiment-runner-sdk

Though we are heavily investing in evaluation, unfortuantely we do not support custom code evaluators in the playground yet.