Availability of evals when self-hosting

[derevyan]

Describe the feature or potential improvement

Error: Evals available in cloud only

Error: Evals available in cloud only
05/13/2024
10:01:40 PM
    at /app/web/.next/server/pages/api/trpc/[trpc].js:90:10178

Additional information

No response

[marcklingen]

Hi Derevyan,

Model-based evaluation jobs within Langfuse UI is currently in public beta on Langfuse Cloud as it uses some preview infrastructure of Langfuse v3 (more on this here). We wanted to be able to test this feature while being thoughtful about the migration path to new infrastructure for everyone self-hosting Langfuse.

When self-hosting, you can easily also run model-based evaluations on data in Langfuse via the SDKs, more on this here including cookbooks: https://langfuse.com/docs/scores/model-based-evals#sdk. Also, all other evaluation options (via API, manual review in UI) are fully available when self-hosting Langfuse today as they do not require any additional infrastructure.

Tracing, all integrations, datasets, and prompt management are fully available under an MIT license. In addition, we will release some add-on features under a commercial license. We have seen many OSS projects who make self-hosting unnecessarily complex to drive users to their cloud offering for production use (which we do not like) and we think offering add-ons under a commercial license is a way better way to ensure we can work on Langfuse long-term while releasing most of our work under an MIT license.