Listening to eval scores

[sagarwal5-godaddy]

Describe your question

Is there a way to trigger an event as soon as evaluation scores are generated for a specific traceId, so that I can publish those scores to an SQS queue for downstream processing?

I also need all evaluation metric scores for that traceId to be included in a single event before sending them to Lambda or any other consumer. The goal is to use the full set of eval scores as feedback in an agent loop.

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Langfuse does not currently provide a built-in event-driven mechanism or webhook system to trigger notifications when evaluation scores are generated for a specific trace.(1)(2)(3)

However, you can implement a polling-based approach to retrieve scores for downstream processing:

1. Fetch scores programmatically using the Langfuse API or SDKs to retrieve all scores associated with a specific traceId(3)

2. Poll for score completion by periodically querying the scores endpoint filtered by traceId until all expected evaluation scores are present(3)

3. Aggregate and publish once you've confirmed all evaluation metrics are available, you can aggregate them and publish to your SQS queue(3)

The documentation shows you can create and retrieve scores via the SDK/API, but there's no mention of real-time event notifications or webhooks for score generation.(1)(2)(3)

Workaround approach:

• Use the Langfuse API to periodically check for scores on a given traceId(3)
• Implement logic in your application to detect when all expected scores are present
• Once complete, trigger your downstream Lambda/SQS workflow

For a more integrated solution, you might want to submit a feature request on the Langfuse GitHub discussions for webhook support or event-driven score notifications.(2)

📚 Sources:

1. Evaluation Overview
2. Troubleshooting and FAQ
3. Custom Scores

---

Have another question? Just tag @inkeep.

[sagarwal5-godaddy]

@inkeep, The challenge with the polling-based approach is that it requires continuously querying the Langfuse score APIs, and we also need to know the traceId in advance in order to poll for the corresponding scores. This means we would still need a separate mechanism to push these traceIds into our downstream system before it can begin polling Langfuse.

Given this limitation, would Langfuse consider supporting a feature where evaluation metric scores (e.g., Hallucination, Helpfulness, etc.) can be published directly to an SQS queue or similar downstream system, rather than requiring the client to poll the API? This would help avoid continuous polling and simplify downstream processing.

[inkeep-ai-assistant[bot]]

I understand the limitation you're facing with the polling approach - it creates a circular dependency where you need the traceId to poll, but you also need a mechanism to know when to start polling in the first place.

Based on the available documentation, Langfuse doesn't currently offer native webhook or event-driven integrations to push evaluation scores to external systems like SQS queues. This type of event-driven score publishing feature would indeed be valuable for your use case of feeding evaluation scores back into an agent loop without continuous polling overhead.

Potential workarounds to consider:

1. Capture traceIds at creation time - Since you're generating traces in your application, you already know the traceIds. You could push these to your downstream system immediately after trace creation, then have your system poll for scores after an expected evaluation completion time.

2. Use Langfuse's batch export capabilities - Langfuse v3.x.x introduced worker containers and batch export functionality. While not real-time, you might be able to leverage these for periodic score exports.

3. Self-hosted with custom extensions - If you're open to self-hosting, you could potentially add custom event handlers to the Langfuse worker container to publish scores as they're generated.

Feature request recommendation:

Your use case for event-driven score publishing (webhooks, SQS integration, or similar pub/sub mechanisms) is a legitimate architectural need. I'd recommend opening a feature request on the Langfuse GitHub repository describing:

• The event-driven score notification requirement
• Your specific use case (agent feedback loops)
• Preferred integration patterns (webhooks, SQS, SNS, etc.)

This would help the Langfuse team understand the demand for this capability and potentially prioritize it in their roadmap.

---

Have another question? Just tag @inkeep.

[sagarwal5-godaddy]

@inkeep, I have another question: Is it possible to retrieve evaluation metric scores aggregated at the conversationId or agentId level, or does Langfuse only provide scores at the traceId level—meaning we would need to handle the aggregation ourselves?

[jannikmaierhoefer]

@inkeep

[inkeep-ai-assistant[bot]]

Langfuse Scores can be attached to multiple levels beyond just traces(1). According to the documentation, Scores reference one of the following objects(1):

• Trace - for evaluation of a single interaction (most common)(1)
• Observation - for evaluation of a single observation below the trace level(1)
• Session - for comprehensive evaluation of outputs across multiple interactions(1)
• DatasetRun - for performance scores of a Dataset Run(1)

For your specific use case:

ConversationId level: If you're using sessionId to represent conversations, you can retrieve scores at the session level(1). Scores can be linked to a sessionId for comprehensive evaluation across multiple interactions(1).

AgentId level: There's no native "agentId" concept in Langfuse's score data model(1). However, you could potentially use metadata filtering if you tag traces/sessions with agent identifiers, though the GitHub discussion indicates metadata filtering isn't fully supported across all APIs yet(2).

Aggregation: Langfuse doesn't appear to provide built-in aggregation of scores across sessions or custom groupings. You would need to:

1. Query scores filtered by sessionId(1)
2. Handle the aggregation logic yourself in your application

The Score object includes these reference fields: traceId, observationId, sessionId, and datasetRunId(1), so you can filter by whichever level is relevant to your use case.

📚 Sources:

1. Scores Data Model
2. Support for "tenantId" as an Optional Parameter in Traces

---

Have another question? Just tag @inkeep.