Batch/Multi-Trace LLM Evaluators for Pattern Discovery

[omaaartamer]

Describe the feature or potential improvement

I'm building an evaluator to identify user behavior patterns that could become standalone product features. The challenge:

• Single trace: User pastes text, asks for sources → LLM scores it as "high feature potential"
• Reality: Is this a one-off request or a recurring pattern across multiple users?

Without seeing multiple traces together, the LLM can't distinguish between:

• A genuinely novel, repeated pattern (real feature signal)
• A single creative use that won't recur (noise)

---

Proposed Solution

Add support for evaluator to batch multiple traces as single input.

Additional information

No response

[jannikmaierhoefer]

Hi @omaaartamer, thanks for adding this!