Add pattern-based trace selection for LLM-as-a-Judge evaluator targets (probably can be added everywhere where this component is used)

[pavzagor]

Problem

When setting up an LLM-as-a-Judge evaluator, we often need to run the evaluator only on traces whose Name follows a naming pattern (experiment prefix, template name, run id, etc.).

In the current UI, Name selection is checkbox-based. Search narrows the list, but you still must click each matching checkbox one-by-one. For dozens/hundreds of names this is effectively impossible.

Current behavior

• Path: Evaluation → LLM-as-a-Judge → Set up evaluator → Target → Target filter → Name

• Name values appear as a dropdown with checkboxes.

• You can search within the dropdown.

• There is no:

  • contains / substring operator for Name
  • regex / wildcard matching
  • “Select all (filtered)” / “Deselect all (filtered)”

Impact

• Targeted evals (only a subset of traces) become slow and error-prone.
• Teams either skip evaluations or waste time clicking checkboxes.
• Only practical workaround is a custom browser-console JavaScript script.

Examples

We frequently need:

• All traces where Name contains test
• All traces where Name starts with attempt_
• All traces where Name matches a regex like ^experiment-123-.*

Proposed solutions

Option A — Add string operators to Target filter

Add operators for Name:

• contains
• starts_with
• ends_with
• matches_regex (optional but ideal)

Example:

• Where Name contains attempt_
• Where Name matches_regex ^test\d+_attempt_.*

Option B — Add bulk selection for filtered dropdown results

Inside the Name dropdown:

• Button: “Select all (filtered)”
• Button: “Deselect all (filtered)”
• Show a counter: “Selected X / Y filtered”

Behavior:

• Search narrows visible values.
• “Select all (filtered)” selects all values currently matching the search.

Best outcome

Implement both A and B:

• A for precise, auditable filtering.
• B for quick selection when users prefer the value list.

Acceptance criteria

• Users can target all traces matching a substring/pattern without manual checkbox clicking.
• Bulk selection applies only to the filtered subset.
• Works for large lists (hundreds/thousands) without freezing.
• The filter remains readable/auditable (e.g., “Name contains attempt_”).

Steps to reproduce

1. Open: Evaluation → LLM-as-a-Judge
2. Create or configure an evaluator
3. Go to Target → Target filter
4. Add condition on Name
5. Search for a common prefix/pattern
6. Observe that you must manually click each checkbox to select all matching values

Loom

https://www.loom.com/share/4a4983e9cb2f4e8ab639115efd4d356b

Additional information

No response

[boshasiv]

Great idea!

[hassiebp]

Thanks for raising this idea!