ordering dataset items within experiment results by a number that is in the llm response for each item?

[flavio87]

can we order the dataset items within experiment results / run results by certain numerical numbers/columns/json fields contained in the llm results for each dataset item?

Say we are asking the llm to give a score from 0 to 100 about some text we pass in the prompt. I want to order that list of items in descending order within the experiment results table.

[inkeep-ai-assistant[bot]]

Based on the available documentation, the dataset items table in experiment results currently has limited ordering capabilities. The table is sorted by creation time, with support for ordering and filtering coming soon(1).

The recent updates (>v3.91.1) have added sorting by created_at with the ability to switch between ascending and descending order(1). However, there's no documented feature for custom ordering based on specific numerical values or JSON fields within the LLM responses themselves.

For your use case where you want to order items by a score (0-100) that the LLM returns, you would need to:

1. Extract that score from the LLM output using an evaluator function
2. Store it as a score on the trace

Here's how you can capture custom scores during experiments(2):

def score_evaluator(*, input, output, **kwargs):
    # Extract the score from the LLM output
    score = extract_score_from_output(output)  # Your custom extraction logic
    
    return Evaluation(
        name="llm_score",
        value=score,
        comment=f"LLM assigned score: {score}"
    )

result = langfuse.run_experiment(
    name="Scoring Experiment",
    data=test_data,
    task=my_task,
    evaluators=[score_evaluator]
)

(2)

While the scores will be captured and visible in the UI, custom sorting by these score values in the dataset runs table isn't currently supported based on the available documentation. You may need to export the results and sort them externally, or request this as a feature enhancement.

📚 Sources:

1. feat: dataset item runs table sorted by Run At
2. Experiments via SDK

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Thanks @flavio87, inkeep is correct - we have this on our roadmap.