You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am playing a bit trying to replicate the OpenAI fine-tuning guide using Langfuse, my dataset is:
"{""ticket_text"":""My monitor won't turn on!""}","{""correct_label"":""Hardware""}""{""ticket_text"":""I'm in vim and I can't quit!""}","{""correct_label"":""Software""}""{""ticket_text"":""Best restaurants in Cleveland?""}","{""correct_label"":""Other""}"
So when configuring an LLM-as-a-judge evaluator I have a prompt looking like:
Task:
Compare a model output string to a ground truth
Ground truth: {{ground_truth}}
Model output: {{model_output}}
And I bind ground_truth to a "Dataset Item > Expected output" with a $.correct_label JsonPath, however since jsonpath always output arrays I'm left with a clunky:
Ground truth: ["Other"]
Model output: Other
That won't match and need to workaround by adding normalisation rules like:
Normalize BOTH sides:
- trim whitespace
- lowercase
- remove surrounding quotes if present
- if the ground truth looks like ["label"], extract that element
- then compare the normalized strings
Which works but isn't great!
Adding a checkbox to unwrap the array to a string, or better yet directly support jq syntax which is more expressive would be nice.
Tried $.correct_label[0] but it returns the first letter, still wrapped in an array.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Describe the feature or potential improvement
I am playing a bit trying to replicate the OpenAI fine-tuning guide using Langfuse, my dataset is:
So when configuring an LLM-as-a-judge evaluator I have a prompt looking like:
And I bind ground_truth to a "Dataset Item > Expected output" with a
$.correct_labelJsonPath, however since jsonpath always output arrays I'm left with a clunky:Ground truth: ["Other"]
Model output: Other
That won't match and need to workaround by adding normalisation rules like:
Which works but isn't great!
Adding a checkbox to unwrap the array to a string, or better yet directly support jq syntax which is more expressive would be nice.
Tried
$.correct_label[0]but it returns the first letter, still wrapped in an array.Thanks!
Additional information
No response
All reactions