Skip to content

Conversation

@shahules786
Copy link
Member

@shahules786 shahules786 commented Nov 14, 2024

New formating example.

Given a input and response. Evaluate the submission only using the given criteria. Use only 'Yes' (1) and 'No' (0) as verdict.
Please return the output in a JSON format that complies with the following schema as specified in JSON Schema:
{'properties': {'reason': {'description': 'Reason for the verdict', 'title': 'Reason', 'type': 'string'}, 'verdict': {'description': 'The verdict (0 or 1) for the submission', 'title': 'Verdict', 'type': 'integer'}}, 'required': ['reason', 'verdict'], 'title': 'AspectCriticOutput', 'type': 'object'}

--------EXAMPLES-----------
Example 1
Input: {
    "user_input": "Who was the director of Los Alamos Laboratory?",
    "response": "Einstein was the director of Los Alamos Laboratory.",
    "criteria": "Is the output written in perfect grammar"
}
Output: {
    "reason": "the criteria for evaluation is whether the output is written in perfect grammar. In this case, the output is grammatically correct.",
    "verdict": 1
}
-----------------------------

Now perform the same with the following input
Input: {
    "user_input": "What is x",
    "response": "Y",
    "criteria": "no one"
}
Output: 

@shahules786 shahules786 marked this pull request as ready for review November 14, 2024 04:27
@dosubot dosubot bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Nov 14, 2024
@shahules786 shahules786 merged commit d18a6e7 into explodinggradients:main Nov 14, 2024
16 checks passed
shahules786 added a commit to shahules786/ragas that referenced this pull request Nov 14, 2024
New formating example. 

```
Given a input and response. Evaluate the submission only using the given criteria. Use only 'Yes' (1) and 'No' (0) as verdict.
Please return the output in a JSON format that complies with the following schema as specified in JSON Schema:
{'properties': {'reason': {'description': 'Reason for the verdict', 'title': 'Reason', 'type': 'string'}, 'verdict': {'description': 'The verdict (0 or 1) for the submission', 'title': 'Verdict', 'type': 'integer'}}, 'required': ['reason', 'verdict'], 'title': 'AspectCriticOutput', 'type': 'object'}

--------EXAMPLES-----------
Example 1
Input: {
    "user_input": "Who was the director of Los Alamos Laboratory?",
    "response": "Einstein was the director of Los Alamos Laboratory.",
    "criteria": "Is the output written in perfect grammar"
}
Output: {
    "reason": "the criteria for evaluation is whether the output is written in perfect grammar. In this case, the output is grammatically correct.",
    "verdict": 1
}
-----------------------------

Now perform the same with the following input
Input: {
    "user_input": "What is x",
    "response": "Y",
    "criteria": "no one"
}
Output: 
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant