EVALMY.AI

Python client library for EVALMY.AI, a public service for evaluating GPT answers based on semantics.

This service enables cost-effective, reliable, and consistent automated testing of GenAI solutions like RAGs and others.

Using EVALMY.AI, you can accelerate your development process, reduce testing costs and enhance the reliability of your AI applications.

Example

You are developing a RAG (Retrieval-Augmented Generation) to answer simple geographical questions. It's essential to test its performance both during development and after release to ensure the model maintains its accuracy. For this purpose, you create a set of test questions along with their respective correct answers.

1. What is the capital of France?               --> Paris
2. What are three longest rivers in the world?  --> Nile, Amazon, Yanktze
3. Which continent is the second smallest?      --> Europe

Your RAG provides following answers:

1. The capital of France is Paris.
2. Nile, Mississippi and Amazon.
3. The second smallest continent in the world is Australia.

Pretty well but not yet perfect.

Reading through long sets of AI-generated answers can become tedious and monotonous, especially if the test set remains unchanged. This costs time and can lead to people making errors.

Fortunately, AI can handle the task for us. With the help of EVALMY.AI, simply send us the questions along with the expected and actual answers, and you'll receive the results effortlessly.

CONTRADICTIONS IN TEXTS:
1. Score: 1.0, 
Reasoning: "Both texts identify the capital of France correctly."

2. Score: 0.5,
Severity: Large
Reasoning: "Different rivers listed as the three longest." 

3. Score: 0.0, 
Severity: Critical
Reasoning: Different continents identified as the second smallest.

Instalation

The evalmyai requires python 3.8 or higher.

python -m pip install evalmyai 

Simple usage

from evalmyai import Evaluator

data = {
    "expected": "Jane is twelve.",
    "actual": "Jane is 12 yrs, 7 mths and 3 days old."
}

ev = Evaluator(...) # see authentication later

result = ev.evaluate(data)

print(result['contradictions'])

The result of the evaluation is as follows:

{
    "contradictions": {
        "scores": {
            "score": 1.0
        },
        "reasoning": {
            "statements": [
                {
                    "severity": "negligible",
                    "summary": "Different descriptions of Jane's age.",
                    "reasoning": "The statement from <TEXT 1> 'Jane is twelve.' is a general statement about Jane's age, indicating that she is twelve years old. The statement from <TEXT 2> 'Jane is 12 yrs, 7 mths and 3 days old.' provides a more precise age, including years, months, and days. There is no contradiction between these statements as the second text simply provides more detail about Jane's age, but does not conflict with the first text's assertion that she is twelve years old."
                }
            ]
        }
    }
}

Authentication

First, you need your EVALMY.AI service token, which you can get here.

The service runs on your own instance of GPT, either in Azure or directly on an OpenAI endpoint you provide.

Due to capacity limits per organization, we cannot provide an GPT endpoint directly.

Azure

If you use an Azure endpoint, the configuration should look like this:

from evalmyai import AzureAuth, Evaluator

token = "YOUR_EVALMYAI_TOKEN_expected_to_be_64_characters_long_CHANGE_IT!"

auth_azure = AzureAuth(
    api_key="cd0...101",
    azure_endpoint="https://...azure.com/",
    api_version="2023-07-01-preview",
    azure_deployment="...",
)

ev = Evaluator(auth_azure, token)

OpenAI

In case you use OpenAI endpoint, the configuration should look like this:

from evalmyai import OpenAIAuth, Evaluator

token = "YOUR_EVALMYAI_TOKEN_expected_to_be_64_characters_long_CHANGE_IT!"

auth_open_ai = OpenAIAuth(
    api_key= "...",
    model= "gpt-4" # select your model, we strongly recommend GPT-4.
)

ev = Evaluator(auth_open_ai, token)

The EVLAMY.AI tutorial with practical exmaples can be found here.