[v0.0.1] Flexible Comparator Implementation

[v0.0.1] Flexible Comparator Implementation

Allows for:

• Custom semantic comparators
• Custom embedding clients
• Improved ease of use

Usage

import semtest

cosine_similarity = semtest.CosineSimilarity(
    semantic_expectation="A dog is in the background of the photograph"
)

# First test picked up and executed by framework mode
@semtest.benchmark(
    comparator=cosine_similarity,
    iterations=3
)
def mock_prompt_benchmark():
    """Mock example of benchmarking functionality."""

    # intermediary logic ...

    mocked_llm_response = query_llm(...)  # example llm autocomplete response

    return mocked_llm_response  # return LLM string response for benchmarking

res: semtest.Benchmark = mock_prompt_benchmark()  # manually executed benchmark (non-framework mode)

print(res.benchmarks())

Next Steps:

• Implement SchemaComparator and GroundTruth comparators