OpenAI compare

This repo contains a Jupyter notebook to that allows you to compare different language models and see which one gives the best results for the given prompt.

Features

• Compare different language models.
• Use any language model host that exposes OpenAI-compatible APIs.
• Included code for Foundry Local and Ollama.
• Load prompts from Prompty files.
• Select from common scoring functions, such as:
  • BLEU
  • ROUGE 1
  • ROUGE 2
  • ROUGE L
  • BERT F
  • BERT R
  • BERT P
  • Edit distance
• Define weighted scores to evaluate the results.
• Cache LLM responses and scores for faster evaluation.
• Compare results in a chart.
• Define threshold to quickly see which models give acceptable results.

Installation

uv sync

Structure

• scoring_functions.py - contains the definition of the different scoring functions available for you to use.
• test_*.ipynb - contains different sample test cases to evaluate.
• workbench.ipynb - contains the main notebook to run the tests.

Usage

1. Define a test case. Best, start from one of the examples.
   1. Specify the title, used in charts.
   2. Specify the test tasks. For each task, define:
      1. Name
      2. One or more reference answers
      3. Prompt
      4. Optionally, temperature or top_p if you want to test them.
   3. Specify the scoring functions and weights to use.
   4. Specify the threshold for the test, to visually indicate acceptable results in the chart.
2. In the workbench.ipynb notebook:
   1. Select the language models you want to test.
   2. Choose how many times you want to invoke each prompt (default 20)
   3. Choose if you want to cache LLM results (default True).
   4. Choose if you want to cache score results (default True).
   5. Choose if you want to normalize BERT scores. More information in workbench.ipynb.
3. Run the notebook.
4. Check the results.