PromptLab

PromptLab is a command-line tool for testing and improving prompts against a YAML test suite.

You give it:

• a prompt template
• a task description
• a set of test cases with inputs and optional expected outputs
• an OpenAI-compatible chat completion endpoint

It runs the prompt on every test case, asks an LLM judge to score each output, and can try several rewrite strategies to find a better prompt. It also includes a small local prompt library, run history, console summaries, and optional HTML reports.

This project is useful when you want a repeatable prompt regression test instead of judging a prompt by a few ad hoc examples.

What it actually does

PromptLab has three main workflows:

1. promptlab eval Runs one prompt against a YAML test suite and prints scores.

2. promptlab optimize Scores the starting prompt, generates improved candidates with one or more strategies, evaluates those candidates, and keeps the best prompt found.

3. promptlab compare Evaluates two prompts against the same test suite and prints the winner.

During evaluation, PromptLab:

• formats your prompt with the test case input
• calls the target model to produce an answer
• calls a judge model to score that answer from 0 to 10
• reports aggregate score, pass rate, latency, token counts, and failed cases

During optimization, PromptLab can use these strategies:

• rewrite: asks an optimizer model to rewrite the prompt based on failures
• chain_of_thought: adds explicit step-by-step reasoning instructions
• few_shot: inserts examples from the best-scoring test cases
• failure_repair: targets observed failure patterns without changing the whole prompt

Strategy candidates are generated with an LLM, so optimization costs more than a plain evaluation.

System requirements

• Python 3.10 or newer
• pip
• Git, if you are installing from a clone
• Network access to an OpenAI-compatible chat completions API
• An API key for that provider

The package depends on:

• openai
• click
• rich
• pydantic
• aiosqlite
• jinja2
• pyyaml
• anyio

The default model names are gpt-4o-mini for the task model, optimizer model, and judge model. Use --model, --optimizer-model, and --judge-model if your provider uses different names.

Installation

Clone the repository and install it in editable mode:

git clone https://github.com/lord-Rheagar/prompt-lab.git
cd prompt-lab
python -m pip install -e .

For development and tests, install the dev dependencies:

python -m pip install -e ".[dev]"

Check that the CLI is available:

promptlab --version

Expected output:

promptlab, version 0.1.0

API configuration

PromptLab uses the OpenAI Python client and supports OpenAI-compatible endpoints.

For OpenAI:

export OPENAI_API_KEY="your-api-key"

On Windows PowerShell:

$env:OPENAI_API_KEY="your-api-key"

For another compatible provider, set both the base URL and key:

export PROMPTLAB_BASE_URL="http://localhost:8000/v1"
export PROMPTLAB_API_KEY="your-api-key"

PowerShell:

$env:PROMPTLAB_BASE_URL="http://localhost:8000/v1"
$env:PROMPTLAB_API_KEY="your-api-key"

You can also pass these per command:

promptlab eval \
  --base-url http://localhost:8000/v1 \
  --api-key your-api-key \
  --prompt "Classify this ticket: {input}" \
  --task "Support ticket classification" \
  --tests examples/classification.yaml

If PROMPTLAB_BASE_URL does not end in /v1, PromptLab appends /v1 for you. If the value does not start with http, it prefixes http:// and appends /v1.

Prompt templates

Use {input} or {text} where the test case input should be inserted:

Classify this support ticket. Reply with only the category.

Ticket:
{input}

If your prompt does not contain {input} or {text}, PromptLab appends the input after the prompt.

Unknown placeholders are left in place. For example, {query} is not replaced by PromptLab.

Test suite format

Test suites are YAML files. A minimal suite looks like this:

task: "Classify customer support tickets into BILLING, TECHNICAL, SHIPPING, RETURNS, or GENERAL."

judge_rubric:
  correctness: 0.8
  format_compliance: 0.2

test_cases:
  - input: "I was charged twice and need a refund."
    expected: "BILLING"
    weight: 1.0

  - input: "The app crashes every time I open it."
    expected: "TECHNICAL"
    weight: 1.0

Supported fields:

• task: description shown to the judge model
• model: optional default target model for this suite
• judge_model: optional judge model for this suite
• judge_rubric: score criteria and weights; weights are normalized automatically
• test_cases[].input: required input text
• test_cases[].expected: optional reference answer
• test_cases[].weight: optional case weight, default 1.0
• test_cases[].tags: optional list of labels

If expected is present, the judge compares the model output to that reference. If it is omitted, the judge scores without a reference answer.

Example suites are included in:

• examples/classification.yaml
• examples/summarization.yaml

Evaluate one prompt

promptlab eval \
  --prompt "Classify this support ticket into one of: BILLING, TECHNICAL, SHIPPING, RETURNS, GENERAL. Reply with only the category. Ticket: {input}" \
  --task "Support ticket classification" \
  --tests examples/classification.yaml \
  --model gpt-4o-mini \
  --judge-model gpt-4o-mini

Write an HTML report:

promptlab eval \
  --prompt "Summarize this review in two concise sentences: {input}" \
  --task "Product review summarization" \
  --tests examples/summarization.yaml \
  --report eval-report.html

Optimize a prompt

promptlab optimize \
  --prompt "Summarize: {input}" \
  --task "Product review summarization" \
  --tests examples/summarization.yaml \
  --iterations 3 \
  --model gpt-4o-mini \
  --optimizer-model gpt-4o-mini \
  --judge-model gpt-4o-mini \
  --report optimize-report.html \
  --save-best review-summary-v1

Useful options:

• --iterations: maximum optimization rounds, default 5
• --strategies: comma-separated list, default rewrite,chain_of_thought,few_shot,failure_repair
• --objectives: scoring weights for quality, cost, and latency
• --report: writes an HTML report
• --save-best: saves the best prompt to the local library
• --no-history: skips writing the run to local history

Example objective weighting:

--objectives quality=0.8,cost=0.1,latency=0.1

Quality is the LLM judge score. Cost and latency are estimated from token usage and response time relative to the initial run.

Compare two prompts

promptlab compare \
  --prompt-a "Summarize this review: {input}" \
  --prompt-b "Write exactly two concise sentences covering sentiment and key points: {input}" \
  --label-a "baseline" \
  --label-b "two-sentence version" \
  --task "Product review summarization" \
  --tests examples/summarization.yaml

Prompt library

The prompt library stores prompts in a local SQLite database at:

~/.promptlab/library.db

Save a prompt manually:

promptlab library save \
  --name support-classifier-v1 \
  --prompt "Classify this support ticket: {input}" \
  --task classification \
  --model gpt-4o-mini \
  --score 8.7 \
  --tags "support,classification"

List saved prompts:

promptlab library list
promptlab library list --tag support
promptlab library list --task classification --min-score 8

Show or delete one prompt:

promptlab library get 1
promptlab library delete 1

Run history

Optimization runs are stored in:

~/.promptlab/history.db

List recent runs:

promptlab history --limit 10

Use --no-history with promptlab optimize if you do not want a run written to that database.

Running tests

Install the dev dependencies, then run:

python -m pytest

The tests mock model calls, so they should not require an API key.

License

PromptLab is released under the MIT License. See LICENSE for the full license text.

Practical notes

• Keep test suites small at first. One optimization run can call the target model, judge model, and optimizer model many times.
• Put strict output requirements in the prompt and in the judge rubric. For classification tasks, say whether the model should return only the label.
• Use eval before optimize. It gives you a baseline score and catches broken API configuration quickly.
• Review optimized prompts before using them in production. PromptLab chooses the best candidate by judge score, not by human taste or business risk.
• Generated reports are written wherever you pass --report. Prompt history and library data are written under ~/.promptlab/.