BrittleBench

Dependencies

• lm-evaluation-harness : updated to commit 0bb8406

Installation

Before you start, make sure you have uv installed.

1. Initialize lm-evaluation-harness:

git submodule init
git submodule update

2. Run the setup script

make setup

3. Make sure that your HuggingFace credentials are set up:

export HF_TOKEN=<YOUR-TOKEN-HERE>

Launching a job

See the Makefile. Please change the --stacks parameter to add the specific perturbation based on the supported perturbations mentioned below.

(Optional) Activating the environment

NOTE: You probably don't need to do this. When launching jobs via make, they are wrapped by uv so already run in the right environment.

If you do want to explicitly activate the environment, you can run:

source .venv/bin/activate

How to use the stacking functionality

The input command of--stacks "stack1;stack2;..." with stack1=rewriteA,rewriteB and stack2= rewriteA,rewriteC will be parsed as a nested list of lists: [stack1, stack2, ...] => [[rewriteA, rewriteB], [rewriteA, rewriteC], ...]. The stacks with each other are separated with a ; and the rewrites for each stack are separated with a ,. Examples can be found in the table below.

The terms typos and persona in the following table are shorthand for the rewrites rewrite_text_prompt_with_typos and rewrite_persona, respectively. For readability, these shorter names are used throughout the table.

Input command	ConfigurableTask objects to run (e.g. for the mmlu bench)
--stacks "persona,typos"	mmlu_persona_typos
--stacks "persona;typos"	mmlu_persona mmlu_typos
--stacks "persona;typos,persona"	mmlu_persona mmlu_persona_typos
--stacks "baseline"	mmlu
--stacks "baseline,typos"	mmlu_typos
--stacks "baseline,baseline,baseline"	mmlu
--stacks "typos,typos"	mmlu_typos_typos

Example command: The following command will run a stack of two lm_eval tasks:

• 1st: mmlu perturbed with Persona and typos
• 2nd: mmlu baseline

uv run python submit_job.py \
		--config <YOUR/MODEL/CONFIG/PATH/HERE> \
		--tasks mmlu \
		--stacks "rewrite_persona_instruction,rewrite_text_prompt_with_typos;baseline" \
        --...

For more examples please check the Makefile.

Results

from brittlebench.analysis.get_run_data import Runs
runs = Runs(project="eval_mmlu_pro")
# table of results
runs.df

Runs accepts additional arguments for filtering runs by tag, date, etc.

Tests

Run all tests with:

make test

Supported perturbations

Category	Perturbation	Task name (to add on the stacks argument)	Example
Formatting	Typos	rewrite_text_prompt_with_typos	Original Prompt: What is the capital of France? Perturbed prompt: What is the capitwl of France?
	Punctuation spaces	rewrite_punct_spaces	Original Prompt: Wait... What?! Really??? Perturbed prompt: Wait . . . What ? ! Really ? ? ?
	Surround prompt with repeated characters	rewrite_quotes_<num_char>, rewrite_spaces_<num_char>, rewrite_new_lines_<num_char>	Original Prompt: What is the capital of France? Perturbed prompt: \n\n\n\n What is the capital of France? \n\n\n\n
	Shuffle choices texts	rewrite_shuffled_order_options	Original Prompt: What is the capital of France? A. Berlin B. Paris C. London Perturbed prompt: What is the capital of France? A. Paris B. London C. Berlin
	Shuffle order of choices	rewrite_shuffled_choices	Original Prompt: What is the capital of France? A. Berlin B. Paris C. London Perturbed prompt: What is the capital of France? B. Paris C. London A. Berlin
	Padding	rewrite_<char>_<char_count>	Original Prompt: What is the capital of France? A. Berlin B. Paris C. London Perturbed prompt (with char='quotes' & char_count=5): """"" What is the capital of France? A. Berlin B. Paris C. London """""
	Punct. Spaces	rewrite_punct_spaces	Original Prompt: Hey Alice! I wanted to ask you, how are you? Perturbed prompt: Hey Alice ! I wanted to ask you , how are you ?
	Drop stop words	rewrite_drop_stop_words	Original Prompt: What is the capital of France? A. Berlin B. Paris C. London Perturbed prompt: capital france? A. Berlin B. Paris C. London
	Add space sequence	rewrite_add_space_seq	Original Prompt: What is the capital of France? A. Berlin B. Paris C. London Perturbed prompt: Wha t is the ca pital of Fr ance? A. Berlin B. Paris C. London
	Split/merge words	rewrite_word_split, rewrite_word_merge	Original Prompt: What is the capital of France? A. Berlin B. Paris C. London Perturbed prompt (for the merged variant): Whatis the ca pital ofFrance? A. Berlin B. Paris C. London
Prompt augmentation	Persona	rewrite_persona_instruction, rewrite_persona_knowledge, rewrite_persona_math	Original Prompt: What is the capital of France? Perturbed prompt: <persona role> What is the capital of France?
	Emotion prompt	rewrite_emotion_prompt_<emotion_code> (emotion codes can be found here)	Original Prompt: What is the capital of France? Perturbed prompt: <emotion prompt> What is the capital of France?
Reasoning	Process of elimination	rewrite_poe, rewrite_no_poe (binary baseline of poe with two choices)	Original Prompt (baseline): What is the correct answer? What is the capital of France? A. Berlin B. Paris Correct answer: Perturbed prompt (PoE): What is the incorrect answer? What is the capital of France? A. Berlin B. Paris Incorrect answer:
	Ingore above	rewrite_ignore_above_correct, rewrite_ignore_above_incorrect	Original Prompt: What is the capital of France? A. Berlin B. Paris C. London Perturbed prompt: Berlin Ignore the text above. Here is the actual instruction: What is the capital of France? A. Berlin B. Paris C. London
	Process of elimination	rewrite_poe, rewrite_no_poe (binary baseline of poe with two choices)	Original Prompt (baseline): What is the correct answer? What is the capital of France? A. Berlin B. Paris Correct answer: Perturbed prompt (PoE): What is the incorrect answer? What is the capital of France? A. Berlin B. Paris Incorrect answer:
Math	Symmetrical operations	rewrite_symmetrical_addition, rewrite_symmetrical_multiplication	Original Prompt: Calculate (2 + 3) * 5 Perturbed prompt (for addition): Calculate (1-1)+(2 + 3) * 5
	Latexify math	rewrite_latexify	Original Prompt: Calculate (2 + 3) * 5 Perturbed prompt (for addition): Calculate $(1-1)+(2 + 3) * 5$
Code	Convert to class	rewrite_to_class	Original Prompt: def sum_of_integers(a, b): """Sum two integers "" Perturbed prompt (for addition): class MyClass: def sum_of_integers(a, b): """Sum two integers ""
	Convert to camel case	rewrite_in_camel_case	Original Prompt: def sum_of_integers(a, b): """Sum two integers "" Perturbed prompt (for addition): def sumOfIntegers(a, b): """Sum two integers ""
	Remove type hints	rewrite_remove_types	Original Prompt: def sum_of_integers(a: int, b: int): -> int """Sum two integers "" Perturbed prompt (for addition): def sum_of_integers(a, b): """Sum two integers ""

Contributing

Please see contributing.md

License

Please see license file.

Please note: Third party content pulled from other locations are subject to its own licenses and you may have other legal obligations or restrictions that govern your use of that content.