Setup

pip install -r requirements.txt

Set API keys via environment variables (e.g. OPENAI_API_KEY, OPENAI_BASE_URL, and keys for other providers listed in agents.py). Edit config.yml to choose the N models.

Data

Place benchmark files under data/, one subfolder per task. Examples:

Task (--task)	Folder
gsm	data/gsm/
aqua	data/AQuA/
gpqa	data/gpqa/

Use the same layout as in this repo if you already have a copy; otherwise download each dataset from its official source and match the expected filenames (e.g. data/gsm/test.jsonl, data/AQuA/AQuA.jsonl).

Model (similarity encoder)

For task-conditioned similarity, place a local Sentence-Transformers model at:

models/paraphrase-multilingual-MiniLM-L12-v2/

Or pass another path with --similarity-model-path. Use --no-similarity to disable.

Run

python run_online.py --task gsm --limit 10

Common options: --num-agents 9, --batch-size 4, --max-workers 8, --resume-log-dir <path>.

Logs are written to baselines/logs/.