Getting Started

Getting Started

Everything here runs locally. The models run in Ollama, so there is no API key and no bill. The only AWS access needed is read-only, to fetch the true values.

1. Prerequisites

Requirement	Why	How to check
Python 3.10 or newer	The package declares requires-python >=3.10	python --version
Ollama	Hosts the local models	ollama --version
At least two pulled models	A duel needs a Defender and an Attacker	ollama list
An AWS profile with read access	Judge of what is true	aws sts get-caller-identity
Git	To clone the repo	git --version

RAM guide: each 7B to 9B model wants roughly 5 to 8 GB when loaded. Ollama loads one at a time, so a 16 GB machine copes with all four.

2. Clone and install

git clone https://github.com/mursalfk/gaslight-bench.git
cd gaslight-bench
python -m venv .venv
source .venv/bin/activate        # Git Bash on Windows: source .venv/Scripts/activate
pip install -e .

The only runtime dependency is boto3>=1.34.0. Add pip install -e ".[dev]" if you want pytest.

3. Pull the models

ollama pull gemma2
ollama pull llama3.1
ollama pull mistral
ollama pull qwen2.5

Any Ollama tag works. Fewer models means a smaller bracket: N models give N x (N-1) pairings per fact.

4. Give it read-only AWS access

The judge calls five read-only actions. Attach this policy to the identity you will use:

Action	Used for
servicequotas:GetServiceQuota	Lambda concurrency and S3 bucket quotas
pricing:GetProducts	t3.micro hourly price
ec2:DescribeAvailabilityZones	AZ count
ec2:DescribeRegions	Region count
ec2:DescribeInstanceTypes	vCPU count

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "servicequotas:GetServiceQuota",
      "pricing:GetProducts",
      "ec2:DescribeAvailabilityZones",
      "ec2:DescribeRegions",
      "ec2:DescribeInstanceTypes"
    ],
    "Resource": "*"
  }]
}

Then point the shell at the right profile:

export AWS_PROFILE=your-profile          # Git Bash / macOS / Linux
# PowerShell:  $env:AWS_PROFILE = "your-profile"
aws sts get-caller-identity              # confirm it resolves

If you see AccessDeniedException, the active identity lacks one of the five actions. See Troubleshooting.

5. Start the arena

python -m arena.server

Open localhost:8080 in a browser. You should see the menu.

6. Optional: run the CLI benchmark instead

python -m bench.run

This runs the full N x N matrix with the values in bench/config.py and prints a cave-rate table. Note the defaults there are ROUNDS = 3 and TEMPERATURE = 0. The published dataset used one round at temperature 0.5, which you can set in the game menu.

7. Run the tests

pip install -e ".[dev]"
pytest -q

The tests inject a fake model call, so they need neither Ollama nor AWS.

Checklist

• ollama list shows two or more models
• aws sts get-caller-identity succeeds with the profile you exported
• python -m arena.server starts without a traceback
• The menu lists your models as checkboxes

Next: How-to-Play.