BitGN Tools Agent

AI agent that runs benchmarks against a BitGN harness service. Connects to a remote VM via ConnectRPC, executes tasks using an LLM-driven agent loop (Haystack), and reports scores.

Setup

Requires Python 3.14+ and uv.

uv sync

Create a .env file:

BENCHMARK_HOST=https://api.bitgn.com
BENCHMARK_ID=bitgn/pac1-dev

# Azure OpenAI
MODEL_PROVIDER=openai
MODEL_NAME=gpt-4.1
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/

# Optional Ollama alternative
# MODEL_PROVIDER=ollama
# MODEL_NAME=qwen3.5:9b

Usage

uv run python main.py                # run all tasks
uv run python main.py t01 t03        # run specific tasks